Early Prediction of Museum Visitor Engagement with Multimodal Adversarial Domain Adaptation

June 21st, 2021 | RESEARCH

Recent years have seen significant interest in multimodal frameworks for modeling learner engagement in educational settings. Multimodal frameworks hold particular promise for predicting visitor engagement in interactive science museum exhibits. Multimodal models often utilize video data to capture learner behavior, but video cameras are not always feasible, or even desirable, to use in museums. To address this issue while still harnessing the predictive capacities of multimodal models, we investigate adversarial discriminative domain adaptation for generating modality-invariant representations of both unimodal and multimodal data captured from museum visitors as they engage with interactive science museum exhibits. This approach enables the use of pre-trained multimodal visitor engagement models in circumstances where multimodal instrumentation is not available. We evaluate the visitor engagement models in terms of early prediction performance using exhibit interaction and facial expression data captured during visitor interactions with a science museum exhibit for environmental sustainability. Through the use of modality-invariant data representations generated by the adversarial discriminative domain adaptation framework, we find that pre-trained multimodal models achieve competitive predictive performance on interaction-only data compared to models evaluated using complete multimodal data. The multimodal framework outperforms unimodal and non-adapted baseline approaches during early intervals of exhibit interactions as well as entire interaction sequences.



Team Members

Nathan Henderson, Author, North Carolina State University
Wookhee Min, Author, North Carolina State University
Andrew Emerson, Author, North Carolina State University
Jonathan Rowe, Co-Principal Investigator, North Carolina State University
Seung Lee, Author, North Carolina State University
James Minogue, Co-Principal Investigator, North Carolina State University
James Lester, Principal Investigator, North Carolina State University


Publication: Proceedings of the Fourteenth International Conference on Educational Data Mining
Page(s): 93-104


Funding Source: NSF
Funding Program: Advancing Informal STEM Learning (AISL)
Award Number: 1713545
Funding Amount: $1,951,956.00

Related URLs

Multimodal Visitor Analytics: Investigating Naturalistic Engagement with Interactive Tabletop Science Exhibits


Audience: Learning Researchers | Museum | ISE Professionals
Discipline: Climate | Ecology | forestry | agriculture
Resource Type: Conference Proceedings | Research
Environment Type: Games | Simulations | Interactives | Media and Technology | Museum and Science Center Exhibits

Linkedin   Youtube   Facebook   Instagram
Search: repository | repository and website pages | website pages
NSF logo

This material is supported by National Science Foundation award DRL-2229061, with previous support under DRL-1612739, DRL-1842633, DRL-1212803, and DRL-0638981. Any opinions, findings, conclusions, or recommendations contained within InformalScience.org are those of the authors and do not necessarily reflect the views of NSF.

NSF AISL Project Meetings

Privacy Policy | Terms of Use | Contact Us