Toward Data-Rich Models of Visitor Engagement with Multimodal Learning Analytics

June 25th, 2019 | RESEARCH

Visitor engagement is critical to the effectiveness of informal learning environments. However, measuring visitor engagement raises significant challenges. Recent advances in multimodal learning analytics show significant promise for addressing these challenges by combining multi-channel data streams from fully-instrumented exhibit spaces with multimodal machine learning techniques to model patterns in visitor experience data. We describe initial work on the creation of a multimodal learning analytics framework for investigating visitor engagement with a game-based interactive surface exhibit for science museums called Future Worlds. The multimodal visitor analytics framework involves the collection of multichannel data streams, including facial expression, eye gaze, posture, gesture, speech, dwell time, and interaction trace log data, combined with traditional visitor study measures, such as surveys and field observations, to triangulate expressions of cognitive, affective, behavioral, and social engagement during museum-based learning. These data streams will be analyzed using machine learning techniques, with a focus on deep recurrent neural networks, to train and evaluate computational models of engagement using non-intrusive data sources as input (e.g., interaction logs, non-identifying motion tracking data). We describe distinctive opportunities and challenges inherent in using multimodal analytics within informal settings, as well as directions for utilizing multimodal visitor analytics to inform work by exhibit designers and museum educators.



Team Members

Jonathan Rowe, Co-Principal Investigator, North Carolina State University
Wookhee Min, Author, North Carolina State University
Seung Lee, Author, North Carolina State University
Bradford Mott, Author, North Carolina State University
James Lester, Principal Investigator, North Carolina State University


Publication: Workshop on Adaptive and Intelligent Technologies for Informal Learning, co-located with the 20th International Conference on Artificial Intelligence in Education (AIED-2019)


Funding Source: NSF
Funding Program: Advancing Informal STEM Learning (AISL)
Award Number: 1713545
Funding Amount: $1,951,956.00

Related URLs

Multimodal Visitor Analytics: Investigating Naturalistic Engagement with Interactive Tabletop Science Exhibits


Audience: Learning Researchers | Museum | ISE Professionals
Discipline: Climate | Ecology | forestry | agriculture
Resource Type: Conference Proceedings | Research
Environment Type: Games | Simulations | Interactives | Media and Technology | Museum and Science Center Exhibits

Linkedin   Youtube   Facebook   Instagram
Search: repository | repository and website pages | website pages
NSF logo

This material is supported by National Science Foundation award DRL-2229061, with previous support under DRL-1612739, DRL-1842633, DRL-1212803, and DRL-0638981. Any opinions, findings, conclusions, or recommendations contained within are those of the authors and do not necessarily reflect the views of NSF.

NSF AISL Project Meetings

Privacy Policy | Terms of Use | Contact Us