Toward Data-Rich Models of Visitor Engagement with Multimodal Learning Analytics

Visitor engagement is critical to the effectiveness of informal learning environments. However, measuring visitor engagement raises significant challenges. Recent advances in multimodal learning analytics show significant promise for addressing these challenges by combining multi-channel data streams from fully-instrumented exhibit spaces with multimodal machine learning techniques to model patterns in visitor experience data. We describe initial work on the creation of a multimodal learning analytics framework for investigating visitor engagement with a game-based interactive surface exhibit for science museums called Future Worlds. The multimodal visitor analytics framework involves the collection of multichannel data streams, including facial expression, eye gaze, posture, gesture, speech, dwell time, and interaction trace log data, combined with traditional visitor study measures, such as surveys and field observations, to triangulate expressions of cognitive, affective, behavioral, and social engagement during museum-based learning. These data streams will be analyzed using machine learning techniques, with a focus on deep recurrent neural networks, to train and evaluate computational models of engagement using non-intrusive data sources as input (e.g., interaction logs, non-identifying motion tracking data). We describe distinctive opportunities and challenges inherent in using multimodal analytics within informal settings, as well as directions for utilizing multimodal visitor analytics to inform work by exhibit designers and museum educators.