What’s Fair is Fair: Detecting and Mitigating Encoded Bias in Multimodal Models of Museum Visitor Attention

October 18th, 2021 | RESEARCH

Recent years have seen growing interest in modeling visitor engagement in museums with multimodal learning analytics. In parallel, there has also been growing concern about issues of fairness and encoded bias in machine learning models. In this paper, we investigate bias detection and mitigation techniques to address issues of algorithmic fairness in multimodal models of museum visitor visual attention. We employ slicing analysis using the Absolute Between-ROC Area (ABROCA) statistic to detect encoded bias present in multimodal models of visitor visual attention trained with facial expression and posture data from visitor interactions with a game-based museum exhibit about environmental sustainability. We investigate instances of gender bias that arise between different combinations of modalities across several machine learning techniques. We also measure the effectiveness of two different debiasing strategies—learned fair representations and reweighing—when applied to the trained multimodal visitor attention models. Results indicate that patterns of bias can arise across different modality combinations for the different visitor visual attention models, and there is often an inherent tradeoff between predictive accuracy and ABROCA. Analyses suggest that debiasing strategies tend to be more effective on multimodal models of visitor visual attention than their unimodal counterparts.



Team Members

Halim Acosta, Author, North Carolina State University
Nathan Henderson, Author, North Carolina State University
Jonathan Rowe, Co-Principal Investigator, North Carolina State University
Wookhee Min, Author, North Carolina State University
James Minogue, Co-Principal Investigator, North Carolina State University
James Lester, Principal Investigator, North Carolina State University


Identifier Type: doi
Identifier: 10.1145/3462244.3479943

Publication: Proceedings of the 2021 International Conference on Multimodal Interaction
Page(s): 258-267


Funding Source: NSF
Funding Program: Advancing Informal STEM Learning (AISL)
Award Number: DRL-1713545
Funding Amount: $1,951,956.00

Related URLs

Multimodal Visitor Analytics: Investigating Naturalistic Engagement with Interactive Tabletop Science Exhibits


Audience: Elementary School Children (6-10) | Learning Researchers | Middle School Children (11-13) | Museum | ISE Professionals
Discipline: Climate | Ecology | forestry | agriculture
Resource Type: Conference Proceedings | Research
Environment Type: Games | Simulations | Interactives | Media and Technology | Museum and Science Center Exhibits

Linkedin   Youtube   Facebook   Instagram
Search: repository | repository and website pages | website pages
NSF logo

This material is supported by National Science Foundation award DRL-2229061, with previous support under DRL-1612739, DRL-1842633, DRL-1212803, and DRL-0638981. Any opinions, findings, conclusions, or recommendations contained within InformalScience.org are those of the authors and do not necessarily reflect the views of NSF.

NSF AISL Project Meetings

Privacy Policy | Terms of Use | Contact Us