The primary objective of this project was to investigate museum visitor engagement using multimodal data and learning analytic techniques in order to devise an enriched understanding of naturalistic visitor engagement at the individual and small-group levels. During free-choice learning in museums and science centers, visitor engagement shapes how learners interact with exhibits, move around the exhibit space, and form attitudes, interests, and understanding of science. With the aim of enriching our understanding of naturalistic visitor engagement in informal environments, we investigated museum visitors' learning experiences with multimodal visitor analytics, marrying the rich multichannel data streams produced by fully-instrumented exhibit spaces with the data-driven modeling functionalities afforded by recent advances in machine learning and educational data mining. With a focus on modeling visitors' cognitive, affective, and behavioral engagement during free-choice learning, the project collected, analyzed, and modeled visitor experience data using a range of multimodal channels, including motion tracking, facial expression, eye-tracking, gesture, and when available, trace log data generated by visitors' interactions with interactive surface science exhibits. These data streams complemented measures of dwell time, questionnaires, visitor location tracking, conversation, and field observations.

The project had two major thrusts that drove research activities:

1. Conducting a series of visitor studies on naturalistic engagement in individual, dyad, and group visitor interactions with multimodal visitor analytics. To collect multichannel interaction data on visitors' cognitive, affective, and behavioral engagement, we conducted studies with museum visitors as they interacted with the Future Worlds interactive surface science exhibit, which focuses on environmental sustainability and human impact on the environment. The project's initial focus was to obtain baseline multichannel data on visitor engagement by conducting fully-instrumented studies with recruited participants exploring the interactive surface exhibits. In subsequent years, the project transitioned toward more naturalistic studies and organically formed participant groups, all occurring within the museum. 

2. Creating learning analytic models of visitors' cognitive, affective, and behavioral engagement with interactive surface science exhibits. We investigated applications of probabilistic and neural machine learning techniques to devise predictive models of visitor engagement. These models focused on run-time detection of visitors' engagement signatures using non-intrusive data streams, such as telemetry data (i.e., interaction trace logs) and non-identifying behavioral sensor data (i.e., motion tracking, gesture), as visitors interacted with the Future Worlds exhibit. We operationalized visitor engagement in terms of variables triangulated from multichannel data. To predict these variables from non-intrusive data streams, we employed data fusion methods and extended techniques from educational data mining, as well as evaluated the sensitivity of these models to individual data sources.

The project has made significant contributions to our understanding of naturalistic visitor engagement while interacting with an exhibit in museums using multimodal data and learning analytic techniques. The project has seen the formulation of an empirically-based framework for multimodal learning analytics for visitors' engagement during free-choice learning. By utilizing a suite of rich multimodal data captured from a series of visitor studies on engagement in individual, dyad, and group visitor interactions, the project has created a multimodal learning analytics framework that models engagement with telemetry data (interaction trace logs) and non-identifying behavioral sensor data (e.g., posture, facial action units, gaze) from visitors' science problem solving exploring environmental sustainability and human impact on the environment. The project has produced several learning analytic techniques that can be leveraged in multimodal, predictive models of engagement in science problem solving, including (1) adversarial discriminative domain adaptation models trained with multimodal data that be adapted to circumstances where only a subset of data channels are available by generating modality-invariant representations, (2) bias detection and mitigation models to address issues of algorithmic fairness in multimodal models of museum visitor visual attention, and (3) early prediction models of visitor engagement that utilize multimodal sensor data including eye gaze, facial expression, posture, and interaction log data to induce predictive models of visitor dwell time. Together, these findings highlight the efficacy of multimodal data for modeling museum exhibit visitor engagement, and the project has made methodological contributions that pave the way toward going beyond self-report measures, field observation, and interviews, which have long dominated research on engagement. In addition, the project has made contributions on dissemination of the software and data. The project has also created a multimodal visitor informatics platform to support multimodal data collection (e.g., body movement, facial expression, acoustic features) and a multimodal dataset on visitor free-choice learning. In addition, the project also saw enhancements to the Future Worlds environmental science exhibit, which is now a permanent installation serving the general public at the North Carolina Museum of Natural Sciences. The project provided training and preparation of 14 graduate and undergraduate students in computer science, psychology, and design at North Carolina State University and Lafayette College.

Last Modified: 04/30/2023
Modified by: James C Lester