We showed that detection of affect and “state of grit” in intelligent tutors is a promising avenue for improving online learning and education. The state of grit entails working to overcome challenges and maintaining effort over time despite failures. We used computer vision techniques to analyze students’ faces and gestures and to develop methods to respond to their perceived engagement. The facial expression detection mechanisms worked in real time so that the online system both recognized grit and responded to it.

Predicting whether students have trouble with exercises supports online systems to provide interventions, such as hints or encouragement. We detected students’ grit state in real time; investigated which interventions were effective when students’ initiative began to fade; and contributed to theories of affect and motivation. These studies were conducted in the context of MathSpring.org, an intelligent tutor that supports practice with Common Core mathematics problems online.  

In order to measure grit, we first created videos of online student engagement. While students work online, they demonstrated various levels of engagement and emotions (e.g., confusion, boredom, excitement). Having such information automatically accessible to teachers aids them to understand students’ progress, suggesting when and who needs further assistance.

We classified facial expressivity to predict student behavior. Specifically, we developed automated facial expression detectors that differentiated between grit and its absence, relatively early (e.g., a few seconds into the learning session). Accuracy in detecting grit grew with the number of problems attempted by each student. The most accurate detectors of grit were based on a combination of machine learning models and human-engineered factors. The system could personalize its interventions, such as hints and encouragement. 

The computer vision system predicted student behavior based on students’ facial expressions, seconds before students responded or answers were made. It made predictions with relatively high accuracy using only several seconds of video footage, well before students had solved the mathematics problems. 

We also developed Affective Teacher Tools, including a report card that presents measures of students’ engagement and a live affective states Dashboard which senses students’ affect and performance. We designed several prototypes of these tools. We produced both a student “Emotion Chart” and an “Effort Chart” that visualized students’ subjective self-reports of their frustration, excitement, interest, and confidence while they solved mathematical problems. 

Our facial expression recognition software detected students' engagement and emotional expressions and inferred student disengagement. The detector achieved 97.94% accuracy as compared with human annotations and earlier system predictions. By analyzing the estimation results of Head Pose Detector on MathSpring videos, we further found invaluable information in the juxtaposition of head position and the logs of problem-solving activity. For example, it appears that a “head tilt" is a sign of concentration and cognitive engagement. We showed that finetuning of the affect network with age-appropriate images and video further improves performance in this scenario. 
We manually created our own "ground truth" about recognizing student engagement by examining multiple videos to determine the angles for each student’s head position while they solved problems or were distracted. The machine learning model automated this process.

Finally, we collected and made public a video dataset of nearly three thousand frames of college students solving math problems (https://www.cs.bu.edu/faculty/betke/research/learning/). 

We also worked in Latin America with a version of the mathematics system translated into Spanish. This data set contains over 35 hours of facial expressions of 11-year-old children using MathSpring to practice math problem solving as part of their regular mathematics classes in either Spanish speaking or bilingual schools. 

In sum, we showed that an intelligent tutor can estimate the real-time gaze of students and respond to loss of grit, and consequently disengagement. We produced and tested Teacher Tools that identified students' affective states and provided enhanced information to teachers. Previous work has shown that real time signals from students can be used to improve learning. Our publications summarized both the report card and the affective dashboard, the research studies, results, the implications, and future planned experiments. These results pave the way for future improvements for this task and future tutor systems may use our outcome prediction models to deliver real-time interventions to improve students’ learning.

Last Modified: 01/31/2023
Modified by: Margrit Betke