Over the five years of this IUSE Engaged Student Learning project, our five-year agenda had three foci: (1) developing, testing, and elaborating upon previous successful prediction modeling research by transferring it to a new institutional context, (2) investigating the efficacy of a variety of science of learning interventions to determine the optimal match of intervention design to student needs, and (3) to examine whether the efficacy of these interventions varied depending upon whether a student was a member of an underrepresented minority or first-generation college student group. Building on previous NSF-funded research at another institution, we applied and refined methods of gathering trace data from students’ engagement with course digital tools (e.g., learning management systems, video hosting software, assessment tools) and then using theory-aligned coding schemes to categorize and understand that trace data, separating the student learning signal from the digital noise. Then, we used those theory-aligned digital trace data that reflected student learning to predict, with a high degree of accuracy, which students would successfully obtain the course grade they needed to proceed in their major, and which ones would not. Using these prediction models, we were able to test a variety of interventions, based in empirical work on the science of learning, designed to help those students predicted to struggle in the course. We were particularly interested in examining which interventions worked, for whom, and under what conditions.

Due to efforts of the research team over the five years of this project, we produced numerous findings that enhanced the scholarly literature and evidenced, and showed promise of, broader impact. First, we showed previous prediction modeling efforts could be successfully transferred to a new context and then elaborated to include new forms of data that informed prediction through multimodal learning analytics methods. We developed prediction models with a high degree of accuracy that could be reapplied across numerous semesters. These models can be used to identify students likely to struggle in the course after only three class periods worth of digital trace data. The value of using a prediction model that can accurately identify struggling students early in the semester is that there is still ample time to intervene and help these students succeed. In essence, these prediction models identify students before their academic difficulties threaten their STEM persistence and careers. Further, because we co-designed courses and data models with instructors, our prediction models are an example of explainable artificial intelligence, where the prediction models include intuitive predictors that derive from instructional design, and can thus provide insight on ways to support learning through redesign and responsive types of learner support. Second, we tested a variety of learner supports by redesigning and testing new versions of the Science of Learning to Learn interventions. These included highly scalable flyer-like digital advice pages, 15-minute trainings on essential course design features and strategies to navigate them, and 90-minute interactive video-based modules to provide more comprehensive learning skill development. These can be delivered at low to no cost. Prediction models can also be used to fast-track referrals to one-on-one, in person coaching interventions that are less scalable but more adaptable to individual student needs and that can be commenced earlier in the semester than when students typically begin to use these services (i.e., after a first poor grade). Across eight semesters of intervention deployment and evaluation, with 3,321 consented students, we found that the success of the intervention depended critically upon students’ motivations for learning. Students with positive motivations, students who adhered to their instructors’ design for the course, and students who fully engaged with the interventions performed better in the course. Our results showed there are complex interactions between motivation, student learning activities, and their success in the course, all indicating the need for more research with larger samples to investigate how these interactions might vary for students who are a member of an underrepresented minority or first-generation college student group. Finally, our efforts elucidated how to conduct effective, ethical, and transformative learning analytics research.

To date, our scholarly efforts have been shared via 29 presentations at national and international conferences, six presentations to practitioner and technical audiences, and five peer-reviewed journal publications. This work has involved four doctoral students and two postdoctoral research associates, both of whom have secured tenure-line academic positions. Also, we have developed productive relationships with numerous industry partners, each of whom has benefitted our project with the infrastructure and support while also benefitting themselves from our findings and insights. Finally, our findings have informed scholarly, educational, and industry partners on how to understand and promote undergraduate STEM student success via learning analytics and theory- and empirically-supported efforts to develop successful courses and provide scalable and effective support to students who need assistance maximizing the opportunities those courses provide.

Last Modified: 01/27/2024
Modified by: Jeffrey A Greene