Computer-based educational tools vary widely, and so do their effects. Pane, et al. (2014) reported the results of a large-scale effectiveness study of Cognitive Tutor Algebra I (CTAI) by Carnegie Learning, Inc. This curriculum is designed to have students work individually with tutorial software for 40 percent of class time. Cognitive Tutor is one of the most prominent and influential intelligent tutors. The study found that in its second year of use, CTAI boosted high school students’ test scores substantially. Over the course of the study, the software gathered student log data, capturing which problems, sections, and units students worked on, data on hints and errors, and time stamps. Since the experiment was an effectiveness study, testing the curriculum in a wide variety of settings under real-life conditions, student usage in the experiment reflects typical usage in the field.

The purpose of our study was to explore the log data to understand how students varied in the way they used the software. Next, we attempted to understand how the effectiveness of the software varied for students who used it differently. To do this, we used principal stratification, a framework for modeling treatment effect variation as a function of variables, like software use, that themselves are affected by treatment assignment.

We learned that students varied widely in the amount of time spent using the software, and the number of problems and sections worked. In the second year of the study, they also varied in which sections and topics they worked on. This variation occurred mostly at the school or state level, suggesting that teachers or contextual factors are influential. Ideally, students would progress from one section to the next only after mastering the contents of the first section, but roughly 15% of the time, they exit the section without mastering its contents. Typically, this is because they exhaust all of the sections’ problems before achieving mastery. Using a novel principal stratification technique, we found, somewhat surprisingly, that students who are more likely to master the sections they work may experience smaller treatment effects. This may be because CTAI is most beneficial for weaker students, who are also less likely to master worked sections. Using a similar technique, we also found that treatment effects are smaller for students who are either very likely or very unlikely to ask for hints or make errors, and larger for students in the middle. We found that students who work sections out of order appear to have smaller treatment effects, and that reassigning a student out of a section before mastery appears to lower posttest scores. Together, these results suggest that CTAI works best for students who use it as intended, particularly if they struggle moderately in doing so.

We also sought to spur interest and expertise in principal stratification and causal modeling among researchers who study systems like this, and to further develop statistical methods for principal stratification. To that end, we ran a principal stratification workshop and submitted a principal stratification tutorial paper to a journal. We also worked with researchers studying other online tutoring systems, particularly ASSISTments, to help them use principal stratification to analyze their log data.

The project resulted in three journal articles, three conference proceedings papers, six conference presentations, three seminars, and a tutorial workshop.

Journal articles:

Sales, AC and Pane, JF. “The Role of Mastery Learning in Intelligent Tutoring Systems: Principal Stratification on a Latent Variable.” Under revision.

Sales, AC and Pane, JF. “Principal Stratification for Intelligent Tutors: A Tutorial.” Submitted.

Israni, A, Sales, AC, and Pane, JF. “Mastery Learning in Practice:?A (Mostly) Descriptive Analysis of Log Data from the Cognitive Tutor Algebra I Effectiveness Trial.” Submitted.

Conference papers:

Sales, AC and Pane, J. (2015). “Exploring Causal Mechanisms in a Randomized Effectiveness Trial of the Cognitive Tutor.” Proceedings of the 8th International Conference on Educational Data Mining.

Williams, JJ, Botelho, A, Sales, AC, Heffernan, N, and Lang, C. (2016) “Discovering ‘Tough Love’ Interventions Despite Dropout.” Proceedings of the 9th International Conference on Educational Data Mining.

Sales, AC, Wilks, A, and Pane, JF. (2016) “Student Usage Predicts Treatment Effect Heterogeneity in the Cognitive Tutor Algebra I Program.” Proceedings of the 9th International Conference on Educational Data Mining.

Other conference presentations (not listed above):

Atlantic Causal Inference Conference (2016), Conference of the Association for Education Finance and Policy (2016), Conference of the Society for Research on Educational Effectiveness (2018).

Tutorial workshop:

2017 International Conference on Educational Data Mining (co-hosted with Artificial Intelligence in Education conference). 

Seminars:

University of Michigan; Carnegie Mellon University; University of Texas, Austin.

Reference cited in this report (not produced by this project):

Pane, J. F., Griffin, B. A., McCaffrey, D. F., & Karam, R. (2014). Effectiveness of Cognitive Tutor Algebra I at Scale. Educational Evaluation and Policy Analysis, 36(2), 127-144.

Last Modified: 01/29/2018
Modified by: John F Pane