While traditional textbooks are designed to transmit information from the printed page to the learner, contemporary digital textbooks offer the opportunity to study learners as they interpret and process information being read, make personalized recommendations for further study and review, and to thereby promote long-term retention and conceptual understanding. Our project explored this objective by instrumenting digital textbooks and e-readers to collect data from learners as they read and interact with the material. Two data sources were considered: (1) highlighting annotations that learners make as they read, indicating material the learner considers to be key content, and (2) and finger-scrolling actions using an electronic e-reader.

Although students believe that highlighting and subsequent review of the highlights will further their educational goals, the psychological literature provides little evidence of benefits. Nonetheless, students' choice of text for highlighting may serve as a window into their mental state—their level of comprehension, grasp of the key ideas, reading goals, and so on. We explored this hypothesis via an experiment in which 400 participants read three sections from a college-level biology text, briefly reviewed the text, and then took a quiz on the material. During initial reading, participants were able to highlight words, phrases, and sentences, and these highlights were displayed along with the complete text during the subsequent review. Consistent with past research, the amount of highlighted material is unrelated to quiz performance. However, we found that the specific content highlighted could help us to predict quiz performance. We explored a range of models and a range of highlighting representations to determine which combination yielded the best predictive accuracy. We also explored matrix factorization models that allowed us to show that individuals' highlighting pattern is informative of what they highlight elsewhere, which could be useful for content-recommendation engines.

We moved from laboratory experiments to an analysis of a large-scale corpus of highlighting data collected in a genuine educational context by a digital open-access platform, OpenStax. The corpus consisted of data from over 11,000 students and nearly 900 sections from College Biology and Physics textbooks. In addition to highlighting data, the corpus provided scores on delayed quizzes associated with the sections. We considered two approaches to encoding highlighting patterns. A positional representation indicated where highlights were made in the stream of text for a particular section. A semantic representation was based on a state-of-the-art deep-learning sentence embedding technique (SBERT) that captures the content-based similarity between quiz questions and highlighted (as well as non-highlighted) sentences in the text. We construct regression models that include latent variables for student skill level and question difficulty and augment the models with highlighting features. We find that both positional and semantic highlighting features reliably boost model performance, with semantic features being the more predictive. We conduct experiments that validate models on held-out questions, students, and student-questions and find strong generalization for the latter two but not for held-out questions. Surprisingly, highlighting features improve models for questions at all levels of the Bloom taxonomy, from straightforward recall questions to inferential synthesis/evaluation/creation questions.

Highlights explain only a portion of observed variability in performance. As is typical in big-data applications, one seeks multiple weak predictor variables which in combination can make strong predictions. Toward this goal, we examined readers' scrolling patterns on a personal e-reader (table, phone). We collected a data set consisting of 20 readers reading 20 newspaper articles. The data indicated the exact timing of finger motions and flicks, allowing us to determine how long text appeared on the screen, and thus allowed us to infer content-specific reading rates. We found that we could predict difficulty of content from reading rates and we could identify individual differences in scrolling patterns. In follow up work, we plan to explore the relationship between scrolling patterns and comprehension and retention.

Our long-term goal is to design digital textbooks that serve not only as conduits of information into the reader’s mind but also allow us to draw inferences about the reader at a point where interventions may increase the effectiveness of the material. Interventions might include alterting the instructor, posing questions that allow students to evaluate their understanding, or guiding students to improve by suggesting content to review.

Last Modified: 10/29/2021
Modified by: Michael C Mozer