Award Abstract # 1739012
EXP: Linguistic Analysis and a Hybrid Human-Automatic Coach for Improving Math Identity

NSF Org: IIS
Division of Information & Intelligent Systems
Recipient: TRUSTEES OF THE UNIVERSITY OF PENNSYLVANIA, THE
Initial Amendment Date: April 25, 2017
Latest Amendment Date: December 14, 2017
Award Number: 1739012
Award Instrument: Standard Grant
Program Manager: Hector Munoz-Avila
IIS
 Division of Information & Intelligent Systems
CSE
 Directorate for Computer and Information Science and Engineering
Start Date: September 1, 2016
End Date: August 31, 2020 (Estimated)
Total Intended Award Amount: $540,047.00
Total Awarded Amount to Date: $556,047.00
Funds Obligated to Date: FY 2016 = $540,047.00
FY 2017 = $16,000.00
History of Investigator:
  • Jaclyn Ocumpaugh (Principal Investigator)
    jlocumpaugh@gmail.com
  • Ryan Baker (Co-Principal Investigator)
  • Scott Crossley (Co-Principal Investigator)
  • Matthew Labrum (Co-Principal Investigator)
  • Victor Kostyuk (Former Co-Principal Investigator)
Recipient Sponsored Research Office: University of Pennsylvania
3451 WALNUT ST STE 440A
PHILADELPHIA
PA  US  19104-6205
(215)898-7293
Sponsor Congressional District: 03
Primary Place of Performance: University of Pennsylvania
PA  US  19104-6205
Primary Place of Performance
Congressional District:
03
Unique Entity Identifier (UEI): GM1XX56LEP58
Parent UEI: GM1XX56LEP58
NSF Program(s): Cyberlearn & Future Learn Tech
Primary Program Source: 01001617DB NSF RESEARCH & RELATED ACTIVIT
01001718DB NSF RESEARCH & RELATED ACTIVIT
Program Reference Code(s): 8045, 8841, 9251
Program Element Code(s): 802000
Award Agency Code: 4900
Fund Agency Code: 4900
Assistance Listing Number(s): 47.070

ABSTRACT

The Cyberlearning and Future Learning Technologies Program funds efforts that support envisioning the future of learning technologies and advance what we know about how people learn in technology-rich environments. Cyberlearning Exploration (EXP) Projects explore the viability of new kinds of learning technologies by designing and building new kinds of learning technologies and studying their possibilities for fostering learning and challenges to using them effectively. This project addresses the effect of students' social identity on learning, an important factor in math and science education. Specifically, it will advance the scientific understanding of math identity (i.e. "I'm (not) a math person") by studying the over 100,000 diverse students who use Reasoning Mind, a blended learning system for K-8 mathematics with demonstrated results. Reasoning Mind supports math identity with an innovative design that allows students to email an animated character (aka the Genie) and receive a human-crafted response. This study will show how math identity manifests and changes during students' use of Reasoning Mind in order to inform software designers and classroom teachers on best practices for encouraging math identity. This will have the broader impact of strengthening our nation's ability to supply science and technology fields with a well-trained workforce.

This study examines two components of math identity: self-efficacy and interest in mathematics. Both self-efficacy and interest can be enhanced by curricula that are individualized to appropriately challenge each student (a strength of educational technology), but social stereotypes (i.e. "Girls aren't good in math") may decrease math identity or otherwise interfere with its development. This study investigates how educational technology can reverse these trends. In the first phase of this study, a combination of survey methods, Natural Language Processing (NLP), and Educational Data Mining (EDM) techniques will be used to identify how poor and/or changing math identity emerges in the linguistic patterns of student interaction with GenieMail (as well as in other parts of Reasoning Mind). These findings will then be used to enhance GenieMail and other instructional interactions with the Reasoning Mind system by creating a hybrid human/AI system, with the goal of improving math identity across diverse populations of students.

PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:  When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

(Showing: 1 - 10 of 14)
Crossley, S. A., Bradfield, F., & Bustamante, A "GAMET: A text analysis tool to automatically assessing grammar, syntax, and mechanical errors." Journal of Writing Research. , 2019
Crossley, S. A., Bradfield, F., & Bustamante, A. "Using human judgments to examine the validity of automated grammar, syntax, and mechanical errors in writing." Journal of Writing Research , v.11 , 2019 10.17239/jowr-2019.11.02.01
Crossley, S. A., Karumbaiah, S., Labrum, M., Ocumpaugh, J., & Baker, R. "Predicting Math Success in an Online Tutoring System Using Language Data and Click-Stream Variables: A Longitudinal Analysis." Proceedings of Language Data and Knowledge , v.70 , 2019 10.4230/OASIcs.LDK.2019.25
Crossley, S. A., Karumbaiah, S., Labrum, M., Ocumpaugh, J., & Baker, R. "Predicting Math Success in an Online Tutoring System Using Language Data and Click-Stream Variables: A Longitudinal Analysis." Proceedings of Language Data and Knowledge, , v.70 , 2019 , p.1 Doi: 10.4230/OASIcs.LDK.2019.25
Crossley, S. A., Kyle, K., & Dascalu, M. "The Tool for the Automatic Analysis of Cohesion 2.0: Integrating semantic similarity and text overlap." Behavior Research Methods , v.51 , 2019 , p.14
Crossley, S. A., Kyle, K., & Dascalu, M. "The Tool for the Automatic Analysis of Cohesion 2.0: Integrating semantic similarity and text overlap." Behavior Research Methods , v.51 , 2019 10.4230/OASIcs.LDK.2019.25
Crossley, S. A., Kyle, K., & Dascalu, M. "The Tool for the Automatic Analysis of Cohesion 2.0: Integrating Semantic Similarity and Text Overlap." Behavioral Research Methods. , 2019
Crossley, S.A., Ocumpaugh., J., Karumbaiah, S., Labrum, M.J., Baker, R. "Predicting math identity through language and click-stream patterns in a blended learning mathematics program for elementary students." Journal of Learning Analytics , v.7 , 2020 , p.19
Crossley, S. A., Ocumpaugh, J., Labrum, M., Bradfield, F., Dascalu, M., & Baker, R. "Modeling Math Identity and Math Success through Sentiment Analysis and Linguistic Features." Proceedings of the 10th International Conference on Educational Data Mining (EDM) , 2018 , p.11
Karumbaiah, S., Ocumpaugh, J., & Baker R. "The Influence of School Demographics on the Relationship Between Students Help-Seeking Behavior and Performance and Motivational Measures." Proceedings of the 12th International Conference on Educational Data Mining , 2019 , p.99
Karumbaiah, S., Ocumpaugh, J., Labrum, M.J., Baker, R.S. "Temporally Rich Features Capture Variable Performance Associated with Elementary Students Lower Math Self-concept." Proceedings of the Workshop on Social-Emotional Learning at the 9th International Learning Analytics and Knowledge Conference. , 2019
(Showing: 1 - 10 of 14)

PROJECT OUTCOMES REPORT

Disclaimer

This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.

This project studied emails between elementary school students, and the Genie, a character in the Reasoning Mind blended learning system for mathematics. Analyses of students who emailed the Genie found that differences in their language predicted differences in both Math Identity and math success. In addition, we developed a new approach to automatically examine links between student’s self-reported math identity, interactions within the Reasoning Mind system, and the language used in their emails.

Analysis of more general student interactions with the system has also been conducted. These papers have explored how the consistency of student behaviors correlates with higher Math Identity scores; students with high but inconsistent performance over the year have lower self-reported scores of self-concept, interest, and value of mathematics than students with lower but consistent performance. We found that Reasoning Mind students start and end the year with high self-concept scores, despite past research that suggests that students often begin to see declines in these scores during middle school. We also show that school-level demographics appear to moderate the relationship between student help-seeking behaviors and Math Identity Scores. Finally, our deployment of automated interventions indicated that proactive interventions – messages sent to all students -- achieve a lower degree of response and uptake than reactive interventions sent in response to email. This finding suggests that reactive interventions – because they come as a response to the student’s own choice to interact – may be a more felicitous situation to deploy interventional content.

The project also led to the development and refinement of new tools for Natural Language Processing, including a tool for automatically assessing writing errors in emails written by young children, which represents an important milestone in the field since most tools have not been trained on data from such young writers.

 


Last Modified: 11/24/2020
Modified by: Jaclyn Ocumpaugh

Please report errors in award information by writing to: awardsearch@nsf.gov.

Print this page

Back to Top of page