ABSTRACT

In ecosystems that form professional engineers, community members produce text through many activities such as end-of-semester feedback to instructors, transcripts of instruction, open-ended survey items, and interviews. In each case, there is abundant text available to educators and researchers that could provide insight into how we form engineers. Unfortunately, while these texts have the potential to provide novel insights, traditional analytic techniques do not scale well. Time investments, bias, interrater reliability, and intrarater reliability each present significant challenges. To address this problem, we aim to develop and characterize approaches for human-in-the-loop (HITL) natural language processing (NLP) systems to augment human analysis, facilitating and enhancing the work of one person (or team). Such systems can help reduce the amount of time needed to analyze texts by grouping similar texts together. The human user can utilize these groupings for further analysis and identify meanings in ways only a human could. The system will also improve consistency by analyzing across the entire collection of texts simultaneously and grouping similar items together. This is in contrast with a single person or a team that would analyze responses sequentially, creating the potential for inconsistencies across time.

We will accomplish this work in three phases. In Phase 1, we will conduct a series of experiments to test potential system configurations. The goal will be to identify optimal components and parameter settings for four of the steps in the proposed pipeline. We will use datasets from (i) students? written responses to an instrument for assessing their systems thinking and (ii) students? responses to open-ended course feedback surveys. We will measure performance based on consistency of thematic clusters, using standard metrics for homogeneity in text clustering and classification tasks. In Phase 2, we will study system performance on a series of five datasets. These datasets will come from multiple sources: extant NSF-funded projects, longitudinal data from the Virginia Tech College of Engineering, current data in engineering courses, and freshly collected data from online outlets. These represent important areas of the broader ecosystem that supports how we form future engineers. We will test the system for thematic clusters, employing similar metrics as in Phase 1 to identify potential inconsistencies in how different datasets are handled. We will specifically look for homogeneity of texts within a cluster and shared semantic meaning. We will also update the original system designs in the event of systematic differences (e.g., longer texts require a different system configuration). For Phase 3, we will study how it can affect human performance. Since we anticipate significant improvements in human efficiency and consistency, it is important to conduct analyses that can accurately assess the veracity of that proposition. These studies will assess the HITL aspect of this process since many relevant applications of the system will require additional interpretation of the raw output. To accomplish this, we will collect data on differences in human performance when analyzing 1,500 student responses with and without the system?s assistance. We will look at differences when (a) one person alone codes the data and when (b) a team of three researchers codes the data (i.e., we will have two studies: one person with vs one person without and team with vs team without). We will measure differences in coding (whether different themes emerge), reliability (how consistently similar texts are grouped together), time needed to code the data, and differential treatment of student responses associated with student group characteristics. We will host all code on public repositories and notebooks for easy access, copying, and application by other engineering education researchers and teachers along with any new datasets, where appropriate.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Please report errors in award information by writing to: awardsearch@nsf.gov.