This project developed computationally-infused methodologies for qualitative analysis of video data of STEM classrooms. Specifically, we developed (1) computational techniques to extract meaningful features from video and audio data; and (2) methodological frameworks and approaches that integrated computational tools in ways that supported the goals of qualitative analysis. Our case examples showcasing our use of these tools and approaches have also contributed to our understanding of learning in the context of mathematical problem solving.

Intellectual Merit. Our project’s commitment to centering the goals and values of qualitative research contrasts with predominant uses of computational tools to automate analyses, such as training a computer to code large amounts of qualitative data. While automating analysis might make qualitative research more efficient, we argue that such efficiency is in direct opposition to the broader epistemologies driving qualitative research. Accordingly, instead of using computational tools to code more data more quickly, we have instead used computational tools to construct increasingly rich descriptions of (large amounts of) complex data in ways that allow us to understand the particularities, nuances, and complex processes evident within these data. The goals of such uses of computational tools are to develop new insights; to trouble existing understandings; and to generate new theoretical connections. To this end, we have shown how “errors” in computational detection, differences in human vs. computer noticing, and “outlier” moments of particular sets of audio and visual features have been especially important components of our methodological approaches. We position these approaches as explicitly feminist and anti-capitalist in that they expand beyond automation and seek to resist making data analysis more efficient at the expense of particularity, nuance, richness, and complexity. 

Technical developments. Building from existing open-source tools for extraction of audio features (OpenSmile) and visual aspects of human skeletal movement (OpenPose), we built custom tools for processing audiovisual data recorded in high school mathematics classrooms. These tools include an algorithm to track unique individuals across frames of video data within OpenPose (Hur & Bosch, 2022); bespoke methods for feature engineering (Hur, Machaka, Krist, & Bosch, 2023); and protocols for merging and syncing extracted audio feature data from OpenSmile (Palaguachi, D’Angelo, Dyer, & Machaka, 2024; Cox, Dyer, & Krist, forthcoming). 

Methodological developments. We developed a suite of methodological frameworks that provide epistemological and ideological guidance for integrating computational tools to augment qualitative analyses. These frameworks are not uniform; instead, they highlight the breadth of possibilities for careful and creative uses of computational methods in qualitative research. First, and most broadly, the DEFT framework (Distributing Epistemic Functions and Tasks; Kubsch, Krist, & Rosenberg, 2023) challenges researchers to consider the alignment between the goals of their research and the role of the computational tools they choose to use as part of an integrated human-computational system. Second, the CADAs framework (Computationally Assisted Descriptive Approaches; D’Angelo, Krist, & Dyer, forthcoming) provides a rationale and guidance for using audio and visual data as additional layers of rich description within analysis, making a dataset more complex and nuanced. Third, two papers present empirical examples of variations on Laura Nelson’s (2020) computational grounded theory (CGT) method, “remixing” the pattern exploration and data interpretation phases of CGT (Hur, Palaguachi, Machaka, Dyer, D’Angelo, & Bosch, forthcoming; Krist, Dyer, & Hall, forthcoming). Finally, in a forthcoming manuscript and 2025 AERA Presidential Session (Krist, Dyer, Rosenberg, & Cox), we articulate the ideological tensions we encountered throughout the project. Namely, we note that most computational tools and their associated metrics for rigor, reliability, accuracy, etc., have been developed with goals of automation and efficiency in mind. When we intentionally sought to subvert those goals, we were faced with new tensions and sets of decisions around when, why, and how to deploy various computational tools. We discuss those dilemmas and provide guidance for others as they grapple with similar tensions and decisions.

Broader Impacts. Our work has been disseminated broadly through a range of venues, including 8 empirical journal articles; 12 terminal conference proceedings; 14 conference presentations; 2 invited keynote presentations; 4 workshops; and an open-access textbook on integrating machine learning in science education research co-edited by PI Krist and including 5 chapters contributed by project team members. Our project website includes all workshop materials and a link to our github repository.

Our frameworks hold broad applicability beyond the integration of computational tools focused on audio and visual features of data. For example, ChatGPT and generative AI rapidly emerged and gained popularity midway through the life of the project. We have continually responded to the AI “boom” with critical optimism, presenting our approaches and frameworks to challenge the field to proceed with caution (or to avoid altogether) when attempting automation in research. Our responses demonstrate our frameworks’ relevance and applicability to developing methodologies that infuse a broad range of computational tools into qualitative research.

Last Modified: 01/02/2025
Modified by: Christina Krist