Survey data has been, and continues to be, essential for public and private organizations in order to inform responses and policy choices related to the COVID-19 pandemic, and it is equally important for researchers in understanding the pandemic's impact. In general, surveys tend to use closed-end responses like multiple-choice questions or ratings on a 1-to-5 scale; open-ended questions, eliciting a written response, are used far less. This is true despite the fact that unconstrained text responses can yield rich and nuanced information as well as bottom up information that the survey designers did not know to ask about.

One of the central obstacles to using open-ended questions, though, is that analyzing people's language in survey responses is a costly and labor-intensive enterprise. Generally analysts read all the responses in a slow, systematic process in order to manually identify themes or categories of response. In more rigorous analyses, additional effort is expended to establish that multiple analysts agree on what the categories are and that they make sense.  As an alternative, there are automated ways to identify categories in bodies of text and such methods are highly scalable; however, in practice there is no single accepted way to automate the process, and many survey analysts are unsatisfied with the ability of fully automatic techniques to produce sufficiently high quality results. In addition, there tends to be a sociological gap between the survey research and technological research communities, resulting in automated techniques not being widely adopted.

This project developed new computational methods for addressing limitations of open-ended responses in surveys, and it has applied these methods to support partners conducting and analyzing surveys related to the pandemic. On the technical side, the project focused on topic models, a category of computational methods that can extract human-interpretable categories from large collections of text. One of the project's surprising findings was that, although newer "neural network" topic models had been claimed to improve on the earlier topic models introduced in the early 2000s, flaws in the way that neural models are evaluated cast significant doubt on the validity of those claims. This led to a new approach to computational model evaluation that is much more directly tied to the processes that survey researchers actually follow in practice when they analyze text responses in the traditional, labor-intensive way. Results of careful and comprehensive experimentation found that the earlier "classical" models are actually superior to the newer neural methods when it comes to criteria that survey analysts care about. Building on that finding to address more generally the crucial issue of analyst trust in the results of such computational methods, the project developed TOPCAT (Topic-Oriented Protocol for Content Analysis of Text), a software-enabled, human-centered process designed around the traditional qualitative content analysis process, with the goal of widespread utility for "qual" researchers who analyze open-ended responses and other collections of text.

In addition to its technical aims, the project aimed from the very beginning to use its methods and the knowledge being developed in an ongoing way to help organizations "in the trenches" with their surveys during the pandemic. This included successfully assisting in the analysis of open-ended responses for a number of surveys: one in collaboration with the CDC/National Center for Healthcare Statistics; one  conducted by the Pandemic Crisis Services Response Coalition, a national organization focused on mental health issues; one conducted by the NYU School of Nursing looking at the experiences of front-line healthcare providers; a nationally representative survey conducted by Westat and the Stanford Medical School on the social and economic impact of COVID-19; a large survey by the Global Consortium of Nursing & Midwifery Studies with both English and Spanish responses from the U.S. and Latin America (with further plans to continue working with surveys in multiple languages being conducted in the Caribbean, Europe, the Middle East, Asia, and Africa); and a second nationally representative survey on COVID impact designed and deployed in collaboration with Westat.  Although not directly related to COVID-19, analysis was conducted using TOPCAT with 16,000 responses in a Reddit thread asking formerly suicidal Redditors what had gotten them through the dark times, with two suicide prevention experts providing the necessary subject matter expertise, providing insights that will be valuable to suicidologists and clinicians encountering people in suicidal crisis.

This project supported the training of several graduate students, and the TOPCAT protocol, including detailed analyst instructions and supporting software for the end-to-end process, is being made publicly available.

Last Modified: 08/07/2023
Modified by: Philip Resnik