ABSTRACT

Using statistics to model data often requires making assumptions about what the data represent, including the nature of underlying patterns and errors. For example, an analyst interested in the relationship between wealth and age might remove extremely low or high reported ages from her data as outliers, and choose to model the remaining data using a linear model, which implies that age increases wealth by some constant factor. Her results could imply substantially different conclusions about how wealth and age relate compared to an analysis that made different decisions about which data to include. One way for analysts to account for such sensitivity is by reporting the results of many reasonable analyses given a dataset and questions they want to answer using the data. Unfortunately, existing data analysis and visualization tools offer little support for reasoning about such a ?multiverse analysis.? They provide limited support for comparison of multiple models and visualizations that make different choices, and even less support in helping analysts reason about and express those choices. Further, there are few known ways to effectively convey both uncertainty in the results of a given analysis and uncertainty related to the assumptions made in that analysis. This project?s goal is to improve multiverse analysis: to better understand how analysts currently think about multiverse analysis, and to identify needs, opportunities, and approaches to help analysts use multiverse analyses.

This project addresses these challenges to expressing hard-to-quantify uncertainty related to analysis choices by creating new methods and tools to help analysts define, reason about, and express multiple alternative ways they could analyze their data. The research will focus on two common types of tools analysts use: visual analysis software that makes it easy to plot and compare data, and computational notebooks that allow for more seamless integration of code and narrative commentary. The project team will develop new user interfaces and programming libraries to elicit analysts? knowledge, as well as new visual representations and interaction techniques by which an analyst can compare between alternative models or analysis paths. The project will also produce novel software infrastructure to make conducting and evaluating multiple analyses feasible within existing tools and workflows. Further, the team will develop ways to better communicate multiverse analyses: ways to make multiverse analysis reports shareable, interactive documents that contain both the analysis code and figures as well as narrative context, and empirical results describing how different representations of plausible analyses impact readers? understanding. These research activities will be guided by the results of formative studies with real-world analysts that will address gaps in existing knowledge about the difficulties analysts face in defining and reasoning about alternative models or analysis steps they could have taken. All study results and computational tools will be made freely and publicly available.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Please report errors in award information by writing to: awardsearch@nsf.gov.