ABSTRACT

Developing methods to do effective, usable design for mobile applications is an important problem, affecting domains ranging from healthcare to finance. Current methods for app design are largely based on either individual designer expertise and intuition, leading to variable results, or else require a large number of active users and engineering resources that are typically only available to large companies. This project's goal is to solve the research challenges involved in automatically capturing and aggregating design features and user interactions across the large number of existing mobile applications already available for download and develop a platform allowing designers to learn from the choices and experiences other designers have made. The platform will allow designers to find relevant third-party apps that have relevant design requirements and features, define experiments involving those design choices, and run the experiments by leveraging the existing user base of those apps. This will help individual designers explore options more cheaply than developing their own prototypes while providing data-driven arguments to support design decisions and communicate with other members of their project teams. The platform could also lead to tools to accumulate collective design knowledge, identifying emerging trends as well as best practices, useful for both practicing designers and as an educational resource for existing courses on web and app design.

Developing the platform will require a number of technical advances. The first involves developing scalable systems for capturing design features and interaction data from large numbers of mobile apps and combining them into representations useful for data mining. To do this, the team will expand the existing ERICA platform for black-box design capture in a single application, developing it into a background monitoring app that captures interaction and design data for any app the user gives permission for, while detecting and obscuring personally identifying information. The monitoring app will be deployed primarily via long-term crowdworkers who are paid to install it and participate in designer-defined experiments. The second main advance is creating functional semantic embeddings for interface components based on the collected data that define common features, app states, and functions. To do this the team will capture visual, textual, structural, and interactive information about each interface element, then use multimodal embeddings of the data to first classify individual interface elements, then use those labels to identify semantics of app screens and interaction flows between elements and screens. The third advance involves developing a series of tools to use the data and semantic embeddings. This includes working with designers to develop query techniques and views over sets of applications to find relevant applications and flows; creating visualizations of user behavior based on Sankey flow diagrams that allow designers to make sense of user behavior across applications; and designing app analytics tools to support meta-analysis across applications and experiments.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Please report errors in award information by writing to: awardsearch@nsf.gov.