ABSTRACT

Cameras of all types continue to proliferate in our environments, continuously recording our everyday actions and interactions with surrounding objects, scenes, and other people. This growing capacity to collect visual data has not, however, been matched by a comparable acceleration in the ability of individuals being recorded (by themselves or others) to measure, analyze, and predict from this data as they desire. This project develops a computer vision system that performs tasks of learning from personal visual data, e.g., dietary monitoring via a user's wearable cameras. The project will develop a common 3D geometry representation to encode multiview image streams that can provide an auxiliary spatial constraint to enable semi-supervised learning. This representation will enable many computer vision tasks by leveraging a large or potentially infinite number of unlabeled multiview image streams. The investigator plans to disseminate this research through multiple educational activities, including: (a) K-12 students of under-represented groups through a series of workshops in Summer Technology Camp, Girls' Machine Learning Day Camp, and the Robot Show at the University Minnesota; (b) public demonstration of the research outcomes annually at the Minnesota State Fair; (c) undergraduate research involvement through short course material developments; (d) new curriculum development on the raster multiview algebra that consolidates multiview geometry and visual recognition (deep learning); and (e) dissemination of research findings through tutorial and workshop organization for broader audiences in computer vision.

This research investigates a new raster representation of multiview geometry that offers a tight integration of 3D reconstruction with the training of a visual recognition model. This theory will allow extensive utilization of the unlabeled multiview image streams in two ways: (1) by actively exploring multiview visual data through 3D reconstruction in an unsupervised manner, and (2) by geometrically transferring a probabilistic belief across views for cross-view supervision. The project will generalize this theory to uncalibrated moving cameras through a novel raster bundle adjustment. This research program will lay a computational foundation for two major scientific disciplines: (1) Behavioral science: the raster theory will address many customized visual tasks to characterize microscopic social signals, enabling us to overcome the fundamental limitations of existing approaches in behavioral assessments for at-risk children (such as those with autism spectrum disorder, which have typically been assessed using a subjective and sporadic measure); and (2) Neuroscience: the planned research will enable computational measurement of free-ranging activities of primate subjects.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Please report errors in award information by writing to: awardsearch@nsf.gov.