ABSTRACT

This project will address the problem of Visual Recognition with Knowledge (VR-K): a challenging Artificial Intelligence task to enable a seeing machine to identify unknown visible concepts from previous encounters (annotated data samples) and knowledge (other contextual information). For example, consider such a system that has never encountered a zebra, but which has previous visual encounters with "horses" and "black and white striped" patterns. Incorporating the linguistic input that, "A zebra is a horse-like animal with a black and white striped appearance", the machine's task is to formulate a new recognizer for the visual concept "zebra" and to recognize this new concept later. A system that integrates visual and linguistic information in this way can provide the basis for robust personal mobile applications or service robots, such as visual assistants to the vision-impaired, and voice-enable agents for elder care.

Conventional supervised learning techniques have been perfected to perform increasingly well on narrow performance tasks. To enable satisfactory performance in service robots and mobile multimedia applications, this research will integrate background and commonsense knowledge models to enable higher level reasoning together with such high-performance recognizers. This project will develop the VR-K framework focused on enabling more generalizable computer vision algorithms through integration with natural language understanding and grounding in knowledge-based reasoning. The research program will include 1) developing efficient probabilistic reasoning engines to construct recognition models of unseen concepts (object and attribute) without new annotation through probabilistic semantic parsing; 2) setting up new large-scale visual challenges and testbeds as the basis for rigorous performance evaluation of visual recognition with knowledge models and ablation analysis; and 3) prototyping the proposed framework on service robots and mobile devices for evaluation of the proposed framework's performance in complex real-world applications over a variety of user studies. The project will include education and outreach activities advancing AI in undergraduate research, diversity enhancement, Entrepreneurial Mindset (EM) education, and K-12 classrooms, and will include workshops to introduce AI and deep learning to professionals in non-CS professions such as medical research and pathology.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Please report errors in award information by writing to: awardsearch@nsf.gov.