This project explores (semi-)automatic ways to create "semantically discriminative" mid-level cues for visual object categorization, by introducing external knowledge of object properties into the statistical learning procedures that learn to distinguish them. In particular, the PIs investigate four key ideas: (1) exploiting taxonomies over object categories to inform feature selection algorithms such that they home in on the most abstract description for a given granularity of label predictions; (2) leveraging inter-object relationships conveyed by the same taxonomies to guide context learning, so that it captures more than simple data-driven co-occurrences; (3) exploring the utility of visual attributes drawn from natural language, both as auxiliary learning problems to bias models for object categorization, as well as ordinal properties that must be teased out using non-traditional human supervision strategies; (4) mining attributes that are both distinctive and human-nameable, moving beyond manually constructed semantics.

The project entails original contributions in both computer vision and machine learning, and is an integral step towards semantically-grounded object categorization. Whereas mainstream approaches reduce human knowledge to mere category labels on exemplars, this work leverages semantically rich knowledge more deeply and earlier in the learning pipeline. The approach results in vision systems that are less prone to overfit incidental visual patterns, and representations that are readily extendible to novel visual learning tasks. Beyond the research community, the work has broader impact through inter-disciplinary training of graduate and undergraduate students, and outreach to pre-college educators and students through workshops and summer camps encouraging young students to pursue science and engineering.

During the lifespan of this project, the support from NSF has generated significant outcomes in several aspects, summarized as follows: (1) interdisciplinary research between machine learning and computer vision in the areas of learning attributes as mid-level representations, addressing domain adaptations, a very challenging problem in computer vision, zero-shot learning for building large-scale and robust computer vision systems that can operate in the wild, preliminary research into visual question and answering for artificial general intelligence; (2) applying reserarch results to real-world problems such as collaboration with government labs and industry research teams; (3) educating next-generation academics including graduating two students who have taken academic teaching and research jobs; (4) training undegraduate and graduate students. 

The project had also catalyzed the collaborative research between us and the research lab led by Prof. Kristen Grauman at U of Texas (Austin). Building on top of the collaborative research conducted in this project, the two labs have formed broader collaborations and have since taken another joint research project under NSF support.  

Last Modified: 04/20/2018
Modified by: Fei Sha