Award Abstract # 1617917
RI: Small: Texture2Text: Rich Language-Based Understanding of Textures for Recognition and Synthesis

NSF Org: IIS
Division of Information & Intelligent Systems
Recipient: UNIVERSITY OF MASSACHUSETTS
Initial Amendment Date: June 10, 2016
Latest Amendment Date: September 22, 2016
Award Number: 1617917
Award Instrument: Continuing Grant
Program Manager: Jie Yang
jyang@nsf.gov
 (703)292-4768
IIS
 Division of Information & Intelligent Systems
CSE
 Directorate for Computer and Information Science and Engineering
Start Date: September 1, 2016
End Date: August 31, 2020 (Estimated)
Total Intended Award Amount: $450,000.00
Total Awarded Amount to Date: $450,000.00
Funds Obligated to Date: FY 2016 = $450,000.00
History of Investigator:
  • Subhransu Maji (Principal Investigator)
    smaji@cs.umass.edu
Recipient Sponsored Research Office: University of Massachusetts Amherst
101 COMMONWEALTH AVE
AMHERST
MA  US  01003-9252
(413)545-0698
Sponsor Congressional District: 02
Primary Place of Performance: University of Massachusetts Amherst
Research Admin Bldg., 70 Butterfield Terrace
Amherst
MA  US  01003-9242
Primary Place of Performance
Congressional District:
02
Unique Entity Identifier (UEI): VGJHK59NMPK9
Parent UEI: VGJHK59NMPK9
NSF Program(s): Robust Intelligence,
Unallocated Program Costs
Primary Program Source: 01001617DB NSF RESEARCH & RELATED ACTIVIT
Program Reference Code(s): 7495, 7923
Program Element Code(s): 749500, 919900
Award Agency Code: 4900
Fund Agency Code: 4900
Assistance Listing Number(s): 47.070

ABSTRACT

This project develops techniques at the interface of vision and natural language to understand and synthesize textures. For example, given a texture the project develops techniques that provide a description of the pattern (e.g., "the surface is slippery", "red polka-dots on a white background"). Techniques for semantic understanding of textures benefit a large number of applications ranging from robotics where understanding material properties of surfaces is key to interaction, to analysis of various forms of imagery for meteorology, oceanography, conservation, geology, and forestry. In addition, the project develops techniques that allow modification and synthesis of textures based on natural language descriptions (e.g., "make the wallpaper more zig-zagged", "create a honeycombed pattern"), enabling new human-centric tools for creating textures. In addition to the numerous applications enabled by this project, the broader impacts of the work include: the development of new benchmarks and software for computer vision and language communities, undergraduate research and outreach, and collaboration with researchers and citizen scientists in areas of conservation.

This research maps visual textures to natural language descriptions and vice versa. The research advances computer vision by providing texture representations that are robust to realistic imaging conditions, clutter, and occlusions in natural scenes; content retrieval by providing new ways to search and retrieve textures using descriptions; and image manipulation by providing new ways to create and modify textures using descriptions. The main technical contributions of the project are: (1) principled architectures that combine aspects of texture models with deep learning to enable end-to-end learning of texture representations; (2) techniques for understanding the properties of these representations through visualizations; (3) a large-scale benchmark to evaluate techniques for language-based texture understanding; (4) new models for texture captioning; (5) applications of texture representations for fine-grained recognition and semantic segmentation; and (6) techniques for retrieving and creating textures using natural language descriptions.

PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:  When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, Subhransu Maji: "PhraseCut: Language-Based Image Segmentation in the Wild" Computer Vision and Pattern Recognition (CVPR) , 2020 10.1109/CVPR42600.2020.01023
Chenyun Wu, Mikayla Timm, Subhransu Maji "Describing Textures Using Natural Language" European Conference on Computer Vision (ECCV) , 2020 https://doi.org/10.1007/978-3-030-58452-8_4
Gadelha, Matheus and Wang, Rui and Maji, Subhransu "Multiresolution Tree Networks for 3D Point Cloud Processing" European Conference on Computer Vision , v.11211 , 2018 Citation Details
Sharma, Gopal and Goyal, Rishabh and Liu, Difan and Kalogerakis, Evangelos and Maji, Subhransu "Neural Shape Parsers for Constructive Solid Geometry" IEEE Transactions on Pattern Analysis and Machine Intelligence , 2020 https://doi.org/10.1109/TPAMI.2020.3044749 Citation Details
Tsung-Yu Lin and Aruni RoyChowdhury and Subhransu Maji "{Bilinear Convolutional Neural Networks for Fine-grained Visual Recognition}" {IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)} , 2017
Tsung-Yu Lin, Aruni RoyChowdhury, Subhransu Maji "Bilinear Convolutional Neural Networks for Fine-grained Visual Recognition" IEEE Transactions of Pattern Analysis and Machine Intelligence (PAMI) , v.PP , 2017 10.1109/TPAMI.2017.2723400
Wu, Chenyun and Lin, Zhe and Cohen, Scott and Bui, Trung and Maji, Subhransu "PhraseCut: Language-Based Image Segmentation in the Wild" 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020 https://doi.org/10.1109/CVPR42600.2020.01023 Citation Details
Yang Zhou and Zhan Xu and Chris Landreth and Evangelos Kalogerakis and Subhransu Maji and Karan Singh "{VisemeNet: Audio-Driven Animator-Centric Speech Animation}" ACM Transactions on Graphics , v.37 , 2018

PROJECT OUTCOMES REPORT

Disclaimer

This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.

The project has contributed to techniques for representing texture in natural images and describing them using natural language. This allows us to 1) infer properties of objects in images such as their materials and identity; 2) retrieve texture from image collections using natural language for applications in graphics and commerce; and 3) explain the behavior of deep networks for image classification as texture cues are often used to learn the classification rule.

 
On the technical side we have contributed novel architectures for detailed image understanding. These models unify decades of work on classical texture representations with recent advances in visual recognition based on deep learning. These models have been found to be more robust to changes in the distribution of images at testing time, and better transferable to fine-grained natural domains. For example, our early work on bilinear convolutional networks improved the accuracy of models at recognizing animal and plant species and reduced the reliance on part annotations which allowed them to be deployed at many more domains. Variants of these techniques have been used to analyze citizen-science data to inform projects in ecology and biology.
 
We have also contributed datasets and benchmarks that allow systematic evaluation and advancement of methods for connecting language and vision for texture understanding. The source code, models for automatic analysis, and datasets have been publicly released to the scientific community, along with the supporting publications. The proposal supported the PI and who mentored several PhD students, MS students, and undergraduates on research projects.

Last Modified: 12/28/2020
Modified by: Subhransu Maji

Please report errors in award information by writing to: awardsearch@nsf.gov.

Print this page

Back to Top of page