
NSF Org: |
IIS Division of Information & Intelligent Systems |
Recipient: |
|
Initial Amendment Date: | June 10, 2016 |
Latest Amendment Date: | September 22, 2016 |
Award Number: | 1617917 |
Award Instrument: | Continuing Grant |
Program Manager: |
Jie Yang
jyang@nsf.gov (703)292-4768 IIS Division of Information & Intelligent Systems CSE Directorate for Computer and Information Science and Engineering |
Start Date: | September 1, 2016 |
End Date: | August 31, 2020 (Estimated) |
Total Intended Award Amount: | $450,000.00 |
Total Awarded Amount to Date: | $450,000.00 |
Funds Obligated to Date: |
|
History of Investigator: |
|
Recipient Sponsored Research Office: |
101 COMMONWEALTH AVE AMHERST MA US 01003-9252 (413)545-0698 |
Sponsor Congressional District: |
|
Primary Place of Performance: |
Research Admin Bldg., 70 Butterfield Terrace Amherst MA US 01003-9242 |
Primary Place of
Performance Congressional District: |
|
Unique Entity Identifier (UEI): |
|
Parent UEI: |
|
NSF Program(s): |
Robust Intelligence, Unallocated Program Costs |
Primary Program Source: |
|
Program Reference Code(s): |
|
Program Element Code(s): |
|
Award Agency Code: | 4900 |
Fund Agency Code: | 4900 |
Assistance Listing Number(s): | 47.070 |
ABSTRACT
This project develops techniques at the interface of vision and natural language to understand and synthesize textures. For example, given a texture the project develops techniques that provide a description of the pattern (e.g., "the surface is slippery", "red polka-dots on a white background"). Techniques for semantic understanding of textures benefit a large number of applications ranging from robotics where understanding material properties of surfaces is key to interaction, to analysis of various forms of imagery for meteorology, oceanography, conservation, geology, and forestry. In addition, the project develops techniques that allow modification and synthesis of textures based on natural language descriptions (e.g., "make the wallpaper more zig-zagged", "create a honeycombed pattern"), enabling new human-centric tools for creating textures. In addition to the numerous applications enabled by this project, the broader impacts of the work include: the development of new benchmarks and software for computer vision and language communities, undergraduate research and outreach, and collaboration with researchers and citizen scientists in areas of conservation.
This research maps visual textures to natural language descriptions and vice versa. The research advances computer vision by providing texture representations that are robust to realistic imaging conditions, clutter, and occlusions in natural scenes; content retrieval by providing new ways to search and retrieve textures using descriptions; and image manipulation by providing new ways to create and modify textures using descriptions. The main technical contributions of the project are: (1) principled architectures that combine aspects of texture models with deep learning to enable end-to-end learning of texture representations; (2) techniques for understanding the properties of these representations through visualizations; (3) a large-scale benchmark to evaluate techniques for language-based texture understanding; (4) new models for texture captioning; (5) applications of texture representations for fine-grained recognition and semantic segmentation; and (6) techniques for retrieving and creating textures using natural language descriptions.
PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH
Note:
When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external
site maintained by the publisher. Some full text articles may not yet be available without a
charge during the embargo (administrative interval).
Some links on this page may take you to non-federal websites. Their policies may differ from
this site.
PROJECT OUTCOMES REPORT
Disclaimer
This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.
The project has contributed to techniques for representing texture in natural images and describing them using natural language. This allows us to 1) infer properties of objects in images such as their materials and identity; 2) retrieve texture from image collections using natural language for applications in graphics and commerce; and 3) explain the behavior of deep networks for image classification as texture cues are often used to learn the classification rule.
Last Modified: 12/28/2020
Modified by: Subhransu Maji
Please report errors in award information by writing to: awardsearch@nsf.gov.