ABSTRACT

For robots to act as ubiquitous assistants in daily life, they must regularly contend with environments involving many objects and objects built of many constituent parts. Current robotics research focuses on providing solutions to isolated manipulation tasks, developing specialized representations that do not readily work across tasks. This project seeks to enable robots to learn to represent and understand the world from multiple sensors, across many manipulation tasks. Specifically, the project will examine tasks in heavily cluttered environments that require multiple distinct picking and placing actions. This project will develop autonomous manipulation methods suitable for use in robotic assistants. Assistive robots stand to make a substantial impact in increasing the quality of life of older adults and persons with certain degenerative diseases. These methods also apply to manipulation in natural or man-made disasters areas, where explicit object models are not available. The tools developed in this project can also improve robot perception, grasping, and multi-step manipulation skills for manufacturing.

With their ability to learn powerful representations from raw perceptual data, deep neural networks provide the most promising framework to approach key perceptual and reasoning challenges underlying autonomous robot manipulation. Despite? ?their success, existing approaches scale poorly to the diverse set of scenarios autonomous robots will handle in natural environments. These current limitations of neural networks arise from being trained on isolated tasks, use of different architectures for different problems, and inability to scale to complex scenes containing a varying or large number of objects. This project hypothesizes that graph neural networks provide a powerful framework that can encode multiple sensor streams over time to provide robots with rich and scalable representations for multi-object and multi-task perception and manipulation. This project examines a number of extensions to graph neural networks in order to address current limitations for their use in autonomous manipulation. Furthermore this project examines novel ways of leveraging learned graph neural networks for manipulation planning and control in clutter and for multi-step, multi-object manipulation tasks. In order to train these large-scale graph net representations this project will use extremely large scale, physically accurate, photo-realistic simulation. All perceptual and behavior generation techniques developed in this project will be experimentally validated on a set of challenging real-world manipulation tasks.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Please report errors in award information by writing to: awardsearch@nsf.gov.