ABSTRACT

As robots become more capable and ubiquitous, they are increasingly moving into traditionally human-centric environments such as as health care, education, and elder care. As robots engage in tasks as diverse as helping with household work, deploying medication, and tutoring students, it becomes increasingly critical for them to interact naturally with the people around them. Key to this progress is the development of robots that acquire an understanding of goals and objects from natural communications with a diverse set of end users. One way to address this is using language to build systems that learn from people they are interacting with. Algorithms and systems developed in this project will allow robots to learn about the world around them from linguistic interactions. This research will focus on understanding spoken language about the physical world from diverse groups of people, resulting in systems that are more able to robustly handle a wide variety of real-world interactions. Ultimately, the project will increase the usability and fairness of robots deployed in human spaces.

This CAREER project will study how robots can learn about noisy, unpredictable human environments from spoken language combined with perception, using context derived from sensors to constrain the learning problem. Grounded language refers to language that occurs in and refers to the physical world in which robots operate. Human interactions are fundamentally contextual: when learning about the world, we focus learning by considering not only direct communication but also the context of that interaction. For much existing work on learning to understand physically situated language, text is the primary interlingua, and context is considered relatively narrowly. Additionally, reliance on pre-existing large datasets has begun to raise questions about bias and inclusivity in learning-driven technologies. To address these limitations, this work will focus on learning semantics directly from perceptual inputs combined with speech from diverse sources. The goal is to develop learning infrastructure, algorithms, and approaches to enable robots to learn to understand task instructions and object descriptions from spoken communication with end users. The project will develop new methods of efficiently learning from multi-modal data inputs, with the ultimate goal of enabling robots to efficiently and naturally learn about their world and the tasks they should perform.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Please report errors in award information by writing to: awardsearch@nsf.gov.