In human-robot communication, robots need to ground human language to their own representations of perception and action. However, although humans and robots are co-present, their representations of the shared world and joint tasks are significantly mismatched due to disparities of their knowledge and perceptual capabilities. Such disparities make language grounding rather difficult. To address this issue, this project has conducted a systematic investigation on the role of collaboration in language communication between humans and robots and developed several collaborative models for grounded language processing.

Our studies have shown that when robots have limited perceptual capabilities, grounding language to perception often cannot be succeeded by one attempt. It is achieved by a collaborative process between humans and robots through multiple episodes. To establish a shared perceptual basis, humans and robots will need to make extra efforts to collaborate with each other and strive for a common ground for language grounding. Based on these observations, this project has developed several collaborative models using graph-based approaches, where a language graph captures collaborative linguistic discourse and a vision graph represents the perceived environment. Language grounding becomes a problem of finding the best match between the language graph and the vision graph that maximizes the overall compatibility between the two graphs. Our empirical results have shown that, graphs can capture rich semantic relations (e.g., spatial relations, group relations) between linguistic expressions as collaborative discourse unfolds. By taking advantage of these relations, the graph-based approaches can compensate for visual recognition errors and mitigate perceptual differences between humans and robots. Robots also need to have the ability to generate linguistic expressions to refer to objects in the environment. Our experiments have shown that, traditional approaches that generate one single description often fail as humans cannot identify the corresponding target objects due to mismatched representations. To address this problem,  this project has developed collaborative referring expression generation models that generate smaller episodes one at a time based on human feedback to mediate perceptual differences. These collaborative models allow humans to identify target objects with a significantly higher accuracy.

This project has also explored connections between language and robotic actions. Most robotic arms are programmed with primitive operations such as move to, open gripper, and close gripper. To perform an action, a discrete controller is often first applied to find a sequence of primitive operations. These primitive operations are then passed to a continuous planner and translated into trajectories of arm motors. Thus a critical question is how to connect language commands with the corresponding sequence of primitive operations. To address this question, this project has focused on the representation of grounded verb semantics for concrete result action verbs (i.e., those which denote changes to the world) and the acquisition of such representation through human-robot communication. Humans can teach robots new actions through step-by-step language instructions based on known actions and operations. At the end of teaching, robots capture the change of world states as they experience during teaching and represent the corresponding new verb with the expected goal state of the world. This state-based representation allows a planner to automatically compute a sequence of low-level primitive operations when the learned verbs are applied in new situations. To handle uncertainties of the environment, this project has also developed collaborative approaches that allow the robot to ask intelligent questions and acquire more reliable grounded representations. 

Last Modified: 01/29/2018
Modified by: Joyce Y Chai