ABSTRACT

Augmented reality (AR) glasses are transforming the workplace by allowing users to manipulate virtual objects in a real environment. From offices to manufacturing to service, AR technology offers far-reaching benefits, but improving how users interact with virtual worlds through AR glasses is essential to this transformation. AR systems must be designed to understand the subtleties and complexities of human communication, which come in a broad combination of modes?gestures, speech, text, gaze, and more. Current AR head-mounted systems cannot process multiple communication modes and they recognize only coarse gestures that are awkward and tiring for users to produce. One approach to reducing fatigue and promoting intuitive interaction is for the system to recognize and process microgestures. In this project, these simple, small finger movements, akin to those needed to control the volume on a car radio, are studied to build foundational knowledge needed to make human-AR interaction systems more intuitive and broadly accessible. Reducing the barriers to interaction with AR technology will allow people to concentrate on their tasks rather than the technology itself. This research impacts fields such as meteorology, finance, healthcare, and scientific visualization, facilitating rich data exploration, and improving data understanding and decision making. Implementing this research in massive open online courses (MOOCs) will benefit students learning independently and in education institutions, allowing access to anyone interested in this topic.

This project advances AR multimodal interaction research by studying intuitive microgesture interaction techniques and additional communication modalities. The proposed research plan for these studies involves inputs combined with speech, a 6 degrees-of-freedom (6DoF) pen, a controller, and midair gestures. Bimanual interaction will also be evaluated. The results of these studies will answer an important question, ?What should microgestures and multimodal interaction look like in AR??, leading to a unified framework of multimodal interactions and the creation of a Multimodal Interaction ToolKit (MITK). Immersive analytics (that is, 3D data visualizations) will be used as a domain for performing system validations. Further, the investigator will evaluate and extend the findings from AR to virtual reality (VR). This research advances the state of the art in AR and VR interaction by: (1) researching and developing microgestures for selection, manipulation, and navigation; (2) designing, developing, and evaluating novel unimodal and multimodal interaction techniques, including virtual widgets; and (3) expanding fundamental knowledge of microgestures and their use in joint tasks. The research team will develop a new multimodal interaction textbook and an open online course on multimodal interaction using immersive analytics.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Please report errors in award information by writing to: awardsearch@nsf.gov.