ABSTRACT

Face-to-face conversation is an important way in which people communicate with each other, but unfortunately there are millions who suffer from disorders that impede normal conversation. This project will explore new real-time communication solutions for people who face speaking challenges, including those with physical or cognitive disabilities, for example by exploiting implicit and explicit contextual input obtained from a person's conversation partner. The goal is to develop technology that improves upon the Augmentative and Alternative Communication (AAC) devices currently available to help people speak faster and more fluidly. The project will expand the resources for research into conversational interactive systems, the deliverables to include a probabilistic text entry toolkit, AAC user interfaces, and an augmented reality conversation assistant. Project outcomes will include flexible, robust, and data-driven methods that extend to new use scenarios. To enhance its broader impact, the project will educate the public about AAC via outreach events and by the online community the work will create. The PI will assemble teams of undergraduates to develop the project's software, and he will host a summer youth program on the technology behind text messaging, offering scholarships for women, students with disabilities, and students from underrepresented groups. Funded first-year research opportunities will further help retain undergraduates, particularly women, in computing.

This project will explore the design space of conversational interactive systems, by investigating both systems that improve communication for non-speaking individuals who use AAC devices and systems that enhance communication for speaking individuals who face other conversation-related challenges. Context-sensitive prediction algorithms that use: 1) speech recognition on the conversation partner's turns; 2) the identity of the partner as determined by speaker identification; 3) dialogue state information; and 4) suggestions made by a partner on a mobile device will be considered. User studies will investigate the effectiveness and user acceptance of partner-based predictions. New methodologies will be created for evaluating context-sensitive AAC interfaces. The impact of training AAC language models on data from existing corpora, from simulated AAC users, and from actual AAC users will be compared. This research will expand our knowledge about how to leverage conversational context in augmented reality, and it will curate a public test set contributed by AAC users.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Please report errors in award information by writing to: awardsearch@nsf.gov.