| NSF Org: |
ITE Innovation and Technology Ecosystems |
| Recipient: |
|
| Initial Amendment Date: | September 4, 2020 |
| Latest Amendment Date: | October 14, 2020 |
| Award Number: | 2040638 |
| Award Instrument: | Standard Grant |
| Program Manager: |
Mike Pozmantier
ITE Innovation and Technology Ecosystems TIP Directorate for Technology, Innovation, and Partnerships |
| Start Date: | September 15, 2020 |
| End Date: | May 31, 2022 (Estimated) |
| Total Intended Award Amount: | $960,000.00 |
| Total Awarded Amount to Date: | $960,000.00 |
| Funds Obligated to Date: |
|
| History of Investigator: |
|
| Recipient Sponsored Research Office: |
3 RUTGERS PLZ NEW BRUNSWICK NJ US 08901-8559 (848)932-0150 |
| Sponsor Congressional District: |
|
| Primary Place of Performance: |
33 Knightsbridge Road Pisacataway NJ US 08854-8019 |
| Primary Place of
Performance Congressional District: |
|
| Unique Entity Identifier (UEI): |
|
| Parent UEI: |
|
| NSF Program(s): | Convergence Accelerator Resrch |
| Primary Program Source: |
|
| Program Reference Code(s): | |
| Program Element Code(s): |
|
| Award Agency Code: | 4900 |
| Fund Agency Code: | 4900 |
| Assistance Listing Number(s): | 47.084 |
ABSTRACT
The NSF Convergence Accelerator supports use-inspired, team-based, multidisciplinary efforts that address challenges of national importance and will produce deliverables of value to society in the near future. American Sign Language (ASL) is the third most studies ?foreign? language in the United States. This project is building 4-dimensional face-tracking algorithms that could be used to separate facial geometry from facial movement and expression. The work supports an application for teaching American Sign Language (ASL) to ASL-learners, an application for anonymizing the signer when privacy is a concern, and research into the role of facial expressions in both sign and spoken language. The privacy preserving application being developed by this project will enable ASL speakers to have private conversations about sensitive topics they would otherwise.
This team of linguists, computer scientists, deaf and hearing experts on ASL, and industry partners will address research and societal challenges through three types of deliverables targeted to diverse user and research communities: 1) Modifications and extension of AI methods and publicly shared ASL data and tools to encompass spoken language. Although facial expressions and head gestures, essential to the grammar of signed languages, also play an important role in speech, this is not well understood because resources of the kind developed by this project have not been available. New data and analyses will open the door to comparative study of the role of facial expressions across modalities, and the role of facial expressions in signed language vs. spoken language. Shared raw data, analyses, and visualizations will open up new avenues for linguistic and computer science research into the role of spatiotemporal synchronization of nonmanual expressions in conjunction with speech and signing. 2) An application to help ASL learners produce facial expressions and head gestures to convey grammatical information in signed languages; and 3) Development of a tool for real-time anonymization of ASL videos to preserve grammatical information expressed non-manually, while de-identifying the signer.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH
Note:
When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external
site maintained by the publisher. Some full text articles may not yet be available without a
charge during the embargo (administrative interval).
Some links on this page may take you to non-federal websites. Their policies may differ from
this site.
PROJECT OUTCOMES REPORT
Disclaimer
This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.
We have been developing sustainable robust AI methods for facial analytics, potentially applicable across domains but targeted here to new applications that address important problems related to use of facial expressions and head gestures in natural language. In sign language, critical linguistic information of many kinds is conveyed exclusively by facial expressions and head gestures. The fact that the face carries essential linguistic information poses major challenges for Deaf signers and for students of ASL as a non-native language.
Problem #1: The >500,000 US ASL signers have no way to communicate anonymously through videos in their native language, e.g., about sensitive topics (such as medical issues). This is perceived to be a significant problem by the Deaf community. It also means, for example, that signed submissions to scholarly journals cannot be reviewed anonymously.
Problem #2: 2nd-language learners of ASL (the 3rd most studied "foreign" language, with US college enrollments >107,000 as of 2016) have difficulty learning to produce these essential expressions, in part because they do not see their own face when signing.
In spoken language, these expressions also play an important role, but they function differently.
Problem #3: The role of co-speech gestures, including facial expressions and head movements, is not well understood because of inadequacies in current analytic tools. This has held back applications that rely on such correlations, such as the development of realistic speaking avatars and robots; technology for the elderly and those with disabilities; and detection of, e.g., deception and intent.
To address these problems, we worked closely with the prospective users of these tools and conducted user studies to identify and respond to their needs and preferences, in order to create prototypes for applications (1) to enable ASL signers to share videos anonymously by disguising their face without loss of linguistic information; (2) to help ASL learners produce these expressions correctly; and (3) to help speech scientists study co-speech gestures.
Our AI approach to analysis of facial expressions and head gestures -- combining 3D modeling, Machine Learning (ML), and linguistic knowledge derived from our annotated video corpora -- overcomes limitations of prior research. It is distinctive in its ability to capture subtle facial expressions, even with significant head rotations.
Beyond the benefits that would accrue to the three target populations, this research has other potential applications, as well, e.g. for sanitizing other data involving video of human faces, medical applications, security, driving safety, and the arts.
In conjunction with the research conducted for this project, we have expanded and enhanced our collection of publicly shared, linguistically annotated ASL video data and refined the software we distribute through an MIT license for linguistic annotation of visual language data, SignStream(R). Data are available from https://dai.cs.rutgers.edu/dai/s/dai and https://dai.cs.rutgers.edu/dai/s/signbank. The software is available from http://www.bu.edu/asllrp/SignStream/3/. The data and software shared freely through our websites will also contribute to linguistic and computational research on sign language. This is especially valuable for students and has been used for many research projects in the US and abroad.
Last Modified: 08/29/2022
Modified by: Carol J Neidle
Please report errors in award information by writing to: awardsearch@nsf.gov.
