| NSF Org: |
CNS Division Of Computer and Network Systems |
| Recipient: |
|
| Initial Amendment Date: | July 27, 2011 |
| Latest Amendment Date: | June 14, 2016 |
| Award Number: | 1059218 |
| Award Instrument: | Standard Grant |
| Program Manager: |
Ephraim Glinert
CNS Division Of Computer and Network Systems CSE Directorate for Computer and Information Science and Engineering |
| Start Date: | August 1, 2011 |
| End Date: | July 31, 2017 (Estimated) |
| Total Intended Award Amount: | $368,205.00 |
| Total Awarded Amount to Date: | $368,205.00 |
| Funds Obligated to Date: |
|
| History of Investigator: |
|
| Recipient Sponsored Research Office: |
1 SILBER WAY BOSTON MA US 02215-1703 (617)353-4365 |
| Sponsor Congressional District: |
|
| Primary Place of Performance: |
1 SILBER WAY BOSTON MA US 02215-1703 |
| Primary Place of
Performance Congressional District: |
|
| Unique Entity Identifier (UEI): |
|
| Parent UEI: |
|
| NSF Program(s): | CCRI-CISE Cmnty Rsrch Infrstrc |
| Primary Program Source: |
|
| Program Reference Code(s): |
|
| Program Element Code(s): |
|
| Award Agency Code: | 4900 |
| Fund Agency Code: | 4900 |
| Assistance Listing Number(s): | 47.070 |
ABSTRACT
The goal of this project is to create a linguistically annotated, publicly available, and easily searchable corpus of video from American Sign Language (ASL). This will constitute an important piece of infrastructure, enabling new kinds of research in both linguistics and vision-based recognition of ASL. In addition, a key goal is to make this corpus easily accessible to the broader ASL community, including users and learners of ASL. As a result of our long-term efforts, we have an extensive collection of linguistically annotated video data from native signers of ASL. However, the potential value of these corpora has been largely untapped, notwithstanding their extensive and productive use by our team and others. Existing limitations in our hardware and software infrastructure make it cumbersome to search and identify data of interest, and to share data among our institutions and with other researchers. In this project, we propose hardware and software innovations that will constitute a major qualitative upgrade in the organization, searchability, and public availability of the existing (and expanding) corpus.
The enhancement and improved Web-accessibility of these corpora will be invaluable for linguistic research, enabling new kinds of discoveries and the testing of hypotheses that would otherwise have be difficult to investigate. On the computer vision side, the proposed new annotations will provide an extensive public dataset for training and benchmarking a variety of computer vision algorithms. This will facilitate research and expedite progress in gesture recognition, hand pose estimation, human tracking, and large vocabulary, and continuous ASL recognition. Furthermore, this dataset will be useful as training and benchmarking data for algorithms in the broader areas of computer vision, machine learning, and similarity-based indexing.
The advances in linguistic knowledge about ASL and in computer-based ASL recognition that will be accelerated by the availability of resources of the kind proposed here will contribute to development of technologies for education and universal access. For example, tools for searching collections of ASL video for occurrences of specific signs, or converting ASL signing to English, are still far from attaining the level of functionality and usability to which users are accustomed for spoken/written languages. Our corpora will enable research that aims to bring such vision-based ASL recognition applications closer to reality. Moreover, these resources will afford important opportunities to individuals who would not otherwise be in a position to conduct such research (e.g., for lack of access to native ASL signers or high-quality synchronized video equipment, or lack of resources/expertise to carry out extensive linguistic annotations). Making our corpora available online will also allow the broader community of ASL users to access our data directly. Students of ASL will be able to retrieve video showing examples of a specific sign used in actual sentences, or examples of a grammatical construction. ASL instructors and teachers of the Deaf will also have easy access to video examples of lexical items and grammatical constructions as used by a variety of native signers for use in language instruction and evaluation. Thus, the proposed web interface to our data collection will be a useful educational resource for users, teachers, and learners of ASL.
PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH
Note:
When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external
site maintained by the publisher. Some full text articles may not yet be available without a
charge during the embargo (administrative interval).
Some links on this page may take you to non-federal websites. Their policies may differ from
this site.
PROJECT OUTCOMES REPORT
Disclaimer
This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.
The goal of this project was to create a linguistically annotated, publicly available, and easily searchable corpus of high-quality video (including multiple synchronized views) from American Sign Language (ASL).
Intellectual merits
This constitutes an important piece of infrastructure, enabling new kinds of research in both linguistics and computer vision-based recognition of ASL. In addition, a key goal has been to make this corpus easily accessible to the broader ASL community, including users and learners of ASL. In this project, we offer hardware and software innovations that constitute a major qualitative upgrade in the organization, searchability, and public availability of our existing and expanding corpus.
Specific accomplishments:
- We purchased file servers at Boston and Rutgers Universities and established mirroring between the two sites. This enables distribution of our video data, software tools, and linguistic annotations.
- We released a new version of our software for linguistic annotation of video language data, SignStream® 3, in August 2017. This was released with the MIT license; our trademark was also renewed.
- SignStream® was used at Gallaudet and Boston Universities for annotation of a substantial set of ASL data (some of which had been collected in conjunction with other NSF-funded projects: "HCC: Collaborative Research: Medium: Generating Accurate, Understandable Sign Language Animations Based on Analysis of Human Signing" and "III: Medium: Collaborative Research: Linguistically Based ASL Sign Recognition as a Structured Multivariate Learning Problem").
- We designed, developed, and released our Data Access Interface (DAI) as well as an updated version thereof to deal with SignStream® 3 files (DAI 2). This allows for easy browsing, searching, and downloading of our linguistically annotated ASL video corpora.
- The original DAI provides access to (1) our National Center for Sign Language and Gesture Resources (NCSLGR) continuous signing corpus, and (2) our American Sign Language Lexicon Video Dataset (ASLLVD) of citation-form signs (made possible by prior NSF funding, grant # 0705749). DAI 2 provides access to a subset of our new ASLLRP SignStream® 3 corpus, still under development; the remainder will be added as soon as verifications are complete.
- DAI 2 includes a new ASLLRP Sign Bank, built initially off of the ASLLVD. The Sign Bank will be expanded as new data sets are uploaded to DAI 2.
- We have used these resources to advance linguistic and computer science research, thereby leading to a better understanding of how language works in the visual-gestural modality and to new approaches to sign language recognition from video by computer. The results of this research have been shared through publications and conference presentations.
Broader impacts
These software tools and corpora are invaluable resources for linguistic research; we expect they will enable new kinds of discoveries and the testing of hypotheses that would otherwise have been difficult to investigate. For computer vision, the linguistically annotated video corpora offer an extensive public dataset for training and benchmarking a variety of computer vision algorithms. This will facilitate research and expedite progress in gesture recognition, hand pose estimation, human tracking, and large vocabulary, and continuous ASL recognition. Furthermore, these datasets will be useful as training and benchmarking data for algorithms in the broader areas of computer vision, machine learning, and similarity-based indexing.
Already our data sets are being used by researchers, educators, and students around the world for linguistic and computer science research and doctoral training. Our work has also influenced the development of corpora for other signed languages (e.g., Arabic Sign Language, Italian Sign Language, and Yiddish Sign Language).
These materials will also be invaluable for those teaching/studying ASL and Deaf culture, ASL literature, and interpreting. Making our corpora available online will also allow the broader community of ASL users to access our data directly, and to analyze new data with the software tools we are sharing. Students of ASL will be able to retrieve video examples of a specific sign used in actual sentences, or examples of a grammatical construction. ASL instructors and teachers of the Deaf will also have easy access to video examples of lexical items and grammatical constructions as used by a variety of native signers for use in language instruction and evaluation. Thus, the new web interfaces to our data collections will be a useful educational resource for users, teachers, and learners of ASL. Moreover, the system will have educational benefits for deaf students, in helping them to learn English vocabulary and to connect ASL signs to English words.
The research made possible by these resources holds great promise for leading to technologies that will benefit the deaf community. These include tools for language learning, mobile sign language dictionaries and retrieval, and tools for searching for signs by example. Ultimately, this resource also is likely to contribute to systems for automated machine translation and human-computer interaction.
Last Modified: 10/29/2017
Modified by: Carol J Neidle
Please report errors in award information by writing to: awardsearch@nsf.gov.
