Language description and documentation efforts are currently at a critical stage. There are probably not enough linguist-hours left to assist all communities interested in documenting languages not currently being transmitted inter-generationally. At the same time, recent advances in natural language processing and speech technology hold the promise of being able to automate many of the more repetitive tasks taken on by field linguists, greatly speeding up the process of documenting a particular language and allowing more thorough description of more languages before they lose their last fully fluent speakers.

Based on discussions among researchers in endangered languages and those in speech and language technologies at the Shared Task Evaluation Campaigns with Endangered Language Data (EL-STEC) workshop held in 2016, the project team identified key pain points in endangered language research as well as tasks in speech and language processing which address these issues.  The current project focused on creation of Shared Tasks which support the development of speech and language technologies to accelerate the processing of new speech materials for language documentation. The project addressed a cascade of speech processing tasks, which take raw recordings of endangered language speech through the steps, such as speaker diarization, speaker identification, language identification, and transcription alignment, to support semi-automatic annotation and enrichment of endangered language recordings.
 
For the various subtasks, the project created corresponding shared tasks to focus research effort in the speech and language processing community. The project developed a pipeline to transform recordings and annotations from existing endangered language archival material into standardized training and test data sets. The project further implemented baseline systems for the shared tasks and developed a methodology for encapsulating these baselines in virtual machines, to lower the barrier of entry for task participants and to facilitate the dissemination of resulting systems to various users.  As a final step, evaluation suites of test data and metrics were also created, and a methodology for deployment to public shared tasks platforms was devised. These strategies were developed and tested on a range of data sets drawn from multiple geographic regions and language typologies, providing a rich and challenging basis to push the state-of-the-art. The project also demonstrated integration of baseline systems with interactive annotation tools widely used in language documentation, making the outcomes of the project available to a larger user base.

The project further conducted detailed comparisons and analysis of baseline systems, encompassing both classical machine learning approaches and current state-of-the-art, typically more data-hungry neural network methods.  Experiments demonstrated successful application of the methods to the project's newly created mutlilingual endangered language data sets and also demonstrated that the strongest neural network models significantly outperformed the classical models on this new data, despite the small size of the datasets.  However, importantly, those systems' effectiveness on the project's novel endangered language data sets significantly lagged that observed on the high-resource language data on which they had been developed and previously evaluated.  The findings highlight both the current limitations and potential strength of these speech processing tools for endangered language data and demonstrating the value of these new data sets and tasks in further propelling the state-of-the-art.

By creating shared tasks in the context of endangered language archive materials, the project contributes to the development of systems that are useful in language documentation and language reclamation projects. The creation of a structured shared task draws the attention of system developers and thus can help guide the research community towards identifying methodologies and technologies that are applicable to the specific needs of language documentation projects. The development of these shared tasks has laid the foundation for future improvements in spoken language processing for endangered languages, with the goal of  engaging participants in speech and language technology to push forward the state-of-the-art in several language processing tasks that can be integrated into tools to support the needs of end users.

Last Modified: 02/18/2025
Modified by: Gina-Anne Levow