Autism Spectrum Disorder (ASD) is a neurodevelopmental communication disorder affecting over 3.5 million Americans that results in functional language delays and behavioral challenges. These delays often result in limited speech and increased communication difficulties -- approximately 40% of people in the ASD population are minimally verbal or non-verbal, having increased communication challenges. While many technologies and teaching tools have surfaced to assess the needs of non-verbal children with ASD, these often rely on the physical presence of a Speech Language Pathologist, teacher, or parent. This creates a sizable bottleneck for those with ASD to further learn and practice language skills. 

It was the goal of this SBIR Phase I to explore the feasibility of an AI Language Coach for non-verbal or minimally verbal children with ASD. The main objectives of this 12-month project were to establish the protocols for data collection and processing, train an analysis tool to assess user proficiency, and construct a pilot recommendation engine. The original end goal for this project was to create an AI algorithm that recommended alternate phrases or correct grammar per the assessed proficiency of the user. This would supplement a teacher or parent’s instructions while still being engaging to the child. 

Establishing protocols. Protocols were established for data storage, access, curation, and utilization. A cloud computing environment was created to facilitate these processes. Using this, data was prepared, analyzed, and utilized for training purposes.

Training an analysis tool. The biggest challenges faced by the team were a lack of properly annotated dialogue-based datasets for children, no datasets providing reliable developmental labels for child speech, and poor performance of available diarization technology for child speakers. The CHILDES dataset was found to provide a wealth of transcribed dialogues between children and parents or therapists, but it was lacking in development metric labels. The original developmental metric of interest was Brown’s Stages, but the metric was changed to IPSyn scoring due to the latter’s more robust analysis of syntax and morphology. A customized IPSyn scoring algorithm was developed to be used in conjunction with the CHILDES data to address the second challenge. In order to address the challenge of child speech diarization, a custom diarization algorithm was created and trained using audio from both normal and speech impared children. In order to assess accuracy, a custom benchmarking dataset was created using the Prodigy annotation to label dialogues between children and adults. Audio for this dataset was chosen for clarity as well as gender and age heterogeneity. 

Construction of the recommendation algorithm. The original goal of the recommendation engine was to provide feedback directly to the child based on their developmental level. While evaluating the use of IPSyn scores without the specificity of individual rules, we investigated recommendations based on speech from children of a higher IPSyn score such as 10 points higher than the user child. Here we pivoted again to instead provide recommendations to clinicians and parents based on feedback from experts in the field. It was decided that the specific IPSyn rules triggered would be provided to parents or clinicians as insight for what is needed in the developmental track of the child. A social function detection algorithm was also developed based on the dialogue observed between children and adults. The social functions detected include questions, comments, requests, commands, and refusals. A recommendation report that included both the IPSyn rules and detected social functions was the final outcome of the recommendation goal. 

The main accomplishments of this Phase I SBIR focused on the dataset and our algorithm. We have created a child-specific dataset that now has the potential for benchmarking in the field of diarization -- the first of its kind. We have leveraged this to make strides in child-specific diarization capabilities for our algorithm. We have also developed programmatic IPSyn rules, opening the possibility for more automated metrics of language development. This addressed a common pain point that we repeatedly encountered in our customer discovery research: clinicians’ need for an automated, time-efficient, and accurate language development metric system. There is also a large research and development gap in the detection of the social function of speech from text dialogue. While several technologies can detect and caption speech, few engines are capable of processing and categorizing the function of utterances. This feature will allow our algorithm to assess how each participant is using language to communicate with each other, or not. More importantly, this feature could help detect social-environmental factors that contribute to MLU length, syntax complexity, and social function application. 

With the help of the NSF, iTherapy is proud to have created one of the first, if not the first, algorithm to integrate state-of-the-art speech-to-text diarization, IPSyn scoring, and social function detection to enable mobile automation of speech and language assessment in addition to social function measurement.

Last Modified: 06/10/2021
Modified by: Lois J Brady