Autism Spectrum Disorder (ASD) is the most prevalent developmental disorder among children in the US. The Center of Disease Control estimated that ASD currently affects 1 in 68 children. Hallmarks of the condition are social and communicative impairments. The former includes poor eye contact, a lack of shared enjoyment and social reciprocity, while the latter leads to significant difficulty in producing speech and using gestures to communicate with others.These impairments could lead to life-long obstacles in developing social relations and maintaining communicative engagements with others. If left untreated, most children with ASD will not be able to live independently as adults. Although early behavioral and educational interventions are effective in addressing many of these deficits, such interventions traditionally require significant effort from parents, therapists, and teachers.

In this NSF project, our interdisciplinary team at University of Kentucky have developed novel multimedia-based instruction (MBI) technologies for children with ASD. MBI presents concepts in a systematic, simple format, and it effectively gains and keeps the child’s attention while providing a less emotionally laden way to learn. Many children with ASD find explicit routines in MBI comforting, circumventing difficult social demands through predictable interaction patterns. From the standpoint of delivering the interventions, MBI offers portability across different learning environments, accessibility by diverse population, controlled presentation of instructional stimuli, and customization based on individual needs. Our project aimed at developing tools that could provide greater flexibility for therapists, teachers, and caretakers to adopt MBI in their programs. The deployment of such tools should lower the technology barrier, increase access to evidence-based treatments, and promote consistent practice of behavioral modeling in home and other places beyond clinics.

The two main prototypes developed in this project are MEBook and Virtual Mirror. MEBook is a social narrative tool combined with serious games for social greeting training (Fig. 1). While most of us do not think much about saying “good morning” or “goodbye,” greetings involve all key elements of social interaction—proper eye gaze, hand movement, and vocalization, all of which are difficult for most children with ASD to master. MEBook combines elements from three different evidence-based interventions: social narratives,video self-modeling (VSM), and reinforcement. A novel component of MEBook is the use of an RGB-depth sensor to implement a gesture-based video game that serves a dual purpose of reinforcing the learning and creating raw material for VSM. Human-subject study on MEBook with ASD children age 7 to 12 showed that subjects demonstrated marked improvements in both vocalization and eye contact after the intervention and were able to maintain even after the end of the MEBook intervention.

The Virtual Mirror system extends beyond MEBook and other VSM systems which rely solely on genuine video records of the target behaviors of the learners. The fidelity of those behaviors is often poor because the learner has not yet mastered the skills. Virtual Mirror is a room-based augmented mirror display system in which the learner can see, in real-time, a computer-generated himself or herself engaging in the target behavior at a slightly higher skill level. It is a complex system with many innovations that break new grounds in computer vision and graphics. These innovations include (1) denoising and segmentation algorithms for both color and depth cameras (Fig. 2), (2) color and depth camera network calibration and rendering systems for 3D virtual environment (Fig. 3), (3) planar and curve mirror simulators (Fig. 4), and (4) a behavior transfer system that can transfer facial expression, eye gaze, and body pose from one image to another (Fig. 5). Human subject studies are underway to measure the effectiveness of the Virtual Mirror system in helping children with ASD in various learning tasks.

The research products stemmed from this project have been widely disseminated. A total of 19 journal papers and 21 conference papers have been published in the past five years. This project has provided partial financial support for 12 graduate students and five undergraduate students from three different departments: Electrical & Computer Engineering, Computer Science, and Educational Psychology. Five doctoral students and six master students have successfully defended their degrees based on the work directly related to the project goals. A broad range of outreach activities have been conducted during the funding period, including a year-long programming class for special-need high school students, project demonstrations in local K-12 schools, and technical talks in autism advocacy groups, local makerspace, as well as government meetings and entrepreneurship groups.

Last Modified: 04/18/2018
Modified by: Sen-Ching S Cheung