A collaboration between the Speech Processing and Auditory Perception Laboratory at UCLA and the Speech Group at ICSI focused on the refinement of the simple models used in Automatic Speech Recognition (ASR) with representations that have been developed from observations of mammalian auditory physiology. While no animal experiments were performed in this study, earlier work reported by researchers at the University of Maryland suggested that auditory systems were particularly sensitive to particular ranges of modulations over time and frequency. Processing techniques based on these observations have proved useful in many previous ASR experiments. In this study, though, the goal was to see if incorporating these insights into ASR approaches would yield a pattern of errors (for consonants in Consonant-Vowel-Consonant  (CVC) syllables) that was more similar to what would be observed for human perception.

More specifically, we wanted to see if correlations with the pattern of human perceptual errors for CVC syllables for noisy and rapid speech were improved by using modulation-based features for a modern ASR system. To explore this, the UCLA team recorded a corpus of spoken CVC syllables, and then conducted listening tests and analyzed results to determine the pattern of errors for the listeners. These results were then passed to the ICSI team, who developed the ASR systems and compared results with the perceptual ones.

One result in particular was quite notable, mainly, that higher temporalmodulations (i.e., components of the speech where the spectral content was varying quickly) were sometimes more helpful than including information from the entire range of temporal modulations. The most prominent of these results occurred for rapid speech, which could be expected since the spectral content varies more quickly in this case. But significant improvement in correlation with human consonant perception was really only observed for noisy speech. But for speech recognition researchers, in an era where the efficacy of machine learning approaches is the focus of much attention, it should be interesting that restricting the observed features (to higher modulations) gives improved performance for noisy and rapid speech.

The table below summarizes these results.