This project addressed fundamental questions about the nature of the link between producing and perceiving speech. Experimental evidence points to the automatic and involuntary activation of the speech production system during speech perception. One such type of evidence comes from response times of subjects when they are instructed to say some syllable based on a visual cue. As they are preparing to speak the required syllable, they hear a distractor. Subjects’ spoken responses start sooner when the auditory distractor is the same as their response, and later when it is different.

One goal of this project was to investigate what properties of speech are important in the perception-production link. The literature has raised this question but not addressed it systematically. Speech production involves spatial properties, i.e., which speech articulators make what constrictions where in the vocal tract, and temporal properties, i.e., how those constrictions are arranged in time. We conducted experiments to elucidate the conditions under which perception-production interactions occur in speech, detected by measuring response times of syllables produced while another syllable was perceived. Subjects’ response times were compared in cases where the response and distractor shared spatial properties (e.g., “pa” and “ba” both involve a closing of the upper and lower lips) or temporal properties (“ba” and “da” both involve similar timing between the oral closure and vowel voicing). The experiments showed that both the spatial and temporal properties of speech play a role in the perception-production link. In one experiment, we found that subjects responded faster when the response and distractor matched in temporal properties than when they differed in them. In another, we found qualitatively similar results for spatial properties. This project provides the first concrete experimental evidence that the same properties of speech that are fundamental in traditional linguistic description are also active in the link between speech perception and production.

Another goal of the project was to develop a formal computational model of the perception-production link to account for disparate sets of results that have been used as evidence of this link. The model we developed focuses on the process by which speech properties are set during production. The perception-production link is formalized as the properties of a perceived utterance serving obligatorily as input to that property-setting process. In our experiments, response times were affected by whether the response and distractor (mis)matched on the properties we tested, but response times in both cases were slower than when subjects heard a tone distractor. Galantucci, Fowler, and Goldstein (2009) found that when the response and distractor were identical, response times were faster than with a tone distractor. Treating the tone as a baseline, we argue that the process of property setting requires both excitation and inhibition of the activation levels associated with the speech properties. When the response and distractor are identical, activation-level excitation is maximized due to fully congruent inputs to the property-setting process. The model predicts response times to be faster in this case than with a tone distractor, as Galantucci et al. found. Each mismatching property between a response and a distractor introduces inhibition to the process, predicting increasingly slower response times based on the number of mismatching properties, as we found in our experiments. Our model provides a unified account of results from these two different studies, formalizes the perception-production link, and identifies the computational principles that are involved in the process where that link is active.

Action and perception are known to be closely li...

Please report errors in award information by writing to: awardsearch@nsf.gov.