Events in the natural world are perceived through multiple sensory modalities, and multi-modal interactions in the brain are critical for generating coherent perceptual experiences and adaptive behavioral responses.  Our NSF-sponsored research has been concerned with understanding how cross-modal interactions take place in the human brain to achieve perceptual integration. The major research theme over the past four years has been to study how the visual and auditory systems of the brain interact, and in particular to show how sounds influence our visual perceptions.  Our principal finding has been that a salient sound automatically attracts attention to its location so that the perception of a subsequent visual stimulus that occurs at the same location is enhanced.  Thus, when a visual stimulus follows a salient sound at the same location it is perceived more accurately, enters awareness more rapidly, appears brighter and elicits a larger evoked potential in the visual cortex than does a visual stimulus presented at a different location (e.g., in the opposite visual field).

            We investigated the neural basis of this cross-modal facilitation of visual perception by recording changes in the electroencephalogram (EEG) elicited by sounds.  We discovered that presenting a salient sound to one side elicits a positive potential shift in the contralateral visual cortex, which was termed the “Auditory Contralateral Occipital Positivity” (ACOP).  We hypothesized that this ACOP was the neural sign of the perceptual priming of the visual cortex by the sound.   In support of this hypothesis, we found that letters that were flashed at the location of an immediately preceding sound were discriminated more accurately than letters flashed in the opposite visual field.  Most importantly, the ACOP triggered by the sound was larger when the subsequent letter discrimination was correct as opposed to when it was incorrect.  We concluded that the spatially non-predictive sound automatically captured visual attention to the sound’s location, which resulted in a lateralized facilitation of visual processing that was indexed by the ACOP.

                Over the past year we have extended these studies of visual cortex activity elicited by sounds into the frequency domain, using wavelet analyses.   We found that the sounds provoked a robust desynchronization (blocking) of the ongoing alpha rhythm of the EEG, and that the time course of this alpha-blocking closely paralleled that of the ACOP slow potential. In support of the close connection between the ACOP and alpha blocking, we reanalyzed the data from the letter discrimination experiment and found that trials with correct discriminations had stronger alpha blocking that incorrect trials, thereby closely paralleling the findings with ACOP.  These studies demonstrate that the blocking of alpha rhythm provoked by a sound represents a fundamental neural mechanism by which cross-modal attention facilitates visual perception.

               These studies have advanced our understanding of the brain mechanisms by which information from different sensory modalities is combined in the brain to achieve perceptual coherence in a multi-sensory world.  This research also has broader implications for the fields of communication, education, and mental health.  Because much of our everyday perceptual experience involves multi-sensory integration, co-ordination between sensory inputs in different modalities helps us localize and re...

Please report errors in award information by writing to: awardsearch@nsf.gov.