Current Biology, Vol. 13, R519–R521, July 1, 2003, ©2003 Elsevier Science Ltd. All rights reserved. DOI 10.1016/S0960-9822(03)00445-7

Multisensory Integration: Maintaining the Perception of Synchrony Charles Spence and Sarah Squire

We are rarely aware of differences in the arrival time of inputs to each of our senses. New research suggests that this is explained by a ‘moveable window’ for multisensory integration and by a ‘temporal ventriloquism’ effect.

We are all familiar with the experience of seeing lightning before hearing the associated thunder. This discrepancy in our perception of a synchronous, albeit distant, multisensory event is caused by physical differences in the relative time of arrival of stimuli at the eye and ear. Light travels through air far more rapidly than sound: 300,000,000 versus 330 metres per second. Differences in arrival time also occur for events that occur much closer to us, and yet we are rarely aware of them [1]. Part of the reason for this is that the mechanical transduction of sound waves at the ear takes less time than that required for the chemical transduction of light at the retina [2]. These physical and biophysical differences in the arrival time of light and sound cancel each other out when stimuli are approximately 10 metres from us, at the so-called ‘horizon of simultaneity’ [1]. Given most audiovisual events are not perceived at this ‘optimal’ distance, however, psychologists, neuroscientists and even philosophers have long puzzled over why the perception of multisensory synchrony should be such a pervasive aspect of our everyday phenomenology [1,3]. New research has provided some intriguing insights into this problem [4–6]. Traditionally, many scientists believed that humans had a relatively wide window for the integration of multisensory stimuli, and that we were therefore simply insensitive to small differences in the arrival time of signals to different sensory modalities (Figure 1A). Indeed, research has shown that auditory speech has to lag behind matching visual speech — lip movements — by more than 250 milliseconds for the asynchrony to be perceived [7]. Although it could be argued that the temporal window for multisensory integration might be particularly wide for more complex stimuli, such as audiovisual speech, single-cell studies in animals have highlighted a wide temporal window of integration for simple tone and light pairs as well (in multisensory convergence sites, such as the superior colliculus [2,8]). Recent research has uncovered two alternative means by which the brain ensures the continued perception of multisensory synchrony despite having to deal with asynchronous inputs. First, psychophysical data reported by Sugita and Suzuki [4] suggest that the window for multisensory temporal integration actually moves as audiovisual stimuli become more distant from Department of Experimental Psychology, University of Oxford, Oxford, OX1 3UD, UK. E-mail:[email protected]

Dispatch

us, in order to accommodate the fact that sound will increasingly lag behind vision with increasing distance (Figure 1b). The existence of a moveable window for multisensory simultaneity, to accommodate the typical delay in auditory arrival times, is consistent with previous studies showing that people (and animals) are more A

Neural response to stimulus Visual stimulus Time Auditory stimulus Wide window for multisensory integration

B

Visual stimuli

Auditory stimuli Near event Far event Moveable window for multisensory integration C

Temporal ventriloquism Visual stimulus

Auditory stimulus Current Biology

Figure 1. Maintaining the multisensory perception of synchrony. Why do we normally perceive the various sensory attributes of audiovisual events to be occurring simultaneously, given that the auditory and visual signals frequently arrive at different times? (A) The traditional view was that we have a large temporal window for multisensory integration, and so are simply insensitive to small differences in the arrival time of different sensory signals, provided that some component of the neural signals associated with each of the signals overlap [8]. (B) More recently, however, Sugita and Suzuki [4] have suggested that, rather than having a large temporal window for multisensory integration, humans may have a ‘moveable window’ for multisensory integration instead. According to Sugita and Suzuki [4], the brain takes account of the fact that auditory stimuli will increasingly lag behind visual stimuli as audiovisual events move further and further away from an observer, at least up to a distance of 10 m (though see [9]). (C) An alternative account comes from Morein-Zamir et al. [5], who propose that the multisensory perception of simultaneity is sometimes maintained by our ability to ventriloquize visual stimuli across time into temporal alignment with subsequently presented auditory stimuli.

Dispatch R520

A Unimodal visual presentation Visual targets Time

Auditory distractors Perceived visual rate B Congruent multisensory presentation Visual targets Auditory distractors Perceived visual rate C Incongruent multisensory presentation Visual targets Auditory distractors Perceived visual rate Visual targets Auditory distractors Perceived visual rate Current Biology

Figure 2. Auditory modulation of visual temporal rate perception. Participants in a recent study by Recanzone [6] judged whether two sequentially presented visual stimuli were presented at the same versus at different temporal rates. One visual stimulus alternated on and off at a rate of 4 Hz, the other was presented at a rate of between 3.5 and 4.5 Hz. The visual stimuli were either presented in silence (A), or else were accompanied by auditory distractors that were presented at the same (B) or at different temporal rates (C). Psychophysical analyses of the results showed that the rate at which the visual stimuli were perceived to flicker was modulated by the rate at which the simultaneously presented, but taskirrelevant, incongruent auditory distractors were heard to ‘flutter’ (see the conditions highlighted inside the red rectangle). This auditory ‘driving’ of visual temporal rate perception was unaffected by changes in the focus of a participant’s attention, but was eliminated if the disparity between the temporal rates in the two sensory modalities became too great. In subsequent experiments, Recanzone demonstrated that changing the rate at which a (distractor) light flickered had no effect on the rate at which (target) sounds were heard to flutter (see also [12]). Therefore, taken together, research now shows that auditory stimuli can dominate both the perceived rate and time-of-occurrence of associated visual stimuli [5,6,12], while visual stimuli dominate the more spatial aspects of multisensory perception [11]. This pattern of results is consistent with the ‘modality appropriateness’ account of sensory dominance [12], according to which the sensory modality that provides the most accurate, or appropriate, information will dominate the sensory attribute being discriminated.

Audition ‘drives’ vision

sensitive to audiovisual asynchrony when sound leads vision than vice versa [2,7,9,10]. Interestingly, however, Sugita and Suzuki [4] found that this moveable window only operates up to a distance of around 10 metres, hence explaining why our perception of synchrony breaks down for more distant audiovisual events. The second novel explanation for our insensitivity to audiovisual asynchrony to emerge from recent research stems from the discovery of a ‘temporal ventriloquism’ effect; this can correct for asynchronous auditory and visual inputs by ventriloquising (or binding) visual stimuli into temporal alignment with the appropriate auditory events. For example, Morein-Zamir and colleagues [5] have shown that the perceived time of arrival of a visual event can be ventriloquized into temporal alignment with a subsequently presented sound (Figure 1c). The phenomenon of temporal ventriloquism not only corrects for differences in the time of arrival of stimuli from different sensory modalities, but is also involved in the synchronization of the rate at which sensory events are perceived to occur. Recanzone [6]

has recently demonstrated that the perceived rate at which people judge a light to be flickering on and off can also be modulated by the rate at which a concurrent stream of auditory stimuli are presented (Figure 2). Importantly, however, it is the visual event that is ventriloquized into temporal alignment with sound in the studies of both Recanzone [6] and Morein-Zamir et al. [5]. This is the opposite of what happens in the wellknown spatial ventriloquism effect, where the perceived location of a sound is ventriloquized toward the location of a simultaneously presented visual stimulus [11]. These findings support the ‘modality appropriateness’ account of sensory dominance [12], according to which the modality that provides the most accurate, or appropriate, sensory information — audition in the case of temporal information and vision in the case of spatial information — dominates the ensuing percept. One of the challenges for future research will be to characterize the most important differences between the ‘moveable window’ and ‘temporal ventriloquism’ accounts of multisensory temporal perception. Another

Current Biology R521

challenge will be to determine the extent to which similar mechanisms for resynchronizing different sensory modalities also apply to the resynchronization of the different attributes of a stimulus within a particular sensory modality, such as the colour and motion of a visual stimulus [9,13,14]. Researchers are also trying to develop a better understanding of both how and where such temporal integration effects occur in the brain [8,14,15]. Recent developments in neuroimaging are now enabling researchers to investigate the neural substrates of synchrony perception (for example, [15]), while avoiding the limitations inherent in the more traditional neuropsychological approach of studying brain-damaged patients exhibiting specific deficits in temporal perception (see [16] for a review). It is important to note, though, that in most neuroimaging and psychophysical studies reported to date, the auditory stimuli were presented over headphones, while visual stimuli were typically presented from directly in front of the participants. In other words, the stimuli in different sensory modalities were presented from different spatial locations. So it is unclear whether the pattern of activation reported in neuroimaging studies, such as that of Bushara et al. [15], represents the neural correlate of temporal perception per se. Instead, the patterns of neural activity may correspond to the neural substrates of spatial ventriloquism, which also occurs for spatially displaced but synchronous audiovisual stimuli ([11] and our unpublished results). Meanwhile, the numerous behavioral studies that have incorporated this spatial confound (for example [4,7]; see [16] for a review) may also have systematically overestimated people’s ability to detect audiovisual asynchrony, given desynchronous inputs, because temporal binding, or ventriloquism, effects are more pronounced for stimuli presented from the same, as compared to different, spatial locations [10,16]. Finally, one long-standing issue, which has yet to receive the attention it deserves, concerns the underlying causes of the large individual differences in the perception of multisensory synchrony reported in previous research [9,17]. For example, Stone et al. [9] found that, while some people perceived simultaneity most convincingly — defined as the stimulus onset asynchrony (SOA) at which people were most likely to judge sound and vision as being synchronous — when sounds precedes vision by 20 milliseconds, others perceive simultaneity when vision leads by as much as 150 milliseconds. These dramatic individual differences, which were robust across testing sessions, hark back to the differences in multisensory perception first noted by astronomers more than 200 years ago, and whose discovery — formalized by the notion of the ‘personal equation’ — led to the very foundation of experimental psychology [16,17]. While it is still unclear exactly what accounts for these individual differences in perception [9,16,17], there can be no doubt that a better understanding of the mechanisms underlying the multisensory perception of synchrony will have important contemporary applications in contexts as diverse as the setting of acceptable asynchronies for multimedia broadcasting

[18] and the design of multimodal warning signals that can more effectively stimulate the senses [19]. References 1. Pöppel, E. (1988). Mindworks: Time and conscious experience. (Harcourt Brace Jovanovich, New York). 2. King, A.J. and Palmer, A.R. (1985). Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp. Brain Res. 60, 492–500. 3. Dennett, D.C. and Kinsbourne, M. (1992). Time and the observer: The where and when of consciousness in the brain. Behav. Brain Sci. 15, 183–247. 4. Sugita, Y. and Suzuki, Y. (2003). Audiovisual perception: Implicit estimation of sound-arrival time. Nature 421, 911. 5. Morein-Zamir, S., Soto-Faraco, S. and Kingstone, A. (2003). Auditory capture of vision: Examining temporal ventriloquism. Cog. Brain. Res. in press. 6. Recanzone, G.H. (2003). Auditory influences on visual temporal rate perception. J. Neurophysiol. 89, 1078–1093. 7. Dixon, N.F. and Spitz, L. (1980). The detection of auditory visual desynchrony. Perception 9, 719–721. 8. Meredith, M.A., Nemitz, J.W. and Stein, B.E. (1987). Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J. Neurosci. 7, 3215–3229. 9. Stone, J.V., Hunkin, N.M., Porrill, J., Wood, R., Keeler, V., Beanland, M., Port, M. and Porter, N.R. (2001). When is now? Perception of simultaneity. Proc. R. Soc. Lond. B Biol. Sci. 268, 31–38. 10. Spence, C., Baddeley, R., Zampini, M., James, R. and Shore, D.I. (2003). Crossmodal temporal order judgments: When two locations are better than one. Percept. Psychophys. 65, 318–328. 11. Bertelson, P. and de Gelder, B. (2003). The psychology of multimodal perception. In Crossmodal space and crossmodal attention, C. Spence and J. Driver, Eds. (Oxford University Press). In press. 12. Welch, R.B., DuttonHurt, L.D. and Warren, D.H. (1986). Contributions of audition and vision to temporal rate perception. Percept. Psychophys. 39, 294–300. 13. Bergenheim, M., Johansson, H., Granlund, B. and Pedersen, J. (1996). Experimental evidence for a sensory synchronization of sensory information to conscious experience. In Toward a science of consciousness: The first Tuscon discussions and debates, S.R. Hameroff, A.W. Kaszniak and A.C. Scott, Eds. (MIT Press, Cambridge, MA), pp. 303–310. 14. Grossberg, S. and Grunewald, A. (1997). Cortical synchronization and perceptual framing. J. Cogn. Neurosci. 9, 117–132. 15. Bushara, K.O., Grafman, J. and Hallett, M. (2001). Neural correlates of auditory-visual stimulus onset asynchrony detection. J. Neurosci. 21, 300–304. 16. Spence, C., Shore, D.I. and Klein, R.M. (2001). Multisensory prior entry. J. Exp. Psychol. Gen. 130, 799–832. 17. Mollon, J.D. and Perkins, A.J. (1996). Errors of judgement at Greenwich in 1796. Nature 380, 101–102. 18. ITU-T. (1990). Television and sound transmission: Tolerances for transmission time differences between the vision and sound components of a television signal. International Telecommunication Union, Telecommunication standardization sector of ITU, Recommendation J.100. 19. Spence, C. and Driver, J. (1999). A new approach to the design of multimodal warning signals. In Engineering Psychology and Cognitive Ergonomics, Vol. 4: Job Design, Product Design and HumanComputer Interaction, D. Harris, Ed. (Ashgate Publishing, Hampshire), pp. 455–461.