Synchronization of perception of sensory information

Question

I am not an expert (actually I am not even psychologist/neuroscientist or anything like it).My question is:

Since light and sound comes to, respectively, our eyes and ears at different times and "visual input" and "sound input" take different times to be processed (and as far as I know, even some traits of vision - like color, motion, shape - take different time to be processed), how does our brain gives rise to the counscious experience of simultaneity (in the case of actual simultaneity, of course)?

How do we experience this synchronization of stimulus (which, of course, can include others senses, like touch and proprioception)? What would be a time line of that? Because it makes a bit confuse. Aren't we experiencing reality in real time (except for the fact that stimulus take time to get to our senses)?

Answer 1

The question you ask is the topic of a very active line of research.

First you are right that light (~300.10^6m.s-1) and sounds (~350m.s-1) reach an observer at different times, as a function of distance. You are also right that visual and auditory inputs are processed at different speeds by the brain, with auditory signals being faster. The auditory system is much better, in general, at processing things quickly. Take for example the fact that the auditory system processes sounds between 20 and 20,000Hz (the auditory bandwidth), while the visual system saturates at ~30Hz (which is why movies are at 60Hz). It doesn't necessarily means you get "aware" of sounds faster than vision but this is true as well. You can react to sounds in 50-150ms, and visual stimuli in 150-250ms.

So how does your brain fuses sounds and vision, and do you get "aware" of this percept relative to "real" time? Your brain needs to make an inference, that is to guess whether a sound and a visual input are generated by the same source. Your brain uses the probabilities of sounds and visual inputs to co-occur to guess whether they should be fused or not. This is called causal inference (scientific review paper below). If a visual and auditory inputs are too discrepant in space (where they come from) or time, your brain will assume they have different sources. If they more or less match, your brain will assume they come from the same source and combine them, which provides some advantages to locate that source, for example, as you now have 2 pieces of information. When you become "aware" of the sound/visual input/combined percept, relative to the "real" world is as far as I know unknown and difficultly testable (how would you estimate when some one becomes aware of an external stimulus when all you have are external stimuli to compare to). This can probably be addressed but as I said it is an ongoing field of research (below a review on the topic).

There is a phenomenon you will probably find interesting (original paper below). Because the brain uses the general statistics of the world to decide whether to fuse or split sound and vision, it is possible to change that. Some scientists have adapted observers to a specific mismatch between sound and vision. After some time of observing a consistent delay, let's say a beep happening consistently 100ms after a flash, the observers were asked to tell whether a beep and a flash happened simultaneously. What scientists have found consistently (there are many studies looking at this phenomenon) is that before adaptation people are mostly accurate. But after adaptation the observers report (in that example) the flash as happening roughly 100ms before the beep (as they were adapted). There are some limits to that, people can't adapt to too big a delay etc. But it is a robust phenomenon. If you are old enough to have experienced bad streaming services, where sound was often mismatched to images, you will remember thatafter some time you got not so annoyed with it. This is the reason, you just adapted to the delay. Scientists have also tried to see if this is distance-dependent but results are still inconclusive (different studies found contradictory results so far).

Shams, L., & Beierholm, U. R. (2010). Causal inference in perception. Trends in cognitive sciences, 14(9), 425-432.

Hanson, J. V., Heron, J., & Whitaker, D. (2008). Recalibration of perceived time across sensory modalities. Experimental Brain Research, 185(2), 347-352.

Sugita, Y., & Suzuki, Y. (2003). Implicit estimation of sound-arrival time. Nature, 421(6926), 911-911.

Answer 2

Short answer
The brain actively integrates and synchronizes sensory inputs, up to the point that it actually delays one modality to match it with another.

Background
Your question is all about intersensory asynchrony and sensory integration.

A well-known example where the two stimulus modalities you mention in your question (light and sound) are perceived as separate, while they are in fact coming from the same event, is a thunder heard after lightning. This is caused by the fact that sound travels at a speed much slower than light, and hence a thunder can lag lightning by seconds (Fig. 1).

^{source: NASA}

In many instances, however, a multimodal perception is actually perceived as being synchronous, while they are in fact offset in time due to differences in physical characteristics of the stimuli. Take the storm as an example - when it is far away, the thunder is perceptually dissociated from the lightning, because the sound lags the light by seconds. But when the thunderstorm is close enough, the auditory crack and visual lightning are perceived as synchronous, while in fact they are still offset because of the sound travelling so much slower than light.

So why does the brain sync perceptions into 1 event, and sometimes not? This has to do with the window of integration. Vroomen and Keetels (2010) conclude in their review on this topic that a stimulus asynchrony in the case of auditory beeps and visual flashes can be between 25 and 50 ms and still be perceived as coming from the same event.

The window of integration between more complex stimuli can be much greater. For example, the window for speech and visual information can be as large as 203 ms. Such large windows of integration point toward higher processes playing a role in the brain. Only temporal lags below 20 msec are expected to go unnoticed because of hard-wired limitations on the resolution power of the individual senses.

Hence, Vroomen and Keetels (2010) argue there must be higher processes at work in the brain that actively synchronize percepts that are offset in time, but seem to belong to one and the same event. One such mechanism is referred to as temporal ventriloquism, which means that a perceptual modality is actively shifted in time to match it with another modality. This effect is most pronounced in visual stimuli. Visual perceptions are in fact actively adjusted in time to match a sound or tactile stimulus. Likely visual percepts are shifted preferably by the brain, because the visual system is the slowest of all the senses.

_{Reference
Vroomen & Keetels, Att Percept Psychophys 2010; 72(4): 871-84}

Answer 3

The synchronization of sensory information is called multisensory integration:

Multisensory integration, also known as multimodal integration, is the study of how information from the different sensory modalities (such as sight, sound, touch, smell, self-motion, and taste) may be integrated by the nervous system.

This is the most salient example of neural "binding".

The examples provided are of "temporal binding" - eg, when you touch your nose and your toes at the same time, the signal from your toes takes much longer to reach your brain, and yet, you experience the simultaneity just fine. However, binding encompasses all aspects of conscious experience, such as object recognition, motor response, language processing, decision-making, emotion, etc, all of which can happen at different times and in different parts of the brain, but yet be experienced as unitary.

The question of how this happens is called the binding problem, or to be more specific:

Secondly, there is the combination problem: the problem of how objects, background and abstract or emotional features are combined into a single experience. The combination problem is sometimes called BP2.

Many hypotheses have been proposed that you can read about on Wikipedia, and some neuroscientific hypotheses as well, but in general, this problem is currently unsolved.

The inevitable corollary of neural binding is that our sense of the real-time present is mostly composed of events taking place in the recent past - sometimes referred to as the "specious present". How far in the past, or more specifically, the "temporal binding window" is highly variable, differs between individuals, and is affected by mental disorders, training, etc. It is also not so much a window as a probability distribution, but certainly extends to the hundreds of milliseconds.