Psychology & Neuroscience Stack Exchange is a question and answer site for practitioners, researchers, and students in cognitive science, psychology, neuroscience, and psychiatry. It only takes a minute to sign up.
Sign up to join this community
I am developing a game where the music needs to be synchronized to the actual game logic. That means that some objects in the game react to events in the music. Of course, the music will never be completely synced to the output on the screen, so I need an indication of the time difference so I can sync the music manually to the game logic.
What is the average/maximum time for a human to still associate a visual event on the screen with an auditory musical event?
Short answer
In the case of simple stimuli, visual and auditory stimuli can be offset between 25 and 50 ms and still be perceived as coming from one and the same same event.
Background
The question can be re-phrased as what is the window of integration of intersensory asynchrony in case of visual and auditory stimuli?
A well-known example where these two stimulus modalities are perceived as separate, while they are in fact coming from the same event, is a thunder heard after lightning. This is caused by the fact that sound travels at a speed much slower than light, and hence a thunder can lag lightning by seconds.
source: NASA
In many instances, however, a multimodal perception is actually perceived as being synchronous, while they are in fact offset in time due to differences in physical characteristics of the stimuli. Take the storm as an example - when it is far away, the thunder is perceptually dissociated from the lightning, because the sound lags the light by seconds. But when the thunderstorm is close enough, the auditory crack and visual lightning are perceived as synchronous, while in fact they are still offset because of the sound travelling so much slower than light.
So the question becomes, as you rightfully ask, what are the margins in which stimuli across modalities can be asynchronous while perceived as one event? In other words, what is the window of integration?
Vroomen and Keetels (2010) conclude in their review on this topic that a stimulus asynchrony in the case of auditory beeps and visual flashes can be between 25 and 50 ms and still be perceived as coming from the same event.
The window of integration between more complex stimuli can be much greater. For example, the window for speech and visual information can be as large as 203 ms. Such large windows of integration point toward higher processes playing a role in the brain. Only temporal lags below 20 msec are expected to go unnoticed because of hard-wired limitations on the resolution power of the individual senses.
Hence, Vroomen and Keetels (2010) argue there must be higher processes at work in the brain that actively synchronize percepts that are offset in time, but seem to belong to one and the same event. One such mechanism is referred to as temporal ventriloquism, which means that a perceptual modality is actively shifted in time to match it with another modality. This effect is most pronounced in visual stimuli, in that a visual percept is actively adjusted in time to match a sound or tactile stimulus. Likely visual percepts are shifted preferably by the brain, because the visual system is the slowest of all the senses.
Reference
Vroomen & Keetels, Att Percept Psychophys 2010; 72(4): 871-84
