Which physical properties of a stimulus can a human attend to?

Question

I am not only curious about speech, but for concreteness in illustrating my question, I consider the case of speech perception. Assume a listener is presented with an acoustic waveform. The wave causes deviations from the atmospheric pressure at the listener's eardrum. Thus, the acoustic pressure as a function of time “comes for free”, so to speak. Mechanically so.

From sound pressure we can derive amplitude, and by the fourier transform, amplitude as a function of time is transformed to the frequency domain. That is, amplitude as a function of time becomes amplitude as a function of frequency. We use this transformation because it reveals properties of the wave which are perceptually relevant, e.g. formant frequencies. Presumably, listeners can attend to properties of the waveform describable under this transformation.

But what about mathematical models of the acoustic wave which are not simply transformations on the wave, but which derive some richer structure from amplitude fluctuations over time. For instance, what about a model which constructs a higher dimensional representation of the wave from the one-dimensional pressure fluctuations? (Imagine that instead of amplitude being a point moving back and forth along the real number line, the wave is represented as a point moving around in, say, 3-dimensional space). Say that such a model revealed a perceptually relevant property P of the waveform. Would the listener be able to “hear” P directly, or would would she need to engage in some information processing task to recover P from pressure fluctuations over time?*

*I realize that human listeners don't actually manipulate equations in their heads. I use the mathematics just to describe the underlying neural computation.

Answer 1

I found a paper called Summary statistics in auditory perception (McDermott, Schemitsch & Simoncelli, 2013) that might be relevant to your question. If you can't access it in full, you can a great description of the paper here. Please note that I'm writing about it from memory and that some details might not be correct.

The authors had a task where participants had to judge whether two sound excerpts are same or different. These sound excerpts could, further, have similar or dissimilar summary statistics, meaning that some higher-order regularity was superimposed on the initially generated sounds (such as the somewhat regular pattern of the sound of sea waves crashing, or the wind picking up, or a fire crackling). So, the sounds could differ in a local way (which specific frequency was displayed at some moment) or in a global way (what were the higher-order regularities of these sounds). They then displayed these sounds in pairs for variable amounts of time.

They found that people were better at discriminating sounds from each other when the sounds lasted longer - which is logical - but this was only true if the complex sound textures were different from each other. When the textures were mutually similar, people actually got worse if they listened to the (potentially different) sounds for a longer time.

The interpretation the authors gave is that the auditory system first encodes simple features of sounds, but, once confronted with too much stimulation to keep track of, it instead extracts higher order regularities and relies on these for perception.

I do not have a technical background so I can't go into detail about what these higher order regularities represent, mathematically speaking. But the links come with some images with explanations, and you might be able to glean more information from them than I.

Here is the abstract:

Sensory signals are transduced at high resolution, but their structure must be stored in a more compact format. Here we provide evidence that the auditory system summarizes the temporal details of sounds using time-averaged statistics. We measured discrimination of 'sound textures' that were characterized by particular statistical properties, as normally result from the superposition of many acoustic features in auditory scenes. When listeners discriminated examples of different textures, performance improved with excerpt duration. In contrast, when listeners discriminated different examples of the same texture, performance declined with duration, a paradoxical result given that the information available for discrimination grows with duration. These results indicate that once these sounds are of moderate length, the brain's representation is limited to time-averaged statistics, which, for different examples of the same texture, converge to the same values with increasing duration. Such statistical representations produce good categorical discrimination, but limit the ability to discern temporal detail.

edit: I found a link to the pdf of the second paper I mention.

Answer 2

I'm not completely sure what you ask. I give my comment as an answer, because there is more space here. I'll delete it, if you feel I'm off the mark.

As far as physical theory goes (and I understand it), sound is the movement of air molecules and the resulting fluctuation in density. Measuring these density fluctuations and representing them in numbers gives us a wavelike, undulating function with the properties of waveform (sinuous, square, sawtooth), frequency, amplitude, and transversality (is the wave upright or leaning backward or forward?). Did I miss anything?

Now, you seem to assume that our eardrums perceive the frequency (tone pitch) and amplitude (loudness) of an acoustic wave, but maybe not the transversality or waveform. But that's an error in reasoning.

We don't perceive waves. We perceive changes in air pressure. It does not matter what wave represents these changes mathematically. Any change (of a certain magnitude) stimulates our ear drums. Measuring the oscillation of the ear drum will give you a similarly complex wave (though probably not the same as the wave of air density fluctuation, because of different physical properties of the material of the ear drum). What we have is the "translation" of density fluctuations in a substance into the oscillations of a membrane. Both can be modelled as waves, and some changes in those waves will occur in the process of translation (probably some fine complexities will get evened out, resulting in a less finely "spiky" but more "smooth" wave), but there is no property of the wave that will get completely lost or be ignored.

The same goes for the other senses. Our eyes don't perceive color, saturation or luminosity, they are stimulated by photons (or whatever). Our cognition represents, i.e. "perceives", the electrical impulses from our receptors as color, brightness etc., so we like to think in a model that includes these properties, but our eyes don't react to "color" at all. There is no color in the physical world, just the movement of particles.

Maybe this helps you to clarify your question.