What is the difference (in a recording) between speech and a sound? What makes the difference to the brain?

Question

How does the brain cognitively differentiate between speech and a sound within a recording? I am asking this theoretical question for speech synthesis, i.e. Making a voice with just waves (no recordings). Is this even possible?

Answer 1

Speech is sound and the brain will recognise the sound as speech. As for the possibility of creating the sound of speech without recordings using sound waveforms I would say that is off topic here and would be a question for sound creators.

The subject of recognising speech from sounds is a big subject but to give a starting point here are a few pointers for further reading and research, depending on what area you want to look at more.

Neurological basis of Speech Perception

Heald & Nusbaum (2014) gives a brief outline of the neural processes involved.

One view of speech perception is that acoustic signals are transformed into representations for pattern matching to determine linguistic structure. This process can be taken as a statistical pattern-matching problem, assuming realtively stable linguistic categories are characterized by neural representations related to auditory properties of speech that can be compared to speech input. This kind of pattern matching can be termed a passive process which implies rigidity of processing with few demands on cognitive processing. An alternative view is that speech recognition, even in early stages, is an active process in which speech analysis is attentionally guided. Note that this does not mean consciously guided but that information-contingent changes in early auditory encoding can occur as a function of context and experience. Active processing assumes that attention, plasticity, and listening goals are important in considering how listeners cope with adverse circumstances that impair hearing by masking noise in the environment or hearing loss. Although theories of speech perception have begun to incorporate some active processing, they seldom treat early speech encoding as plastic and attentionally guided. Recent research has suggested that speech perception is the product of both feedforward and feedback interactions between a number of brain regions that include descending projections perhaps as far downstream as the cochlea.

They also define active and passive processes

The distinction between active and passive processes comes from control theory and reflects the degree to which a sequence of operations, in this case neural population responses, is contingent on processing outcomes (see Nusbaum and Schwab, 1986).

Figure 1A-C [Source: Heald & Nusbaum (2014)]

A passive process is an open loop sequence of transformations that are fixed, such that there is an invariant mapping from input to output (MacKay, 1951, 1956). Figure 1A illustrates a passive process in which a pattern of inputs (e.g., basilar membrane responses) is transmitted directly over the eighth nerve to the next population of neurons (e.g., in the auditory brainstem) and upward to cortex. This is the fundamental assumption of a number of theories of auditory processing in which a fixed cascade of neural population responses are transmitted from one part of the brain to the other (e.g., Barlow, 1961).

By contrast, active processes are variable in nature, as network processing is adjusted by an error-correcting mechanism or feedback loop. As such, outcomes may differ in different contexts. These feedback loops provide information to correct or modify processing in real time, rather than retrospectively. Nusbaum and Schwab (1986) describe two different ways an active, feedback-based system may be achieved. In one form, as illustrated in Figure 1B, expectations (derived from context) provide a hypothesis about a stimulus pattern that is being processed. In this case, sensory patterns (e.g., basilar membrane responses) are transmitted in much the same way as in a passive process (e.g., to the auditory brainstem). However, descending projections may modify the nature of neural population responses in various ways as a consequence of neural responses in cortical systems. For example, top-down effects of knowledge or expectations have been shown to alter low level processing in the auditory brainstem (e.g., Galbraith and Arroyo, 1993) or in the cochlea (e.g., Giard et al., 1994). Active systems may occur in another form, as illustrated in Figure 1C. In this case, there may be a strong bottom-up processing path as in a passive system, but feedback signals from higher cortical levels can change processing in real time at lower levels (e.g., brainstem). An example of this would be the kind of observation made by Spinelli and Pribram (1966) in showing that electrical stimulation of the inferotemporal cortex changed the receptive field structure for lateral geniculate neurons or Moran and Desimone’s (1985) demonstration that spatial attentional cueing changes effective receptive fields in striate and extrastriate cortex.

---

Linguistic Speech Perception

The following information was sourced from an article by John Kingston (Professor in the Department of Linguistics, University of Massachusetts) in 2013.

Links below are for Free PDF files - DOIs are provided in the references section in case of future breaks in links

According to the motor theory of speech perception (Liberman, et al., 1967; Liberman & Mattingly, 1985; Liberman & Mattingly, 1989), listeners recognize speech sounds by emulating their articulation in their heads. The sound is recognized when the acoustic properties of the emulation match those of the speech sound heard...Now, this is embodied cognition in that the cognitive act of perceiving a speech sound consists of emulating the speaker’s articulatory behavior mentally, i.e. behaving in the mind as the speaker would in the world.

...

According to the direct realist theory of speech perception (Fowler, 2006), no such emulation nor embodied cognition is necessary to perceive speech sounds because the speaker’s articulatory behavior in pronouncing them so structures the acoustic properties of the speech signal that the listener can identify the unique articulation that produced those properties. In other words, the acoustic properties are specific information about the articulations that produced them...This is extended cognition because no mental act is required to extract information from the speech signal’s acoustic properties. The information is instead present and patent within those properties and no analysis is required to turn those properties into information. All the cognitive work, other than simply being around to hear the sounds, is done outside the listener’s head.

Kingston says that there are empirical and theoretical challenges to both the motor theory’s and direct realism’s accounts of speech perception, stating that speech perception is neither emboedied or extended; however he did not elaborate on this.

Source

Kingston, J., (2013). Why speech perception is neither embodied nor extended cognition [Online] Available at http://blogs.umass.edu/jkingstn/2013/12/

---

References

Barlow, H. B. (1961). Possible principles underlying the transformations of sensory messages In: Sensory Communication, ed W. Rosenblith (Cambridge, MA: MIT Press): pp 217–234. ISBN-13: 978-0262518420 DOI: 10.7551/mitpress/9780262518420.003.0013

Fowler, C. A. (2006). Compensation for coarticulation reflects gesture perception, not spectral contrast, In: Perception & Psychophysics. 68(2): pp 161-177. DOI: 10.3758/BF03193666 PMID: 16773890

Galbraith, G. C., and Arroyo, C. (1993). Selective attention and brainstem frequency-following responses, In: Biological Psychology 37(1): pp 3–22. DOI: 10.1016/0301-0511(93)90024-3

Giard, M. H., Collet, L., Bouchet, P., and Pernier, J. (1994). Auditory selective attention in the human cochlea, In: Brain Research 633(1-2): pp 353–356. DOI: 10.1016/0006-8993(94)91561-x

Heald, S. L. M., and Nusbaum, H. C., (2014). Speech perception as an active cognitive process In: Frontiers in Systems Neuroscience Vol.8 DOI: 10.3389/fnsys.2014.00035 PMCID: PMC3956139
URL: http://journal.frontiersin.org/article/10.3389/fnsys.2014.00035

Liberman, A.M., Cooper, F.S., Shankweiler, D.P., & Studdert-Kennedy, M. (1967). Perception of the speech code, In: Psychological Review. 74(6): pp 431-461. DOI: 10.1037/h0020279 PMID: 4170865

Liberman, A.M., & Mattingly, I.G. (1985). The motor theory of speech percepton revised, In: Cognition. 21(1): pp 1-36. DOI: 10.1016/0010-0277(85)90021-6 PMID: 4075760

Liberman, A.M., & Mattingly, I.G. (1989). A specialization for speech perception, In: Science 243(4890): pp 489-494. DOI: 10.1126/science.2643163 PMID: 2643163

MacKay, D. M. (1951). Mind-like Behaviour in artefacts, In: The British Journal for the Philosophy of Science 3(12): pp 352-353 DOI: 10.1093/bjps/II.6.105

MacKay, D. M. (1956). The epistemological problem for automata, In: Automata Studies (AM34), eds C. E. Shannon and J. McCarthy (Princeton: Princeton University Press): pp 235-252
ISBN-13: 978-0691079165 Retrieved from http://www.jstor.org/stable/j.ctt1bgzb3s.14

Moran, J., and Desimone, R. (1985). Selective attention gates visual processing in the extrastriate cortex, In: Science 229(4715): pp 782–784. DOI: 10.1126/science.4023713

Nusbaum, H. C., and Schwab, E. C. (1986). The role of attention and active processing in speech perception, In: Pattern Recognition by Humans and Machines: Speech Perception (Vol. 1), eds E. C. Schwab and H. C. Nusbaum (San Diego: Academic Press): pp 113–157.
DOI: 10.1016/B978-0-12-631403-8.50009-6

Spinelli, D. N., and Pribram, K. H. (1966). Changes in visual recovery functions produced by temporal lobe stimulation in monkeys, In: Electroencephalography and Clinical Neurophysiology 20(1): pp 44–49. DOI: 10.1016/0013-4694(66)90139-8

Answer 2

While speech is certainly sound, the brain will not always recognize speech sounds as speech. This is clear when one constructs speech from just sine waves as it reveals how the brain goes about understanding meaningfully linguistic signals.

Sine wave speech is constructed by using the first 3 prominent frequency bands (ie. formants) in a speech signal and replacing them with time-varying sine wave components. Cognitively when a listener is faced with sine wave speech without knowledge that it is speech, he or she often fails to understand it. More shockingly individuals often fail to hear it as an example of speech. Indeed, unless you have the expectation that it is speech, it is often mistaken as birds whistling or computer generated nonsense sounds. However, you can engender understanding with the knowledge that it is language. Presumably this is because the expectation that is it speech, allows a listener to bring to bear the vast amount of knowledge they possess about language to interpret the sinewave speech signal. The idea that expectations play a large role in speech understanding is supported by the view of speech perception held by Heald & Nusbaum (2014):

...even in early stages, [speech perception] is an active process in which speech analysis is attentionally guided. Note that this does not mean consciously guided but that information-contingent changes in early auditory encoding can occur as a function of context and experience. Active processing assumes that attention, plasticity, and listening goals are important in considering how listeners cope with adverse circumstances that impair hearing by masking noise in the environment or hearing loss. Although theories of speech perception have begun to incorporate some active processing, they seldom treat early speech encoding as plastic and attentionally guided. Recent research has suggested that speech perception is the product of both feedforward and feedback interactions between a number of brain regions that include descending projections perhaps as far downstream as the cochlea...

Here are helpful online resources to understand what is cognitively occurring in speech understanding broadly as well as in the context of sine wave speech:

http://www.haskins.yale.edu/featured/sws/sws.html https://www.mrc-cbu.cam.ac.uk/people/matt.davis/sine-wave-speech/