SLIDE 1 Listening without hearing
Nadia M. Biassou MD, PhD
Senior Research Physician Dept of Radiology Division of Neuroradiology NIH Clinical Center and Senior Fellow Linguistics Data Consortium Graduate School of Arts and Sciences University of Pennsylvania
SLIDE 2
Financial Disclosure
l None
SLIDE 3
Learning Objectives
l The basics of the physics of speech l What is currently known about conscious
neurobiologic speech perception?
l Can unconscious speech perception by
reliably measured?
l What can its study tell us about the general
nature of speech perception and about the human brain that processes it?
SLIDE 4
Introduction
l What is speech and
why is it special?
SLIDE 5
Speech is the entryway to human linguistic communication
SLIDE 7
SLIDE 8
Formant frequency
l F0 is called the
fundamental frequency and represents the frequency of vocal cord oscillation
SLIDE 9
Formant frequencies
l Oscillation of vocal
cords and its harmonics
l F0
1
l F1
3
l F2
5
l F3
7
SLIDE 10
SLIDE 11
SLIDE 12
The speech waveform
l The production of any
sound during word production is simultaneously influenced by the sounds that precede and follow it.
l Liberman et al., 1957
SLIDE 13
Coarticulation of sounds
l “ebb” vs. “egg”
SLIDE 14
SLIDE 15
SLIDE 16
The speech spectrograms: formant frequency transitions
l The formant
frequencies transitions reflect coarticulation
SLIDE 17
Does the brain listen to every acoustic variation during speech perception?
SLIDE 18
Bottom Up processes
Bottom-up processing refers to processing sensory information as it is coming in
SLIDE 19
TASK
l PART 1: Actively decided whether real and
nonreal words are real words of English, half of the real and nonreal words are acoustically manipulated
SLIDE 20 STIMULI
l EXPERIMENT 1 l 40 REAL WORDS l HALF ARE
ACOUSTICALLY MANIPUALTED
l HALF ARE NON-
MANIPULATED
l 40 NONREAL WORDS l HALF ARE
ACOUSTICALLY MANIPULATED
l HALF ARE NON-
MANIPULATED
SLIDE 21 RESULTS
10 20 30 40 50 60 70 80 realwords nonwords
l The brain takes
74msecs longer to process the acoustically manipulated realwords, even though subjects could not consciously distinguish the word types
SLIDE 22 Sensory changes affect higher
BLUMSTEIN and colleagues
- LEXICAL DECISION TASKS IN WHICH LEXICAL
ITEMS WERE MANIPULATED ACOUSTIC GAP DETECTION (I.E. VOT) below the conscious level
- Sensory alteration can affect activation semantic priming and
lexical access.
- FARAH et al argue that words may also be stored with visual
associated information.
SLIDE 23 What are the neural networks that subserve subconscious processing of speech during conditions of increased effort?
l
LEFT INFERIOR FRONTAL CORTEX, ANTERIOR CINGULATE AND THALAMUS
l
POSTERIOR SUPERIOR TEMPORAL LOBES BILATERALLY
l
OCCIPITAL LOBES
l
LEFT CEREBELLUM
SLIDE 24 PART 2: Passively listen to real and nonreal words of English, half of which had been acoustically manipulated. STIMULI- PARTS 1 & 2 ARE MATCHED IN WORD FREQUENCY, WORD LENGTH, NUMBER OF SYLLABLES, AND IMAGEABILITY
l EXPERIMENT 2 l 40 REAL WORDS l HALF ARE
ACOUSTICALLY MANIPULATED
l HALF ARE NOT
MANIPULATED
l EXPERIMENT 2 l 40 NONREAL WORDS l HALF ARE
ACOUSTICALLY MANIPULATED
l HALF ARE NOT
MANIPULATED
SLIDE 25 Are the same networks activated in conditions of less effort?
l Activation in (b) posterior
superior temporal lobes and anterior cingulate are sufficiently robust even for the passive presentation of subconsciously manipulated realwords. But right frontal and right parietal lobe networks are activated
SLIDE 26
BUT IS SPEECH PERCEPTION ALL BOTTOM UP?
SLIDE 27 Top Down Processes
Visual Cues and Speech Perception McGuck Effect Baysan, U. (July 2017) "McGurk Effect" in F. Macpherson (ed.), The Illusions Index. Retrieved from https://www.illusionsindex.org/i/ mcgurk-effect.
SLIDE 28 McGuck Effect
BBC – Horizon: Is Seeing Believing Nov 2010
SLIDE 29 TOP DOWN PROCESSES
CONTEXT AND SPEECH PERCEPTION PHONEME RESTORATION (Warren & Warren 1970)
SLIDE 30 Phoneme Restoration Effect
Suboptimal environment input is
- verridden by context of speech
to hear stimuli that is in fact absent. “The State Governors met with their respective legislatures convening in the capitol city.” “The State Governors met with their respective le…latures convening in the capitol city.”
SLIDE 31 Ed Chang and colleagues
Leonard et al 2016, Nature Communications, 7:13619
SLIDE 32 Top Down Processes
Yanny vs Laurel Left: YANNY Right: LAUREL Middle spectrogram is a simulated ambiguous spectrogram – BUT listeners hear Yanny or Laurel
SLIDE 33 PERCEPTION is the point of contact between multisensory information:
BOTTOM UP (Objective)
l Processing of
sensory input
l Can affect higher
linguistic processes such as vision and semantics.
TOP DOWN (Subjective)
l Visual Input l Context l Linguistic
phonotactics (the language that you speak) can all affect the interpretation of sensory cues.
SLIDE 34
This point of contact is dynamic in time and in space
l Different neural networks can process the
same types of speech cues depending on the conditions under which the cues are being processed.
l Neural networks involved in processing
subconscious fine grain speech cues can involve the right hemisphere under passive listening (or lighter attentional load)
SLIDE 35
Attentional networks are always being recruited to varying degrees?
l Even for the passive listening of speech
cues.
SLIDE 36 We hear want we want to hear This doesn’t
cats or dogs!
SLIDE 37 Resting State fMRI
l Passive neural networks
may not be fully representative of the neural networks that subserve linguistic/cognitive processes because the network dynamics change depending on the attentional load to achieve the task at hand. It is NOT solely driven by the stimulus.
SLIDE 38
CLINICAL IMPLICATIONS
l Language mapping for
neurosurgery should reflect natural state of language processing as closely as possible including masked stimuli
SLIDE 39
New research
l Normal aging l Alzheimer’s and other neurodegenerative
disorders
l Autism l Is there a genetic basis for the balance
between objective and subjective speech perception?
SLIDE 40 Future research
l Can we develop new wearable technologies
that can diagnosis changes in the processing
- f sensory input in the preclinical stage of
disease?
SLIDE 41 l “Our imaginations are limited by the
knowledge that we currently possess”
- Helen Neville (IRCS Talk, University of
Pennsylvania, 1995)
SLIDE 42
l THANK YOU!
