Speaker Movement Correlates with Prosodic Indicators of Engagement - - PowerPoint PPT Presentation

Speaker Movement Correlates with Prosodic Indicators of Engagement

Rob Voigt, Robert J. Podesva, and Dan Jurafsky

LINGUISTICS DEPARTMENT STANFORD UNIVERSITY

Links Between Acoustic and Visual Prosody

• Gestural apices align with pitch accents

Jannedy and Mendoza-Denton (2006)

• Production of “visual beats” increases the

prominence of the co-occurring speech

Krahmer and Swerts (2007)

• Speakers move their head and eyebrows more

during prosodically focused words

Cvejic et al. (2010)

• Previous research
• Identified discrete relationships
• Our proposal
• Examine scalar relationships
• Particularly between movement and

affective measures of engagement

Yu et al. (2004), Mairesse et al. (2007), Gravano et al. (2011), Oertel et al. (2011), MacFarland et al. (2013), etc.

Question 1: Is the Relationship Between Acoustic and Visual Prosody Continuous?

Question 2: Methodological Barriers to Studying Visual Prosody

• Prior studies generally employ
• Time-intensive annotation schemes
• r
• Expensive or invasive experimental hardware
• Thus face limitations
• Small amounts of data
• Prohibitive expense

Our Solution: New Data Source

Automatically extract visual and acoustic data from YouTube

• Potentially huge amounts of data
• Ecologically valid (“in the wild”)
• Allows replicability

Our Solution: New Data Source

• We chose “first day of school” video blogs

(“vlogs”)

• 14 videos, 95 minutes of data
• Static backgrounds and stable cameras
• Generally engaged, animated

Our Solution: Automatic Phrasal Units

Approximate pause-bounded units (PBUs)

• Our unit of prosodic analysis
• Calculated with a simple iterative algorithm
• Find silences (Praat) with a threshold of -30.0dB;

sounding portions are approximate PBUs

• If average phrase length > 2 seconds,

raise threshold by 3.0dB and re-extract

Our Solution: New Visual Feature

Movement Amplitude

• Assumes speaker is talking in front of a static

background

• Quantifies speaker movement as pixel-by-pixel

difference between frames

• Calculated in log space, z-scored per speaker

Visualization: Continuous Measurements

• Video at 30 FPS allows observations at 30 Hz

Visualization: Movement-Only Video

• Coarse, but reasonable overall estimation

Acoustic Features

• Following prior work on prosodic engagement
• Pitch (fundamental frequency) and

Intensity (loudness)

• Eight features per phrase
• max, min, mean, standard deviation (std)

for both pitch and intensity

Statistical Analysis

• Movement amplitude measures (max, min, mean, std) are

highly co-linear

• PCA for dimensionality reduction
• Two components explain 96% of variance in MA

Statistical Analysis

• Series of linear regressions
• Predicting acoustic variables from

OVERALL MOVEMENT and MOVEMENT VARIANCE

• Controlling for speaker-specific variation

by including speakers as random effects

• Controlling for log(phrase length)

Experimental Pipeline

• Download videos, extract frames and audio
• Calculate approximate phrase units (PBUs)
• Compute movement amplitude for each frame
• Calculate MA principal components
• Extract acoustic features
• Run statistical models

Results

** is p < 0.01, *** is p < 0.001, — is no significant relationship

Results

• During phrases with more OVERALL MOVEMENT,

speakers use

• higher and more variable pitch
• louder and more variable intensity
• MOVEMENT VARIANCE was not predictive of

any of our acoustic features

Visualization: Across Phrases

• Notice light and dark vertical banding
• Suggests sequence modeling as future work

Moving Forward

• More advanced vision-based features
• Face tracking
• Gesture recognition
• Expanding the data
• Genre effects
• Sociolinguistic variables
• Movement in interaction

Discussion

• Further empirical evidence for a rich link between

acoustic and visual prosody

• Adds dimension of quantity / continuous

association, in addition to previously demonstrated temporal synchrony

• Methodological contributions suggest new

avenues for multi-modal analysis of prosody

• Code and Corpus:

nlp.stanford.edu/robvoigt/speechprosody