[PPT] - Mandarin and English Zhihua Xia, Rivka Levitan, Julia Hirschberg PowerPoint Presentation

SLIDE 1

Prosodic Entrainment in Mandarin and English

Zhihua Xia, Rivka Levitan, Julia Hirschberg

SLIDE 2

Entrainment

2

SLIDE 3

Entrainment

3

SLIDE 4

Entrainment

4

SLIDE 5

Entrainment

5

SLIDE 6

Entrainment

6

SLIDE 7

Entrainment

7

SLIDE 8

Motivation

Prevalence of entrainment

– Lexical, syntactic, stylistic – Phonetic – Intensity, pitch, speaking rate

Liking, positive social characteristics

– Liking, smoother interaction – Task score – Positive interactions between married couples

Lack of cross-linguistic comparisons

8

SLIDE 9

Columbia Games Corpus

9

SLIDE 10

Columbia Games Corpus

12 spontaneous task-oriented sessions
13 participants

– 11 participated in two sessions with different partners.

9h 8m of recorded dialogue
Orthographically transcribed, word-aligned, intonationally transcribed,

annotated for turn-taking behaviors and other phenomena

Acoustic features automatically extracted using Praat

10

SLIDE 11

Tongji Games Corpus

115 spontaneous task-oriented sessions
70 pairs of speakers (40 female, 30 male)
University students with a National Mandarin Test Certificate level 2,

grade A or above

Elicited using two games: Picture Ordering (role imbalance), Picture

Classifying (cooperative)

12 hours of recorded dialogue

11

SLIDE 12

Inter-pausal unit (IPU) Pause-free segment of speech from a

single speaker.

speech <silence> speech <silence> speech

Turn Sequence of speech from one speaker without

intervening speech from the other speaker.

Session Complete interaction between two subjects.

12

Units of analysis

SLIDE 13

Inter-pausal unit (IPU) Pause-free segment of speech from a

single speaker.

speech <silence> speech <silence> speech

Turn Sequence of speech from one speaker without

intervening speech from the other speaker.

Session Complete interaction between two subjects.

13

IPU IPU IPU

Units of analysis

SLIDE 14

Features

Intensity mean
Intensity max
Intensity min
F0 mean
F0 max
F0 min
Speaking rate

14

SLIDE 15

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

15

SLIDE 16

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

16

SLIDE 17

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

17

SLIDE 18

Global entrainment

18

SLIDE 19

Global entrainment

19

SLIDE 20

Global entrainment

20

SLIDE 21

Global entrainment

21

SLIDE 22

Global entrainment

22

Partner difference

SLIDE 23

Global entrainment

23

Other difference

SLIDE 24

Global entrainment

24

Self difference

SLIDE 25

Global entrainment

25

Partner difference

SLIDE 26

Local entrainment

26

IPU ti IPU ti+1 IPU tj IPU tk IPU tl

SLIDE 27

Local entrainment

27

IPU ti IPU ti+1 IPU ti+2 IPU ti+3 IPU ti+4 IPU ti+1 IPU ti+2 IPU ti+3

SLIDE 28

Three aspects of entrainment:

Proximity: similarity between interlocutors, relative to

baseline.

Synchrony: turn by turn relative similarity
Convergence: increase in similarity over time

28

SLIDE 29

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

29

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ ✓ 0.63 Intensity max ✓ ✓ 0.55 (0.47) Intensity min 0.31

0.09

F0 mean 0.66 (0.28) 0.22 F0 max ✓ 0.61 (0.18) 0.24 F0 min 0.63

0.19

Speaking rate ✓ ✓ (0.15)

Chinese

SLIDE 30

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

30

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 31

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

31

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 32

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

32

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 33

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

33

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 34

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

34

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 35

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

35

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 36

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

36

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 37

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

37

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 38

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

38

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 39

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

39

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 40

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

40

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 41

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

41

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 42

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

42

Feature Global proximity Local proximity Synchrony Global convergenc e Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 43

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

43

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 44

Acoustic-prosodic entrainment

Global or local?
Exact or relative?
Convergent or constant?

44

Feature Global proximity Local proximity Synchrony Global convergence Local convergence Intensity mean ✓ (✓) ✓ (✓) 0.63 (0.50) (✓) (0.03) Intensity max ✓ (✓) ✓ (✓) 0.55 (0.47) (0.02) Intensity min

0.31
0.09
F0 mean

(✓) 0.66 (0.28) (✓) 0.22 (0.06) F0 max ✓ (✓) (✓) 0.61 (0.18) 0.24 (0.05) F0 min

0.63
0.19
Speaking rate

✓ (✓) ✓ (✓) (0.15) (0.03)

Chinese (SAE)

SLIDE 45

Acoustic-prosodic entrainment

Mandarin Chinese

1. Evidence of entrainment on three main aspects of prosody
2. Local entrainment is more evident than the global entrainment.
3. Different prosodic features exhibit entrainment at different level.
4. Strong evidence of synchrony, especially F0
5. Some local convergence

Comparisons with SAE

1. Proximity

Global proximity: same as SAE Local proximity: not F0

2. Convergence

No global convergence Stronger local convergence.

3. Synchrony

Stronger synchrony

45

Discussion

SLIDE 46

Acoustic-prosodic entrainment

Mandarin Chinese

1. Evidence of entrainment on three main aspects of prosody
2. Local entrainment is more evident than the global entrainment.
3. Different prosodic features exhibit entrainment at different level.
4. Strong evidence of synchrony, especially F0
5. Some local convergence

Comparisons with SAE

1. Proximity

Global proximity: same as SAE Local proximity: not F0

2. Convergence

No global convergence Stronger local convergence.

3. Synchrony

Stronger synchrony

46

Discussion

SLIDE 47

Acoustic-prosodic entrainment

Mandarin Chinese

1. Evidence of entrainment on three main aspects of prosody
2. Local entrainment is more evident than the global entrainment.
3. Different prosodic features exhibit entrainment at different level.
4. Strong evidence of synchrony, especially F0
5. Some local convergence

Comparisons with SAE

1. Proximity

Global proximity: same as SAE Local proximity: not F0

2. Convergence

No global convergence Stronger local convergence.

3. Synchrony

Stronger synchrony

47

Discussion

SLIDE 48

Acoustic-prosodic entrainment

Mandarin Chinese

1. Evidence of entrainment on three main aspects of prosody
2. Local entrainment is more evident than the global entrainment.
3. Different prosodic features exhibit entrainment at different level.
4. Strong evidence of synchrony, especially F0
5. Some local convergence

Comparisons with SAE

1. Proximity

Global proximity: same as SAE Local proximity: not F0

2. Convergence

No global convergence Stronger local convergence.

3. Synchrony

Stronger synchrony

48

Discussion

SLIDE 49

Acoustic-prosodic entrainment

Mandarin Chinese

1. Evidence of entrainment on three main aspects of prosody
2. Local entrainment is more evident than the global entrainment.
3. Different prosodic features exhibit entrainment at different level.
4. Strong evidence of synchrony, especially F0
5. Some local convergence

Comparisons with SAE

1. Proximity

Global proximity: same as SAE Local proximity: not F0

2. Convergence

No global convergence Stronger local convergence.

3. Synchrony

Stronger synchrony

49

Discussion

SLIDE 50

Acoustic-prosodic entrainment

Mandarin Chinese

1. Evidence of entrainment on three main aspects of prosody
2. Local entrainment is more evident than the global entrainment.
3. Different prosodic features exhibit entrainment at different level.
4. Strong evidence of synchrony, especially F0
5. Some local convergence

Comparisons with SAE

1. Proximity

Global proximity: same as SAE Local proximity: not F0

2. Convergence

No global convergence Stronger local convergence.

3. Synchrony

Stronger synchrony

50

Discussion

SLIDE 51

Acoustic-prosodic entrainment

Mandarin Chinese

1. Evidence of entrainment on three main aspects of prosody
2. Local entrainment is more evident than the global entrainment.
3. Different prosodic features exhibit entrainment at different level.
4. Strong evidence of synchrony, especially F0
5. Some local convergence

Comparisons with SAE

1. Proximity

Global proximity: same as SAE Local proximity: not F0

2. Convergence

No global convergence Stronger local convergence.

3. Synchrony

Stronger synchrony

51

Discussion

SLIDE 52

Acoustic-prosodic entrainment

Mandarin Chinese

1. Evidence of entrainment on three main aspects of prosody
2. Local entrainment is more evident than the global entrainment.
3. Different prosodic features exhibit entrainment at different level.
4. Strong evidence of synchrony, especially F0
5. Some local convergence

Comparisons with SAE

1. Proximity

Global proximity: same as SAE Local proximity: not F0

2. Convergence

No global convergence Stronger local convergence.

3. Synchrony

Stronger synchrony

52

Discussion

SLIDE 53

Acoustic-prosodic entrainment

Mandarin Chinese

1. Evidence of entrainment on three main aspects of prosody
2. Local entrainment is more evident than the global entrainment.
3. Different prosodic features exhibit entrainment at different level.
4. Strong evidence of synchrony, especially F0
5. Some local convergence

Comparisons with SAE

1. Proximity

Global proximity: same as SAE Local proximity: not F0

2. Convergence

No global convergence Stronger local convergence.

3. Synchrony

Stronger synchrony

53

Discussion

SLIDE 54

Acoustic-prosodic entrainment

Mandarin Chinese

1. Evidence of entrainment on three main aspects of prosody
2. Local entrainment is more evident than the global entrainment.
3. Different prosodic features exhibit entrainment at different level.
4. Strong evidence of synchrony, especially F0
5. Some local convergence

Comparisons with SAE

1. Proximity

Global proximity: same as SAE Local proximity: not F0

2. Convergence

No global convergence Stronger local convergence.

3. Synchrony

Stronger synchrony

54

Discussion

SLIDE 55

Entrainment and gender differences

Theories

Dominance – Male dominance hypothesis – Communication Accommodation Theory Perception – Female perceptual sensitivity – Perception-behavior link

Predictions

– In mf conversations, females entrain more. – More entrainment in FF conversations than MM conversations.

Previous work

– Bilous and Krauss (1988) – Namy et al. (2002) – Pardo (2006)

55

SLIDE 56

Entrainment and gender differences

Theories

Dominance – Male dominance hypothesis – Communication Accommodation Theory Perception – Female perceptual sensitivity – Perception-behavior link

Predictions

– In MF conversations, females should entrain more. – There should be more entrainment in FF conversations than MM conversations.

Previous work

– Bilous and Krauss (1988) – Namy et al. (2002) – Pardo (2006)

56

SLIDE 57

Entrainment and gender differences

Theories

Dominance – Male dominance hypothesis – Communication Accommodation Theory Perception – Female perceptual sensitivity – Perception-behavior link

Predictions

– In MF conversations, females entrain more. – More entrainment in FF conversations than MM conversations.

Previous work

– Bilous and Krauss (1988) – Namy et al. (2002) – Pardo (2006)

57

SLIDE 58

Entrainment and gender differences

58

ENTp = - speaker1f -speaker2 f ENT

x = -

speaker1f - Xi, f

i

å

X

X = set of speakers of the same gender and role as speaker1’s partner who are never paired with speaker1.

SLIDE 59

Entrainment and gender differences

59

ENTp = - speaker1f -speaker2 f ENT

x = -

speaker1f - Xi, f

i

å

X

X = set of speakers of the same gender and role as speaker1’s partner who are never paired with speaker1.

SLIDE 60

Entrainment and gender differences

60

ENTp = - speaker1f -speaker2 f ENT

x = -

speaker1f - Xi, f

i

å

X

X = set of speakers of the same gender and role as speaker1’s partner who are never paired with speaker1.

SLIDE 61

Entrainment and gender differences

61

Feature FF MM FM MC SAE MC SAE MC SAE Intensity mean ✓ ✓ ✓ ✓ ✓ Intensity max ✓ ✓ ✓ ✓ ✓ Intensity min

F0 mean

✓ ✓ F0 max ✓ ✓ F0 min

Speaking rate

✓ ✓ ✓ ✓ ✓ ✓

SLIDE 62

Entrainment and gender differences

62

Feature FF MM FM MC SAE MC SAE MC SAE Intensity mean ✓ ✓ ✓ ✓ ✓ Intensity max ✓ ✓ ✓ ✓ ✓ Intensity min

F0 mean

✓ ✓ F0 max ✓ ✓ F0 min

Speaking rate

✓ ✓ ✓ ✓ ✓ ✓

SLIDE 63

Entrainment and gender differences

63

Feature FF MM FM MC SAE MC SAE MC SAE Intensity mean ✓ ✓ ✓ ✓ ✓ Intensity max ✓ ✓ ✓ ✓ ✓ Intensity min

F0 mean

✓ ✓ F0 max ✓ ✓ F0 min

Speaking rate

✓ ✓ ✓ ✓ ✓ ✓

SLIDE 64

Entrainment and gender differences

64

Feature FF MM FM MC SAE MC SAE MC SAE Intensity mean ✓ ✓ ✓ ✓ ✓ Intensity max ✓ ✓ ✓ ✓ ✓ Intensity min

F0 mean

✓ ✓ F0 max ✓ ✓ F0 min

Speaking rate

✓ ✓ ✓ ✓ ✓ ✓

SLIDE 65

Entrainment and gender differences

MM << FF << MF
Similar patterns for Mandarin Chinese (MC) and Standard American English

(SAE)

65

SLIDE 66

Conclusions

Entrainment in MC on three main aspects of prosody
Strong evidence of dynamic entrainment, especially on F0
Gender differences: MM << FF << FM
Consistencies in pattern with SAE

66

SLIDE 67

Conclusions

Entrainment in MC on three main aspects of prosody
Strong evidence of dynamic entrainment, especially on F0
Gender differences: MM << FF << FM
Consistencies in pattern with SAE

67

SLIDE 68

Conclusions

Entrainment in MC on three main aspects of prosody
Strong evidence of dynamic entrainment, especially on F0
Gender differences: MM << FF << FM
Consistencies in pattern with SAE

& differences in pattern with SAE

68

SLIDE 69

Conclusions

Entrainment in MC on three main aspects of prosody
Strong evidence of dynamic entrainment, especially on F0
Gender differences: MM << FF << FM
Remarkable consistencies in pattern with SAE

69

SLIDE 70

Thanks

This work was supported in part by

– NSF IIS-0803148 – Humanities and Social Sciences Foundation of China Education Ministry (11YJA740113).

Many thanks to

– Stefan Benus, Agus Gravano, Daniel Hirst, Qiuwu Ma, and Fengying Mu for useful comments – Yuanyuan Zhang, Jin Peng, Chenxiao Zou, Yalu Chen, Li Jiao, who labeled the Tongji Games Corpus.

SLIDE 71

Thank you !