Social Signal Processing: Understanding Nonverbal Communication in - - PowerPoint PPT Presentation

Social Signal Processing: Understanding Nonverbal Communication in Social Interactions

Alessandro Vinciarelli1,2 and Fabio Valente2

1University of Glasgow - Sir A.Williams Bldg, G12 8QQ (UK) 2IDIAP Research Institute - CP592 Martigny (Switzerland)

e-mail: vincia@dcs.gla.ac.uk, fvalente@idiap.ch

Outline

• Part I - What is SSP?

Slide 2 of 25

Outline

• Part I - What is SSP?
• Nonverbal behavior

Slide 2 of 25

Outline

• Part I - What is SSP?
• Nonverbal behavior
• Machine analysis of social behavior

Slide 2 of 25

Outline

• Part I - What is SSP?
• Nonverbal behavior
• Machine analysis of social behavior
• Part II - SSP in Action

Slide 2 of 25

Outline

• Part I - What is SSP?
• Nonverbal behavior
• Machine analysis of social behavior
• Part II - SSP in Action
• Analyzing conversations

Slide 2 of 25

Outline

• Part I - What is SSP?
• Nonverbal behavior
• Machine analysis of social behavior
• Part II - SSP in Action
• Analyzing conversations
• Roles, groups, stories and conflicts

Slide 2 of 25

Outline

• Part I - What is SSP?
• Nonverbal behavior
• Machine analysis of social behavior
• Part II - SSP in Action
• Analyzing conversations
• Roles, groups, stories and conflicts
• Part III - Future Perspectives

Slide 2 of 25

Outline

• Part I - What is SSP?
• Nonverbal behavior
• Machine analysis of social behavior
• Part II - SSP in Action
• Analyzing conversations
• Roles, groups, stories and conflicts
• Part III - Future Perspectives
• The SSPNet

Slide 2 of 25

Outline

• Part I - What is SSP?
• Nonverbal behavior
• Machine analysis of social behavior
• Part II - SSP in Action
• Analyzing conversations
• Roles, groups, stories and conflicts
• Part III - Future Perspectives
• The SSPNet
• Challenges ahead

Slide 2 of 25

Part I - What is SSP?

Slide 3 of 25

Social Signals and Social Behaviour

Our attention focuses on words, but we are immersed in a rich non-verbal world influencing not only the meaning of words, but also our perception of the social context.

Slide 4 of 25

Social Signals and Social Behaviour

forward posture forward posture vocal behaviour mutual gaze interpersonal distance Nonverbal Behavioural Cues height gesture

Our attention focuses on words, but we are immersed in a rich non-verbal world influencing not only the meaning of words, but also our perception of the social context.

Slide 5 of 25

Social Signals and Social Behaviour

Social Signal forward posture forward posture vocal behaviour mutual gaze interpersonal distance Nonverbal Behavioural Cues height gesture

Our attention focuses on words, but we are immersed in a rich non-verbal world influencing not only the meaning of words, but also our perception of the social context.

Slide 6 of 25

Nonverbal Communication

Nonverbal communications is based on nonverbal behavioural cues, codes, and functions.

Slide 7 of 25

Nonverbal Communication

clothes, attractiveness somatotype, etc. selftouching facial expression prosody, pitch, postural congruence, etc. gaze behaviour, etc. rythm, etc. distamce, seating Behavioural cues

Nonverbal communications is based on nonverbal behavioural cues, codes, and functions.

Slide 8 of 25

Nonverbal Communication

Physical Appearance Gestures Postures Face and Eyes Behaviour Vocal Behaviour Space Environment

clothes, attractiveness somatotype, etc. selftouching facial expression prosody, pitch, postural congruence, etc. gaze behaviour, etc. rythm, etc. distamce, seating Codes Behavioural cues

Nonverbal communications is based on nonverbal behavioural cues, codes, and functions.

Slide 9 of 25

Nonverbal Communication

Physical Appearance Gestures Postures Face and Eyes Behaviour Vocal Behaviour Space Environment

clothes, attractiveness somatotype, etc. selftouching facial expression prosody, pitch, postural congruence, etc. gaze behaviour, etc. rythm, etc. distamce, seating forming impressions deceiving and detecting deception sending messages of power and persasion managing interaction expressing emotion Behavioural cues Functions Codes sending relational messages

Nonverbal communications is based on nonverbal behavioural cues, codes, and functions.

Slide 10 of 25

Social Signal Processing

Data Capture Person Detection Multimodal Behavioural Streams Cues Behavioural Extraction Social Signals Understanding Context Understanding Behavioural Cues Social Behaviours Raw Data Preprocessing Multimodal Behavioural Streams Social Interaction Analysis

• A.Pentland, “Social Signal Processing”, IEEE Signal Processing Magazine,

24(4):108-111, 2007.

• A.Vinciarelli, M.Pantic, H.Bourlard, “Social Signal Processing: Survey of an

Emerging Domain”, Image and Vision Computing, 27(12):1743-1759, 2009.

Slide 11 of 25

Part II - SSP in Action

Slide 12 of 25

A Conversation Analysis Framework

[...] the most widely used analytic approach is based on an analogy with the workings of the market economy. In this market there is a scarce commodity called the floor which can be defined as the right to speak. Having control of this scarce commodity is called a turn. In any situation where control is not fixed in advance, anyone can attempt to get control. This is called turn-taking.

G.Yule,“Pragmatics”, Oxford University Press (1996)

Slide 13 of 25

Turn-Taking

t 1 t

• 2

t 3 t 4 t

• 5

t

• 6

t 7 t s =a

2 3

s

1

s3=a1 s4=a 3 s5=a 2 s6=a 1 s7=a 2 =a1

The turn taking pattern can be represented with a sequence S = {(s1, ∆t1, ), . . . , (sT, ∆tT)} where si ∈ A = {a1, . . . , aG} is a person label, and ∆ti the length of the ith turn.

• Among the most robustly detectable behavior evidences,
• but how far can we go by just modeling who talks when and

how much?

Slide 14 of 25

Role Recognition in Broadcast Material

• People play functional roles (Anchorman, Guest, etc.)
• Adjacent speakers are supposed to interact
• Around 85% of data time correctly labeled in terms of role

A.Vinciarelli, IEEE T-Multimedia, 9(6):1215-1226 (2007) H.Salamin et al., IEEE T-Multimedia, 11(7):1373-1380, 2009

Slide 15 of 25

Role Recognition in Meetings

x1= (1,1,1,1) x2= (0,0,1,1) x3= (1,1,1,0) w1 w2 w3 w4 a1

2

a a3 t 1 t

• 2

t 3 t 4 t

• 5

t

• 6

t 7 w1 w2 w3 w4 t s =a

2 3

t s

1

s3=a1 s4=a 3 s5=a 2 s6=a 1 s7=a 2 =a1 actors events

• Social networks suitable for functional roles, lexical analysis

suitbale for semantic ones

• Affiliation networks are suitable for small groups
• Around 75% of data time correctly labeled in terms of role

N.Garg et al., Proc. of ACM-Multimedia, pp. 693-696 (2008)

Slide 16 of 25

Roles and Prosody

t 1 t

• 2

t 3 t 4 t

• 5

t

• 6

t 7 t s =a

2 3

s

1

s3=a1 s4=a 3 s5=a 2 s6=a 1 s7=a 2 =a1

R∗ = arg max

R∈R p(R|X,

α)

• Conditional Random Fields allow the combination of

turn-taking and prosody

• Entropy of main prosodic features
• Results up to 89%, but combination does not always lead to

significant improvements

H.Salamin et al., Proc. of ACM-Multimedia, to appear (2010)

Slide 17 of 25

Story Segmentation

t 1 t

• 2

t 3 t 4 t

• 5

t

• 6

t 7 t s =a

2 3

s

1

s3=a1 s4=a 3 s5=a 2 s6=a 1 s7=a 2 =a1 Story 1 Story 2 Story 3

The problem consists of finding the following story sequence ˆ H: ˆ H = arg max

H∈H p(X|H)p(H)

(1)

• The purity is around 0.75
• Longer stories are better recognized

A.Vinciarelli et al., Proc. of ACM-Multimedia, pp. 261-264 (2007)

Slide 18 of 25

Conflict Analysis

t 1 t

• 2

t 3 t 4 t

• 5

t

• 6

t 7 t s =a

2 3

s

1

s3=a1 s4=a 3 s5=a 2 s6=a 1 s7=a 2 =a1

ˆ Q = arg max

Q∈Q πq1 N

• n=2

p(qn|qn−1) (2) where qi is one of the two groups or the moderator.

• People tend to react to someone they disagree with
• Groups correctly reconstructued in 66% of the cases (random

grouping has an average 6.5% performance).

A.Vinciarelli, IEEE Signal Processing Magazine, 26(5):133-136, 2009

Slide 19 of 25

Part III - Challenges Ahead

Slide 20 of 25

Open Issues and Challenges

• Getting psychology and engineering closer

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer
• SSP is inherently multidisciplinary

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer
• SSP is inherently multidisciplinary
• Mutual efforts of both disciplines

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer
• SSP is inherently multidisciplinary
• Mutual efforts of both disciplines
• Applying multimodal approaches

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer
• SSP is inherently multidisciplinary
• Mutual efforts of both disciplines
• Applying multimodal approaches
• Social signals are, by evolution, ambiguous

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer
• SSP is inherently multidisciplinary
• Mutual efforts of both disciplines
• Applying multimodal approaches
• Social signals are, by evolution, ambiguous
• Multimodal approaches are more robust to ambiguity

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer
• SSP is inherently multidisciplinary
• Mutual efforts of both disciplines
• Applying multimodal approaches
• Social signals are, by evolution, ambiguous
• Multimodal approaches are more robust to ambiguity
• Working on real-world data

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer
• SSP is inherently multidisciplinary
• Mutual efforts of both disciplines
• Applying multimodal approaches
• Social signals are, by evolution, ambiguous
• Multimodal approaches are more robust to ambiguity
• Working on real-world data
• Artificial settings are sometimes too simple

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer
• SSP is inherently multidisciplinary
• Mutual efforts of both disciplines
• Applying multimodal approaches
• Social signals are, by evolution, ambiguous
• Multimodal approaches are more robust to ambiguity
• Working on real-world data
• Artificial settings are sometimes too simple
• Social interactions are ubiquitous in many kinds of data

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer
• SSP is inherently multidisciplinary
• Mutual efforts of both disciplines
• Applying multimodal approaches
• Social signals are, by evolution, ambiguous
• Multimodal approaches are more robust to ambiguity
• Working on real-world data
• Artificial settings are sometimes too simple
• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer
• SSP is inherently multidisciplinary
• Mutual efforts of both disciplines
• Applying multimodal approaches
• Social signals are, by evolution, ambiguous
• Multimodal approaches are more robust to ambiguity
• Working on real-world data
• Artificial settings are sometimes too simple
• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications
• Applications link research to reality

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer
• SSP is inherently multidisciplinary
• Mutual efforts of both disciplines
• Applying multimodal approaches
• Social signals are, by evolution, ambiguous
• Multimodal approaches are more robust to ambiguity
• Working on real-world data
• Artificial settings are sometimes too simple
• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications
• Applications link research to reality
• Applications provide realistic benchmarks

Slide 21 of 25

The SSPNet

Universita’ di Roma Tre Queen’s University Belfast Unversity of Edinburgh Imperial College London University of Twente Delft University of Technology DFKI Idiap research institute University of Geneva CNRS

Slide 22 of 25

Human-Human and Human-Machine Interaction

SSP in HumanMachine Interaction

Behavior Analysis Behavior Modeling DFKI, CNRS, U. of Twente Delft, U. of Edinburgh Idiap, Imperial College, TU Queen’‚s U. Belfast, U. of Roma Tre, U. of Geneva Behavior Synthesis

SSP in HumanHuman Interaction

2 Research Foci: Human-Human and Human-Computer Interaction 3 Scientific Domains: Behavior Modeling, Analysis and Synthesis 5 Years to go: From 2009 to 2014

Slide 23 of 25

SSPNet: The Portal

The most tangible aspect of the SSPNet will be the web portal: http://www.sspnet.eu

• lowering the entry barrier of SSP, i.e. to reduce significantly

the effort required to start research in the domain,

• providing common benchmarks for rigorous performance

assessment and comparison between different approaches,

• disseminating literature, data and tools relevant to SSP.

Our ambition is to make of the portal THE reference for SSP.

Slide 24 of 25

Thank You!

Slide 25 of 25