Contact-less Based Accelerometers Speaker: Refael Shamir Founder - - PowerPoint PPT Presentation

Affective Categorization Using Contact-less Based Accelerometers

March 26—29, 2018 | Silicon Valley | #GTC18

www.gputechconf.com

Speaker: Refael Shamir Founder and CEO of Letos

Motivation

– Driver monitoring in the new age

Background and Definitions

– First steps to understanding Affect Categorization

Technology Review

– Facial Expressions

– Eye Tracking

– Voice Recognition – Wearable Monitoring

– EEG, ECG, GSR, PPG…

– Sentiment Analysis

Current State of the Art – Gap and Challenges Introducing Letos – How, When and Where

Presentation Outline

Misconception?

Motivation

– Driver monitoring in the new age

Background and Definitions

– First steps to understanding Affect Categorization

Technology Review

– Facial Expressions

– Eye Tracking

– Voice Recognition – Wearable Monitoring

– EEG, ECG, GSR, PPG…

– Sentiment Analysis

Current State of the Art – Gap and Challenges Introducing Letos – How, When and Where

Presentation Outline

Motivation

Motivation

‒ There is a growing debate on the tracking of in-cabin monitoring (e.g. tracking alertness) ‒ Using gaze estimation is only part of the solution

‒ Keeping eyes on the road, does not proclaim alertness level with a good confidence

‒ Need to track engagement level of the driver at all times

Motivation

– Driver monitoring in the new age

Background and Definitions

– First steps to understanding Affect Categorization

Technology Review

– Facial Expressions

– Eye Tracking

– Voice Recognition – Wearable Monitoring

– EEG, ECG, GSR, PPG…

– Sentiment Analysis

Current State of the Art – Gap and Challenges Introducing Letos – How, When and Where

Presentation Outline

Background and Definitions

Background and Definitions – Cnt’d

Affective Computing: Picard first introduced the term “affective computing” in 1995, as a mean to evaluate different emotions, or expressions, from a computer perspective

Affective Computing (MIT Press); Rosalind

• W. Picard, 1997

Arousal (Y-axis) – Indicates excitement/engagement level Valence (X-axis) – Indicates pleasure/comfort level

The Dimensional Affective State Model

Positive Negative Low High

I II III IV

ANGER SADNESS FEAR JOY SURPRISE DISGUST RELIEF

I II III IV IV ???

Motivation

– Driver monitoring in the new age

Background and Definitions

– First steps to understanding Affect Categorization

Technology Review

– Facial Expressions

– Eye Tracking

– Voice Recognition – Wearable Monitoring

– EEG, ECG, GSR, PPG…

– Sentiment Analysis

Current State of the Art – Gap and Challenges Introducing Letos – How, When and Where

Presentation Outline

Tools for Evaluating Affective States

Suggested Prototype for Auto-Classification

Pre- Processing & Object Detection Feature Extraction

Input Source

Post- Processing Training Model Feature Selection Classifier

Output

Ekman’s Model

Paul Ekman argues that there are 6 basic facial expressions which are uniquely distinguished from one another, that have a relationship with an emotional state (Ekman, 1972) These set of emotions, according to Ekman, are being expressed across humans, regardless of age, gender, race, or culture

Facial Expressions

36 21

Feature Extraction for Affect Classification

Geometric Features

– Detecting the face (shape/size) – Detect cue points (lips, eyebrow) → Categorize emotions based on relative position to the face

Appearance Based

– Detecting the face (shape/size) – Texture layering (filters) → Categorize emotions based on extracted feature type

Technology Analysis

Capturing the duration of the emotion

0.5 – 4 seconds

Positive and negative differentiation Spontaneous reaction (faking?) Evaluating intensity level (arousal)

Valence Positive Negative

Confidence Can still be solved using a camera!

Human Eye

Pupil Iris Eyelashes Eyebrows Sclera Eyelids

How Sherlock Does It

https://www.youtube.com/v/- bBHT158E0s?start=107&end=128

Pupillary Response - Explained

2mm 8mm

During rest, the eye’s pupil usually constricts, due to parasympathetic activity When presenting a stimuli, the eye’s pupil tends to dilate, due to sympathetic activity

(Bradley, Miccoli, Escrig, & Lang, 2008)

References and Further Reading

1.Li and Jain. Handbook of Face Recognition, 2nd Edition. New York: Springer, 2011. Print 2.R. Cowie, E. Douglas Cowie, N. Tsapatsoulis, S. Kollias, W. Fellenz, and J. G. Taylor. “Emotion Recognition in Human Computer Interaction”. IEEE Signal Processing Magazine, 18 (2001): 32-80. Print 3.Open source tool: “Free and open source face recognition with deep neural networks”. OpenFace. GitHub (accessed February, 2018)

Human Speech – Overview

‒ Speech is basically a stream of words spoken in a particular way ‒ In order to differentiate between different syllables the vocal cords vibrates, and sound is sequentially being filtered through the mouth and nose ‒ In general, speech is carried over an anchor frequency (which varies within different scenarios). This is often abbreviated as ‘F0’

https://www.youtube.com/watch?v=yxxRAHVtafI

Speech Recognition - Background

Human speech can be modeled through differentiating what is being transmitted during the message, and what is its intended affect Primary Secondary

“This upsets me” “That’s so funny” “I’m so happy”

Feature Extraction for Affect Classification

Pitch: Usually, compared to the base frequency F0 Voice/Volume Level: Higher levels might indicate anger or fear Speaking rate: Can indicate speaker’s confidence level

Generally, voice indicates merely arousal level

Speech to Text

Source: https://hacks.mozilla.org/2017/11/a-journey-to-10-word-error-rate/

meaning

• pportunity

mar arket mar arket mar arket communicati tion Bus usin iness bu busin iness smilin ing ded edic icati tion fi financia ial suc uccessful am ambi bitio ious

de development

em emplo loyment

co corporation

challe llenge wor

• rking

na navigation

moti tivati tion

conference ce individ idual connect ction en enterprise se cust stomers prosp sperity com

• mple

lexity com

• mpetit

itor colle llect ctive une nemployment inf nformatio ion te tech chnolo

• logy

par artn tners

conce centration

• n

ag agreement skyscr craper deci ecisi sions

idea eas

det etermina nation conce centration

• n

at attr tract ctive chall llenge diff ifferent at attr tract ctive at attr tract ctive confi fident

women

contemporary det etermina nation

te team copy

corp

• rporation

individ idual as asso soci ciate as assi sist stance ce

computa tati tion

depressi ssion com

• mple

lexity com

• mple

lexity

ins nspir ire

professi ssional int nteracti ction ad adversity ty begin beginnin ing Busi usiness ss wom

• men

Busi usiness ss man man com

• mple

lexity

smilin ing

sel elective

cust stomers

road

Ent ntrepreneur

• rganization
• n

part artnershi hip pers erspecti ctive ef effect ctive pers ersist stence ce

investment

Sentiment Orientation

Sentiment (or text) analysis can basically infer positive and negative – i.e. valence – opinions which people express either through voice or in writing

Not So Fast...

‒ Not all expressions have a single meaning

• e.g. ”That’s so funny” – positive sentence; different meaning with

use of tone (sarcasm)

Intensity: Rating:

WEAK MEDIUM HIGH Good Wonderful Amazing Bad Poor Terrible

Linguistics as a mean for Classifying Emotions

Guidelines:

“This upsets me” “That’s so funny” “I’m happy that you’re here”

Intensifier

Usage of bad language [Cursing, Insulting, Blaming, etc.]

References and Further Reading

• 1. Johar, Swati. Emotion, Affect and Personality in Speech. Springer, 2016. Print
• 2. R. Cowie, E. Douglas Cowie, N. Tsapatsoulis, S. Kollias, W. Fellenz, and J. G. Taylor.

“Emotion Recognition in Human Computer Interaction”. IEEE Signal Processing Magazine, 18 (2001): 32-80. Print

• 3. Open source tool:

https://sourceforge.net/projects/openart/. Open source project named openEAR (originated at TUM)

• 4. Knowledge center: http://sentic.net (Accessed on March, 2018)

Different Types of Monitoring

Autonomous Nervous System (ANS)

Source: Hemmings, H., Pharmacology and Physiology for Anesthesia: Foundations and Clinical

• Application. Saunders, 2013

Sympathetic “fight-or-flight” Parasympathetic “rest and digest”

ANS – Continued

Sympathetic Nervous System activity: Parasympathetic Nervous System activity:

Human Heart

Heart Rate Measurement

A person heart rate can be extracted through either an ECG,

• r a PPG (usually smart watches nowadays)

Heart Rate Variability

‒ Heart rate variability (HRV) is the variation of consecutive beat- to-beat (b2b) intervals ‒ It indicates the heart's ability to respond to stimuli such as breathing, exercise, stress, diseases or sleep ‒ Decreased with SNS; Increased with Parasympathetic

Presentation Outline

Technology Overview

– Companies; Market; Use Cases

Background and Definitions

– First steps to understanding Affect Categorization

Technology Review

– Facial Expressions – Voice Recognition – Wearable Monitoring Devices

– EEG, ECG, GSR, PPG…

– Sentiment Analysis

Current State of the Art – Gap and Challenges Introducing Letos – How, When and Where

Current State of the Art

NEGATIVE NEUTRAL POSITIVE Currently, most commercial products gives merely a differentiation between Positive and Negative emotions

SADNESS ANGER MISERY CALM JOY HAPPINESS ENTHUSIASM

STRESS!

Challenges

Self Assessment: Multi-Modal Approach: Spontaneous; Unobtrusive Awareness

Presentation Outline

Technology Overview

– Companies; Market; Use Cases

Background and Definitions

– First steps to understanding Affect Categorization

Technology Review

– Facial Expressions – Voice Recognition – Wearable Monitoring Devices

– EEG, ECG, GSR, PPG…

– Sentiment Analysis

Current State of the Art – Gap and Challenges Introducing Letos – How, When and Where

Recognizing anonymously user emotional reaction within different scenarios

Solution

Advanced Machine Learning techniques for performing affective classification based on physiological signals 5 Emotional States ML/DL for Classifying Emotional States Complete Anonymity

Product

Product – Features

Heart Rate; Heart Rate Variability Respiration Rate Contact-less single sensor solution

MEMS

1 Sample/Second Wireless Communication

Demonstration

Demonstration

Respiration Rate Heart Rate

Ballistocardiography

Ballistocardiography is a non-invasive method based on the measurement of the body motion generated by the ejection of the blood at each cardiac cycle. It is one of the many methods relying on detection of cardiac and cardiovascular-related mechanical motions, such as phonocardiography, apexcardiography, seismocardiography, kinetocardiography to list just a few.

http://people.csail.mit.edu/balakg/pulsefromheadmotion.html

www.letos.me

Thank You

info@letos.me