TAPAS: Training Adapted Personalised Affective Social Skills with - - PowerPoint PPT Presentation

▶
tapas training adapted personalised affective
SMART_READER_LITE
LIVE PREVIEW

TAPAS: Training Adapted Personalised Affective Social Skills with - - PowerPoint PPT Presentation

Nov 30, 2023 132 likes •326 views

ANR-CREST 2109 TAPAS: Training Adapted Personalised Affective Social Skills with Cultural Virtual Agents JP: PI Satoshi Nakamura, Ph.D (NAIST) Hiroki Tanaka, Ph. D (NAIST) Takashi Kudo, M.D (Osaka University) Hidemi Iwasaka, M.D (Nara Medical

slide-1
SLIDE 1

ANR-CREST 2109 TAPAS: Training Adapted Personalised Affective Social Skills with Cultural Virtual Agents

JP: PI Satoshi Nakamura, Ph.D (NAIST) Hiroki Tanaka, Ph. D (NAIST) Takashi Kudo, M.D (Osaka University) Hidemi Iwasaka, M.D (Nara Medical University) FR: PI Catherine Pelachaud, Ph.D (CNRS-ISR, Sorbonne U) Jean-Claude Martin, Ph.D (CNRS-LIMSI)

slide-2
SLIDE 2

Research Topics at AHC-lab, NAIST

Speech Translation Machine Translation Spoken Dialogue Multi-modal Dialogue Why don’t you join our lab! I’m looking for a lab. Data Analytics Caption Generation Multi-language ASR, TTS Machine Speech Chain Deep Neural Network

Affective Computing

①Social Skills Training ②Detection of Early Dementia Natural Language Processing Goal-oriented Dialog Non goal-oriented Dialogue

Brain Analysis

Incongruity measurement Cognitive Load EEG Hyper Scanning

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

2

slide-3
SLIDE 3
  • 1. Social Skills Training

Training tools for autism spectrum disorders People with autism have good systemizing skills Speaking skills [Tanaka, et al., 2015] and listening skills [Tanaka, et al., 2019]

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

3

slide-4
SLIDE 4

Role-play

Analyze behaviors of users and generate avatar actions

4

Features based on [Tanaka et al., 2014]

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

slide-5
SLIDE 5

Summary feedback

5

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

After role-play, the system displays feedback Repetitive training until mastery

slide-6
SLIDE 6

Training effect

One-way ANOVA: (F[2,24]=4.70, p<.05) Post hoc comparisons with Bonferroni’s method: significantly different between feedback and reading book (p<0.05)

6

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

slide-7
SLIDE 7

Human-human and human-agent interaction

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

7

slide-8
SLIDE 8
  • 2. Detection of early dementia

10 minutes interaction: 12 early stage of dementia and healthy control

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

8

[Tanaka et al., 2017] [Ujiro et al., 2018]

slide-9
SLIDE 9

Multimodal-based classification

Answers to fixed questions: e.g. what is the date today? Areas under the ROC curve – 0.90 (SVM), 0.88 (Logistic regression)

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

9

Features: response gap, language, speech, face

slide-10
SLIDE 10

ANR-CREST (2019-2024)

TAPAS: Training Adapted Personalised Affective Social Skills with Cultural Virtual Agents

JP: PI Satoshi Nakamura, Ph.D (NAIST) Hiroki Tanaka, Ph. D (NAIST) Takashi Kudo, M.D (Osaka University) Hidemi Iwasaka, M.D (Nara Medical University) FR: PI Catherine Pelachaud, Ph.D (CNRS-ISR, Sorbonne U) Jean-Claude Martin, Ph.D (CNRS-LIMSI)

slide-11
SLIDE 11

Research background and objectives

Increase of people who are not good at social communication and social anxiety disorder in schools and workplace (3 to 13%)

11

Social anxiety disorder Public speech: Feel nervous and stress for attentions by others, and bad reputation ↓ Face red, sweat, shaking, pulsation, breathing, stomachache Fear for the symptom Social anxiety, avoidance Nervous, shameful

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

Try to solve by verbal / non-verbal interactive training system by Embodied Conversational Agent

– Target population: general population, depression, autism spectrum disorders, schizophrenia – Behavioral training: apply Social Skills Training (SST) – Cognitive training: apply Cognitive Behavioral Therapy (CBT)

slide-12
SLIDE 12

Overall framework

12

Nakamura G NAIST Satoshi Nakamura，Hiroki Tanaka Hirokazu Kato，Yuichiro Fujimoto Iwasaka G Nara Med. Univ. Hidemi Iwasaka，Yasuhiro Matsuda, Kosuke Okazaki Kudo G Osaka Univ. Takashi Kudo，Hiroyoshi Adachi Yukako Sakagami Pelachaud G （France CNRS-ISIR） Catherine Pelachaud Donatella Simonetti MARTIN G (France CNRS-LIMSI） Jean-Claude MARTIN

Elise Prigent Ouriel Grynszpan

France Japan Culture difference

Teenagers, children with ASD

WP4

VR/AR/Display

  • ne to one one to multi-party

Behavioral training Cognitive therapy

Agent’s eye gaze, attitude (e.g. cooperative) Multimodal behaviors

Through human-agent interaction

No stress, high self-efficacy Learn appropriate behaviors

Chat-based

AttrakDiff questionnaire Anxiety state inventory Physiological responses

(ISIR) (CNRS-LIMSI) (NAIST) (Nara Med. Univ.)

(Image: psychology toolbox)

Public speaking training Technical approach

(Image: the verge)

WP1 WP2 WP3 Gathering therapy data WP4 (Osaka Univ.) WP5 Situation & immersion Virtual agent platform Dialog modeling Online measuring & feedback

Language processing

Facial expression Body movement

Evaluation Agents integration, translation

SST CBT

Speech recognition

New scenarios System improvement

Theories of social interaction

Data collection

quantitative qualitative

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

slide-13
SLIDE 13

Social skills training (SST)

Medically established method for behavior training for autism, and schizophrenia. Flow: Situation setting -> Modeling -> Role play -> Feedback -> Homework 1 period: 30 minutes or more x 10 times [Nara Med. Univ. 2018]

13

Trainer Participant

Language: words, expression, consistency, coherence, causality, etc. Speech: pitch, speaking speed, filler, voice quality, amplitude etc. Image: facial expression, gesture, posture, eye gaze etc.

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

slide-14
SLIDE 14

Cognitive behavioral therapy (CBT)

Psychotherapy-based instructional method to change the cognitive schema Cognitive Reconstruction, Behavioral Activation, Situation Analysis, Problem Solving, Progressive Relaxation, Assertion, Schema Modification This study uses cognitive reconstruction

– Situation setting -> Initial mood -> Automatic thinking

  • > Disapproval -> Adaptive thinking -> Change in

mood -> Analysis (homework) – 1 period: 30 minutes or more (usually 4-50 minutes) x 16 times

14

e.g. thinking note (column method)

Cog

  • gnitive Th

Therapy Reco cord rd ECA A Dialog

  • gue

Situation

  • n

Poi

  • inted ou
  • ut my

my sm small voi

  • ice in

prese sent ntation

  • n, and w

weakn kness ss log

  • gicality by the a

audienc nce. e. Situation

  • n breakd

kdow

  • wn

In Initial sc scor

  • re Disc

scou

  • urage

ged（８０）、Sad Sad（９０） Desc scribe be emot

  • tion
  • n

st state Ne Nega gative thinki king I I am not

  • t a

able to

  • prese

sent nt in fron

  • nt
  • f
  • f audience.
  • nce. I

I am not

  • t go

good

  • d for
  • r this

s job

  • b.

Ext xtract tendency ency Evidence ce for

  • r

this s thinki king My voi

  • ice i

is s sm small in ge genera

  • ral. I

I am not

  • t able t

to

  • exp

xplain l log

  • gically.

Ext xtract evidenc nce Al Alternative thinki king Th There are m many peop

  • ple who g
  • give

a go good

  • d prese

sent ntation

  • n in sm

small voi

  • ice.

It It is s pos

  • ssi

sible to to prese sent nt lo logi gically if I I prepare e well in a advance. Prov

  • vide infor
  • rmation
  • n

to

  • gu

guide for

  • r

ext xtracting g cou

  • unt

nter er thinki king. g. Change ge thou

  • ugh

ghts I I can gi give a go good

  • d prese

sent ntation

  • n

even voi

  • ice i

is s sm

  • small. I

I will prepare e mor

  • re in a

advance Ext xtract alternative thinki king sc scor

  • re

Disc scou

  • urage

ged（２０）、Mot

  • tivated（８

０） Reco cogn gnize ze chang nge e of

  • f

thou

  • ugh

ghts

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

slide-15
SLIDE 15

Summary

Social Skill Training – SST for Autism Detection of early dementia New project

– ANR-CREST: TAPAS project !

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

15

slide-16
SLIDE 16

Continue to J-C Martin!

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

16

slide-17
SLIDE 17

Children with ASD

Communicate to unfamiliar person Pre-post comparison（p = 0.002, Cohen's d = 1.17)

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

17

slide-18
SLIDE 18

Cognitive reconstruction data

Collecting interaction data between doctors and people with depression e.g. Beck & Beck cognitive therapy live session

2019/12/02 ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

18