ANR-CREST 2109 TAPAS: Training Adapted Personalised Affective Social Skills with Cultural Virtual Agents JP: PI Satoshi Nakamura, Ph.D (NAIST) Hiroki Tanaka, Ph. D (NAIST) Takashi Kudo, M.D (Osaka University) Hidemi Iwasaka, M.D (Nara Medical University) FR: PI Catherine Pelachaud, Ph.D (CNRS-ISR, Sorbonne U) Jean-Claude Martin, Ph.D (CNRS-LIMSI)
Research Topics at AHC-lab, NAIST Brain Analysis Incongruity measurement Cognitive Load EEG Hyper Scanning Speech Translation Machine Translation Multi-language ASR, TTS Data Analytics Machine Speech Chain Caption Generation Affective Computing Natural Language Why don’t you Processing ① Social Skills Training join our lab! Goal-oriented Dialog ② Detection of Early Dementia Non goal-oriented Dialogue I’m looking for a lab. Deep Neural Spoken Dialogue Network Multi-modal Dialogue ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 2 2019/12/02
1. Social Skills Training Training tools for autism spectrum disorders People with autism have good systemizing skills Speaking skills [Tanaka, et al., 2015] and listening skills [Tanaka, et al., 2019] ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 3 2019/12/02
Role-play Analyze behaviors of users and generate avatar actions Features based on [Tanaka et al., 2014] ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 2019/12/02 4
Summary feedback After role-play, the system displays feedback Repetitive training until mastery ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 2019/12/02 5
Training effect One-way ANOVA: (F[2,24]=4.70, p<.05) Post hoc comparisons with Bonferroni’s method: significantly different between feedback and reading book (p<0.05) ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 2019/12/02 6
Human-human and human-agent interaction ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 7 2019/12/02
2. Detection of early dementia 10 minutes interaction: 12 early stage of dementia and healthy control [Ujiro et al., 2018] [Tanaka et al., 2017] ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 8 2019/12/02
Multimodal-based classification Answers to fixed questions: e.g. what is the date today? Areas under the ROC curve – 0.90 (SVM), 0.88 (Logistic regression) Features: response gap, language, speech, face ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 9 2019/12/02
ANR-CREST (2019-2024) TAPAS: Training Adapted Personalised Affective Social Skills with Cultural Virtual Agents JP: PI Satoshi Nakamura, Ph.D (NAIST) Hiroki Tanaka, Ph. D (NAIST) Takashi Kudo, M.D (Osaka University) Hidemi Iwasaka, M.D (Nara Medical University) FR: PI Catherine Pelachaud, Ph.D (CNRS-ISR, Sorbonne U) Jean-Claude Martin, Ph.D (CNRS-LIMSI)
Research background and objectives Increase of people who are not good at social Public speech: Feel nervous and stress for communication and social anxiety disorder in attentions by others, and Nervous, schools and workplace (3 to 13%) bad reputation shameful ↓ Face red, sweat, shaking, Try to solve by verbal / non-verbal interactive pulsation, breathing, stomachache training system by Embodied Conversational Agent – Target population: general population , depression, Fear for the symptom autism spectrum disorders, schizophrenia Social anxiety, avoidance – Behavioral training: apply Social Skills Training (SST) – Cognitive training: apply Cognitive Behavioral Social anxiety disorder Therapy (CBT) ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 11 2019/12/02
Overall framework Public speaking training Nakamura G NAIST Situation & immersion Teenagers, children with ASD Satoshi Nakamura ， Hiroki Tanaka WP1 Hirokazu Kato ， Yuichiro Fujimoto Theories of social interaction Through human-agent interaction (Image: the verge) No stress, high self-efficacy VR/AR/Display Learn appropriate behaviors one to one one to multi-party Iwasaka G Nara Med. Univ. Technical approach France Japan Hidemi Iwasaka ， Yasuhiro Matsuda, (NAIST) Agents integration, translation Virtual agent platform Kosuke Okazaki Agent ’ s eye gaze, attitude (e.g. cooperative) Speech recognition Kudo G Osaka Univ. WP3 Language processing Takashi Kudo ， Hiroyoshi Adachi Dialog modeling Yukako Sakagami Body movement WP4 Chat-based (ISIR) Facial expression Pelachaud G （ France CNRS-ISIR ） Behavioral Cognitive Online measuring Catherine Pelachaud training therapy WP2 & feedback Donatella Simonetti (Nara Med. Univ.) (Osaka Univ.) (CNRS-LIMSI) Multimodal behaviors SST CBT WP5 Evaluation MARTIN G (France CNRS-LIMSI ） qualitative (Image: psychology toolbox) AttrakDiff questionnaire WP4 Jean-Claude MARTIN Culture difference Anxiety state inventory Gathering therapy data quantitative Physiological responses Elise Prigent System improvement New scenarios Ouriel Grynszpan Data collection ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 12 2019/12/02
Social skills training (SST) Medically established method for behavior training for autism, and schizophrenia. Flow: Situation setting -> Modeling -> Role play -> Feedback -> Homework 1 period: 30 minutes or more x 10 times [Nara Med. Univ. 2018] Language: words, expression, Participant consistency, coherence, causality, Trainer etc. Speech: pitch, speaking speed, filler, voice quality, amplitude etc. Image: facial expression, gesture, posture, eye gaze etc. ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 13 2019/12/02
Cognitive behavioral therapy (CBT) e.g. thinking note (column method) Psychotherapy-based instructional method to Cog ognitive Th Therapy Reco cord rd ECA A Dialog ogue change the cognitive schema Poi ointed ou out my my sm small voi oice in Situation on prese sent ntation on, and w weakn kness ss Situation on breakd kdow own log ogicality by the a audienc nce. e. Cognitive Reconstruction, Behavioral Activation, Desc scribe be emot otion on In Initial sc scor ore Disc scou ourage ged （８０）、 Sad Sad （９０） state st Situation Analysis, Problem Solving, Progressive I am not I ot a able to o prese sent nt in fron ont Ne Nega gative Relaxation, Assertion, Schema Modification of audience. of nce. I I am not ot go good od for or this s Ext xtract tendency ency thinki king job ob. This study uses cognitive reconstruction Evidence ce for or My voi oice i is s sm small in ge genera ral. I I am Ext xtract evidenc nce this s thinki king not ot able t to o exp xplain l log ogically. – Situation setting -> Initial mood -> Automatic thinking There are m Th many peop ople who g o give Prov ovide infor ormation on -> Disapproval -> Adaptive thinking -> Change in Al Alternative a go good od prese sent ntation on in sm small voi oice. to o gu guide for or thinki king It It is s pos ossi sible to to prese sent nt lo logi gically if ext xtracting g cou ount nter er mood -> Analysis (homework) I prepare I e well in a advance. thinki king. g. – 1 period: 30 minutes or more (usually 4-50 minutes) x I I can gi give a go good od prese sent ntation on Change ge Ext xtract alternative even voi oice i is s sm small. I I will prepare e 16 times thou ough ghts thinki king mor ore in a advance Disc scou ourage ged （２０）、 Mot otivated （８ Reco cogn gnize ze chang nge e of of scor sc ore ０） thou ough ghts ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 14 2019/12/02
Summary Social Skill Training – SST for Autism Detection of early dementia New project – ANR-CREST: TAPAS project ! ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 15 2019/12/02
Continue to J-C Martin! ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 16 2019/12/02
Children with ASD Communicate to unfamiliar person Pre-post comparison （ p = 0.002, Cohen's d = 1.17) ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 17 2019/12/02
Cognitive reconstruction data Collecting interaction data between doctors and people with depression e.g. Beck & Beck cognitive therapy live session ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST 18 2019/12/02
Recommend
More recommend
