Dance Movement Patterns Recognition (Part I) 1 Israel Sol Cerdn - - PDF document

▶
dance movement patterns recognition part i
SMART_READER_LITE
LIVE PREVIEW

Dance Movement Patterns Recognition (Part I) 1 Israel Sol Cerdn - - PDF document

Sep 08, 2022 274 likes •678 views

Dance Movement Patterns Recognition (Part I) 1 Israel Sol Cerdn EEMCS - HMI Acknowledgments Acknowledgments This project would not has been possible without the This project would not has been possible without the

slide-1
SLIDE 1

EEMCS - HMI 1

Dance Movement Patterns Recognition (Part I)

Israel Solà Cerdán

slide-2
SLIDE 2

EEMCS - HMI 2

Acknowledgments Acknowledgments

  • This project would not has been possible without the

This project would not has been possible without the collaboration of our project directors: collaboration of our project directors:

– – Anton Anton Nijholt Nijholt – – Dennis Dennis Reidsma Reidsma – – Jordi Carrabina Jordi Carrabina

  • Thanks for the received aid of:

Thanks for the received aid of:

– – Herwin Herwin van van Welbergen Welbergen – – Ronald Ronald Poppe Poppe

  • Thanks to our family, friends, girlfriends

Thanks to our family, friends, girlfriends (MLG)

(MLG) and

and partners. partners.

/ 40

slide-3
SLIDE 3

EEMCS - HMI 3

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 40

slide-4
SLIDE 4

EEMCS - HMI 4

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-5
SLIDE 5

EEMCS - HMI 5

Presentation Presentation

  • Joint project

Joint project

  • The Virtual Dancer

The Virtual Dancer

  • Obtaining a greater interaction

Obtaining a greater interaction

  • Recognizing dance movement patterns

Recognizing dance movement patterns

/ 39 / 40

slide-6
SLIDE 6

EEMCS - HMI 6

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-7
SLIDE 7

EEMCS - HMI 7

Goals Goals

1.

  • 1. Starting up The Virtual Dancer application (computer vision part

Starting up The Virtual Dancer application (computer vision part and artificial intelligent part) and artificial intelligent part) 2.

  • 2. Analyzing the application and developing improvements

Analyzing the application and developing improvements strategies strategies 3.

  • 3. Recognition of simple dance movements

Recognition of simple dance movements 4.

  • 4. Recognition of dance movement patterns

Recognition of dance movement patterns 5.

  • 5. Automatic generation of dance movement search patterns

Automatic generation of dance movement search patterns 6.

  • 6. Finding patterns without a reference

Finding patterns without a reference

/ 39 / 40

slide-8
SLIDE 8

EEMCS - HMI 8

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-9
SLIDE 9

EEMCS - HMI 9

Organization of the Exposition Organization of the Exposition

  • Virtual Dancer application

Virtual Dancer application

  • The computer vision system

The computer vision system “ “ParleVision ParleVision” ”

  • The work

The work environment with the environment with the setup application and the used material setup application and the used material

  • The study carried out at the beginning of the project

The study carried out at the beginning of the project

  • The implemented modifications in the computer vision system

The implemented modifications in the computer vision system “ “ParleVision ParleVision” ”

  • Dance Movement Patterns Recognition (Part II)

Dance Movement Patterns Recognition (Part II)

– – HMM HMM – – Recognizing Simple/Complex Movements Recognizing Simple/Complex Movements – – Auto Generation of Complex Patterns Graphs Auto Generation of Complex Patterns Graphs – – Test Bench Test Bench – – Conclusions Conclusions

/ 39 / 40

slide-10
SLIDE 10

EEMCS - HMI 10

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-11
SLIDE 11

EEMCS - HMI 11

The Virtual Dancer The Virtual Dancer

/ 39 / 40

slide-12
SLIDE 12

EEMCS - HMI 12

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-13
SLIDE 13

EEMCS - HMI 13

Parlevision Parlevision

/ 39 / 40

slide-14
SLIDE 14

EEMCS - HMI 14

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-15
SLIDE 15

EEMCS - HMI 15

Material Material

  • Hardware

Hardware

– – 2 computers with the following properties 2 computers with the following properties

  • Pentium 4 2.8GHz

Pentium 4 2.8GHz

  • 512 MB of RAM

512 MB of RAM

  • Microsoft Windows XP with the SP 2

Microsoft Windows XP with the SP 2

– – 1 camera 1 camera

  • Philips ToUcam XS

Philips ToUcam XS

  • Software

Software

– – ParleVision ParleVision – – Virtual Dancer Virtual Dancer

  • Others

Others

– – Human Dancer Human Dancer – – A black background A black background – – t t-

  • shirts with different colors

shirts with different colors

/ 39 / 40

slide-16
SLIDE 16

EEMCS - HMI 16

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-17
SLIDE 17

EEMCS - HMI 17

Setup Setup

4 5

LAN

3 6 1 2

Port 7521 1 Computer Vision PC 2 Artificial Intelligent PC 3 Camera 4 User 5 Background 6 LAN / 39 / 40

slide-18
SLIDE 18

EEMCS - HMI 18

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-19
SLIDE 19

EEMCS - HMI 19

Interaction Dimensions Interaction Dimensions

GOAL OF THE I NTERACTI ON (What we want to do)

ACTI ONS

(What to do to get these goals)

CAUSE OF THE ACTI ON

(Why are we doing this action?)

W AY TO DO THI S ACTI ON

(How we want to do it)

TEMPORAL

(When occurs the fact)

/ 39 / 40

slide-20
SLIDE 20

EEMCS - HMI 20

Interaction Dimensions Interaction Dimensions

  • Temporal dimension

Temporal dimension

– – Before the song Before the song – – During the song During the song – – After the song After the song – – Between songs Between songs

GOAL OF THE I NTERACTI ON

(What we want to do)

ACTI ONS

(What to do to get these goals)

CAUSE OF THE ACTI ON

(Why are we doing this action?)

W AY TO DO THI S ACTI ON

(How we want to do it)

TEMPORAL

(When occurs the fact)

/ 39 / 40

slide-21
SLIDE 21

EEMCS - HMI 21

Interaction Dimensions Interaction Dimensions

  • Goal of the interaction

Goal of the interaction

– – To attract the attention To attract the attention – – To entertain To entertain – – To leave a good memory To leave a good memory

GOAL OF THE I NTERACTI ON

(What we want to do)

ACTI ONS

(What to do to get these goals)

CAUSE OF THE ACTI ON

(Why are we doing this action?)

W AY TO DO THI S ACTI ON

(How we want to do it)

TEMPORAL

(When occurs the fact)

/ 39 / 40

slide-22
SLIDE 22

EEMCS - HMI 22

Interaction Dimensions Interaction Dimensions

  • Actions

Actions

– – To attract the attention To attract the attention

  • To say hello

To say hello

  • To invite the user

To invite the user

  • To be seen

To be seen

– – To entertain To entertain

  • To start the dance

To start the dance

  • To encourage the user

To encourage the user

  • To do a choreography / chorus

To do a choreography / chorus

  • To enrich the dance

To enrich the dance

  • To explain some step

To explain some step

– – To leave a good memory To leave a good memory

  • To congratulate

To congratulate

  • To say goodbye

To say goodbye

GOAL OF THE I NTERACTI ON

(What we want to do)

ACTI ONS

(What to do to get these goals)

CAUSE OF THE ACTI ON

(Why are we doing this action?)

W AY TO DO THI S ACTI ON

(How we want to do it)

TEMPORAL

(When occurs the fact)

/ 39 / 40

slide-23
SLIDE 23

EEMCS - HMI 23

Interaction Dimensions Interaction Dimensions

  • Cause of the action

Cause of the action

– – Reactive Reactive – – Deliberative Deliberative

GOAL OF THE I NTERACTI ON

(What we want to do)

ACTI ONS

(What to do to get these goals)

CAUSE OF THE ACTI ON

(Why are we doing this action?)

W AY TO DO THI S ACTI ON

(How we want to do it)

TEMPORAL

(When occurs the fact)

/ 39 / 40

slide-24
SLIDE 24

EEMCS - HMI 24

Interaction Dimensions Interaction Dimensions

  • Way to do this action

Way to do this action

– – Verbal Verbal – – Non Verbal Non Verbal

GOAL OF THE I NTERACTI ON

(What we want to do)

ACTI ONS

(What to do to get these goals)

CAUSE OF THE ACTI ON

(Why are we doing this action?)

W AY TO DO THI S ACTI ON

(How we want to do it)

TEMPORAL

(When occurs the fact)

/ 39 / 40

slide-25
SLIDE 25

EEMCS - HMI 25

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-26
SLIDE 26

EEMCS - HMI 26

Interaction Tree Interaction Tree

Interaction Before the song During the song After the song To attract the attention To say hello Reactive / Deliberative To invite to dance Reactive / Deliberative To be seen To entertain

To do a Choreography / Chorus

To enrich the dance To attract the attention To be seen Reactive / Deliberative Reactive / Deliberative Reactive / Deliberative To entertain Reactive / Deliberative To leave a good memory To congratulate Reactive / Deliberative To say good bye Reactive / Deliberative To start the dance Reactive / Deliberative Between songs To entertain To encourage the user To explain some step Reactive / Deliberative Reactive / Deliberative To encourage the user Reactive / Deliberative To encourage the user Reactive / Deliberative Verbal / Non Verbal Verbal / Non Verbal Verbal / Non Verbal Verbal / Non Verbal Verbal / Non Verbal Verbal / Non Verbal Verbal / Non Verbal Verbal / Non Verbal Verbal / Non Verbal Verbal / Non Verbal Verbal / Non Verbal Verbal / Non Verbal Verbal / Non Verbal

/ 39 / 40

slide-27
SLIDE 27

EEMCS - HMI 27

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-28
SLIDE 28

EEMCS - HMI 28

Waiting a User (1) To be seen (2) To say hello (3) To invite to dance (4) To start the dance (6) To encourage the user (5) Dance time (7) To congratulate (12) To say good bye (15) To encourage the user (13) To explain some step (14)

CV detection Pad & CV detection Yes No User says Yes User says No User says No User says Yes To start the song Chorus detected Random Absent-minded user To finish the song User go out Random

Interaction State Machine Interaction State Machine

To enrich the dance (9) To encourage the user (10) To be seen (11) To do a choreography (8)

To detect passive movements

/ 39 / 40

slide-29
SLIDE 29

EEMCS - HMI 29

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-30
SLIDE 30

EEMCS - HMI 30

Variables, Animations & Transitions Variables, Animations & Transitions

  • What animations we could use in each state

What animations we could use in each state

  • New animations

New animations

  • Conditions required for changing the state

Conditions required for changing the state

  • Available Variables

Available Variables

  • New variables

New variables

/ 39 / 40

slide-31
SLIDE 31

EEMCS - HMI 31

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-32
SLIDE 32

EEMCS - HMI 32

Possible Improvements

  • Animations

Animations – – To say hello moving the hand, looking at the screen center To say hello moving the hand, looking at the screen center – – To bow to the user To bow to the user – – To clap to the user To clap to the user – – To say hello putting the head up To say hello putting the head up

  • Outputs

Outputs – – That our dancer could speak That our dancer could speak

  • Inputs

Inputs – – To recognize users To recognize users’ ’ speech speech – – To recognize head To recognize head’ ’s movements s movements

  • Pattern Recognition

Pattern Recognition – – Simple movements Simple movements’ ’ recognition recognition – – Patterns recognition Patterns recognition – – Creation of patterns automatically Creation of patterns automatically – – Search of Patterns Search of Patterns

/ 39 / 40

slide-33
SLIDE 33

EEMCS - HMI 33

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-34
SLIDE 34

EEMCS - HMI 34

New Added Features New Added Features

/ 39 / 40

slide-35
SLIDE 35

EEMCS - HMI 35

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-36
SLIDE 36

EEMCS - HMI 36

New Created Pipelines

  • Save Video Pipeline
  • From Video Pipeline

/ 39 / 40

slide-37
SLIDE 37

EEMCS - HMI 37

New Created Pipelines

  • Pipeline with Radial Histogram

/ 39 / 40

slide-38
SLIDE 38

EEMCS - HMI 38

Contents Contents

  • Introduction

Introduction

– – Presentation Presentation – – Goals Goals – – Organization of the Exposition Organization of the Exposition

  • The Virtual Dancer

The Virtual Dancer

  • Parlevision

Parlevision

  • Work Environment

Work Environment

– – Material Material – – Setup Setup

  • Previous Study

Previous Study

– – Interaction Dimensions Interaction Dimensions – – Interaction Tree Interaction Tree – – Interaction State Machine Interaction State Machine – – Variables, Animations & Transitions Variables, Animations & Transitions – – Possible Improvements Possible Improvements

  • The Built System (CV)

The Built System (CV)

– – New Added Features New Added Features – – New Created Pipelines New Created Pipelines – – New Sent Variables New Sent Variables

/ 39 / 40

slide-39
SLIDE 39

EEMCS - HMI 39

New Sent Variables New Sent Variables

  • cumulativeHorActivityNormalized

cumulativeHorActivityNormalized

  • cumulativeVertActivityNormalized

cumulativeVertActivityNormalized

  • lHand

lHand

  • rHand

rHand

  • cumulativeRActivityNormalized

cumulativeRActivityNormalized

  • r

r

  • x

x

  • y

y

/ 39 / 40

slide-40
SLIDE 40

EEMCS - HMI 40

Dance Movement Patterns Recognition Dance Movement Patterns Recognition (Part II) (Part II) … …

Bedankt voor uw aandacht ! Bedankt voor uw aandacht !

/ 39 / 40