Learning Human Interaction by L i H I i b Interactive Phrases - - PowerPoint PPT Presentation

learning human interaction by l i h i i b interactive
SMART_READER_LITE
LIVE PREVIEW

Learning Human Interaction by L i H I i b Interactive Phrases - - PowerPoint PPT Presentation

Learning Human Interaction by L i H I i b Interactive Phrases Interactive Phrases Yu Kong 1,3 , Yunde Jia 1 and Yun Fu 2 1 3 i 1 2 d d 1 Beijing Institute of Technology 2 Northeastern University 3 University at Buffalo Activity


slide-1
SLIDE 1

L i H I i b Learning Human Interaction by Interactive Phrases Interactive Phrases

1 3

d i 1 d

2

Yu Kong1,3, Yunde Jia1 and Yun Fu2

1Beijing Institute of Technology 2Northeastern University 3University at Buffalo

slide-2
SLIDE 2

Activity landscape

Individual action Interaction Group action Crowd action

One person Few people Several people Crowd of people Number of people Id ifi i f h Identification of each person Easy Easy Not accurate but we can Very challenging,

  • pen problem
  • pen problem

Our work

slide-3
SLIDE 3

Objective: recognizing human interactions from videos videos.

Interaction: Boxing

Applications

Motion analysis Detect unusual behavior Group activity understanding Judge sports automatically Video game interfaces Smart surveillance Scene analysis Smart surveillance

slide-4
SLIDE 4

Motivation

An interaction is determined b i di id l

Motivation

by individual actions.

Recognize interaction by action co-occurrence Recognize interaction by action co-occurrence Action co occurrence Action co-occurrence Attack-Protect head Attack-Dodge Attack-Hit back Interaction: Boxing P bl l ti hi t i Problem: co-occurrence relationships are not expressive enough to deal with interactions with large variations.

slide-5
SLIDE 5

Motivation Motivation

We introduce interactive phrases to describe human interactions. b/ ill

describe

  • Int. b/w still arms

NO

  • Int. b/w a chest-level moving arm

and a tilting upward arm YES

criptions

  • Int. b/w a still torso and a bending torso YES
  • Int. b/w leaning forward torsos

NO

Des recognize

Human interaction: Boxing Interactive phrases:

  • More expressive to describe complicated human interactions.
  • Binary motion relationships between people. E.g., relationships between arms,

legs, and torsos, etc.

  • Mid-level feature learned from data
slide-6
SLIDE 6

Flowchart of our method

Video Low-level feature Motion attribute Interactive phrases Interaction Feature extraction Build individual action representation Attribute model Detect individual motion attribute Interaction model Learn interactive phrases and recognize interaction

slide-7
SLIDE 7

Individual action representation Individual action representation

[d1, d2, …,dn]

0 15 0.2 0.25

[ , , , ]

0.05 0.1 0.15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Learned dictionary Low-level local feature

slide-8
SLIDE 8

Attribute model

Objective: Jointly detect individual motion attributes.

Motion attributes: describe individual motion, e.g. arm stretching out, leg stepping forward, etc.

9

still leg

10

leg stepping forward motion

11

leg kicking motion id attributes am

1

still arm

2

hand stretching out motion

12

leg stepping back motion

13

still torso

14

torso leaning back motion

3

arm chest‐level motion

4

two arms chest‐level motion

5

arm raising up motion

15

torso leaning forward motion

16

torso bending motion

17

friendly motion

6

arm embracing motion

7

arm free swinging motion

8

arm intense motion

slide-9
SLIDE 9

Individual attribute detection +1 present

0.15 0.2 0.25

+1, present

0.05 0.1

j-th attribute aj=1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
  • 1, absent

Repeat j=1…M Attribute detector , abse t

slide-10
SLIDE 10

J i tl d t t i di id l ti tt ib t Jointly detect individual motion attributes. Infer the optimal configuration of attributes (a1…aM) a3 Unary attribute potential Pairwise attribute potential S ib l b l S i i ib a1 a a4 Score attribute label from feature Score pairwise attribute relationship Attribute graph a5 a6 a2

slide-11
SLIDE 11

Attribute model

Motion attribute

id attributes am

1

till

1

still arm

2

hand stretching out motion

3

arm chest‐level motion

4

two arms chest‐level motion

1

5

arm raising up motion

6

arm embracing motion

7

arm free swinging motion

8

arm intense motion

9

still leg

10

leg stepping forward motion

11

leg kicking motion

1

11

leg kicking motion

12

leg stepping back motion

13

still torso

14

torso leaning back motion

15

torso leaning forward motion

16

torso bending motion

17

friendly motion

1

slide-12
SLIDE 12

Interaction model Interaction model

Objective: learn interactive phrases and infer interaction class Interactive phrases: motion relationships between people, e.g. relationships between arms, legs, torsos, etc.

slide-13
SLIDE 13

Interactive phrases Interactive phrases

id f i t d

Attributes

id interactive phrases pj id of associated attributes aj(1) ,aj(2) 1 b/w still arms 1,1 2 b/w a chest‐level moving arm and a free swinging arm 3,7 3 b/w outstretched hands 2,2

Person 1 Person 2 Still arm Still arm

/ , 4 b/w raising up arms 5,5 5 b/w embracing arms 6,6 6 b/w a chest‐level moving arm and a still arm 3,1 7 b/w two chest‐level moving arms and a free swinging arm 4,7 8 b/w free swinging arms 7,7 9 b/w intense moving arms 8,8 10 b/w a chest‐level moving arm and a leaning backward torso 3,14 11 b/w two chest‐level moving arms and a leaning backward torso 4,14 12 b/w still legs 9 9 12 b/w still legs 9,9 13 b/w a stepping forward leg and a stepping backward leg 10,12 14 b/w stepping forward legs 10,10 15 b/w a stepping forward leg and a still leg 10,9 16 b/w a kicking leg and a stepping backward leg 11,12

Stepping forward l / till l Still leg/stepping f d l

17 b/w a bending torso and a still torso 16,13 18 b/w a leaning forward torso and a leaning backward torso 15,14 19 b/w leaning forward torsos 15,15 20 b/w leaning backward torsos 14,14 21 b/ l i f d d ill 15 13

leg/still leg forward leg

21 b/w a leaning forward torso and a still torso 15,13 22 b/w still torsos 13,13 23 cooperative interaction 17,17

slide-14
SLIDE 14

Interactive phrases Interactive phrases

Latent variable, learned from data mid-level feature, used for inferring interaction class

slide-15
SLIDE 15
slide-16
SLIDE 16

Experiments p

  • BIT-Interaction dataset

– 8 classes 400 videos 8 classes, 400 videos

  • UT-Interaction dataset

6 l 60 id

bow boxing handshake high‐five hug kick pat push

– 6 classes, 60 videos

handshake hug kick point punch push

slide-17
SLIDE 17

Results on BIT-Interaction dataset

  • 8 interaction classes, 400 videos, 23 interactive phrases, 17 motion attributes

Confusion matrix of our method Classification examples of our method Accuracy = 85.16%

slide-18
SLIDE 18

Results on BIT Interaction dataset Results on BIT-Interaction dataset

slide-19
SLIDE 19

Results on BIT Interaction dataset Results on BIT-Interaction dataset

100

Comparison results of accuracy (%)

Recognition accuracy (%) of methods

50 60 70 80 90 bag‐of‐words no‐phrase method 10 20 30 40 p no‐IPC method no‐AC method Our method

No‐phrase method: remove phrase layer from the full model No‐IPC method: remove phrase connection component from the full model No‐AC method: remove attribute connection component from the full model p

slide-20
SLIDE 20

Results on UT-Interaction dataset Results on UT Interaction dataset

  • 6 interaction classes, 60 videos, 23 interactive phrases, 16 motion attributes

Confusion matrix of our method Accuracy = 88.33% Classification examples of our method

slide-21
SLIDE 21

Results on UT Interaction dataset Results on UT-Interaction dataset

Recognition accuracy (%) of methods

100

Recognition accuracy (%) of methods

50 60 70 80 90 bag‐of‐words no‐phrase method no‐AC method 10 20 30 40 50 no‐IPC method Ryoo & Aggarwal (ICCV 2009) Yu et al. (BMVC 2010) Ryoo (ICCV 2011) y ( ) Our method

[1] [2] [1] [3]

slide-22
SLIDE 22

Thank you! Please email yukong@ece.neu.edu if you have any questions if you have any questions.