Theory Mind from Videos Tao Gao MIT Theory of Mind Can't see - - PowerPoint PPT Presentation

▶
theory mind from videos
SMART_READER_LITE
LIVE PREVIEW

Theory Mind from Videos Tao Gao MIT Theory of Mind Can't see - - PowerPoint PPT Presentation

Dec 09, 2023 449 likes •1.05k views

Theory Mind from Videos Tao Gao MIT Theory of Mind Can't see Can't do Don't care R B A M Theory of Mind 1. Dimensionality 4. Observability 2. Optimality 5. Recursion 3. Physics 6. Hierarchy Theory of Mind 1. Dimensionality 4.

slide-1
SLIDE 1

Tao Gao

MIT

Theory Mind from Videos

slide-2
SLIDE 2

Theory of Mind

M

B

Can't see

A

Can't do

R

Don't care

slide-3
SLIDE 3

Theory of Mind

  • 1. Dimensionality
  • 2. Optimality
  • 3. Physics
  • 4. Observability
  • 5. Recursion
  • 6. Hierarchy
slide-4
SLIDE 4

Theory of Mind

  • 1. Dimensionality
  • 2. Optimality
  • 3. Physics
  • 4. Observability
  • 5. Recursion
  • 6. Hierarchy
slide-5
SLIDE 5

Dimensionality

Point Dot Full Body

slide-6
SLIDE 6
slide-7
SLIDE 7

Don’t-Get-Caught

slide-8
SLIDE 8

Don’t-Get-Caught

Wolf Succeed after 20s! Kill zone

Gao & Scholl (2011, 2013)

slide-9
SLIDE 9

Perfect Heat Seeking

slide-10
SLIDE 10
slide-11
SLIDE 11

Chasing Subtlety

…maximum deviation from heat seeking

Heat Seeking

Sheep

Wolf

Chasing Subtlety =0° (Perfect Heat Seeking) Sheep 3 ° Small Subtlety (30°)

Wolf Heat Seeking

90° Mild Subtlety (90°) Sheep

Heat Seeking Wolf

slide-12
SLIDE 12

Subtlety 90º (Cheating)

slide-13
SLIDE 13
slide-14
SLIDE 14

90° Subtlety

slide-15
SLIDE 15
slide-16
SLIDE 16

Dynamic Bayesian Model

θ C Lt

s

Lt−1

s

θ ~[5,30,60,90,120,150,180] C: onehot encodingwith dim N

Lt

1:N

Lt−1

1:N

p(C ,θ |L0:t

1:N ,L0:t s )∝ p(C ,θ, L0:t 1:N | L0:t s )

p(C ,θ , L0:t

1: N| L0:t s )

p(C| L0:t

1:N L0:t s )∝∑θ p(C ,θ ,L0:t 1:N | L0:t s )

p(Li

1:N |C ,θ ,Li−1 1:N ,Li−1 s )

p(C ,θ , L0:t

1: N| L0:t s )= p(C) p(θ)∏ i=1 t

p(Li

1:N |C ,θ ,Li−1 1: N ,Li−1 s )

slide-17
SLIDE 17

Dimensionality

Point Dot Full Body

slide-18
SLIDE 18
slide-19
SLIDE 19

Baby: I can help

What ? Should ? How ? Why ?

Warneken & Tomasello (2006) Science

slide-20
SLIDE 20

Live Experiment

slide-21
SLIDE 21

20%

slide-22
SLIDE 22

20%

slide-23
SLIDE 23

Theory of Mind

  • 1. Dimensionality
  • 2. Optimality
  • 3. Physics
  • 4. Observability
  • 5. Recursion
  • 6. Hierarchy
slide-24
SLIDE 24

Optimality

Heuristics Rational Planning

slide-25
SLIDE 25

Chasing Subtlety

Heat Seeking

Sheep

Wolf

Chasing Subtlety =0° (Perfect Heat Seeking)

slide-26
SLIDE 26
slide-27
SLIDE 27
slide-28
SLIDE 28

Rational planning

slide-29
SLIDE 29
slide-30
SLIDE 30
slide-31
SLIDE 31

Rational Planning

slide-32
SLIDE 32

Rational Planning

slide-33
SLIDE 33

Theory of Mind

  • 1. Dimensionality
  • 2. Optimality
  • 3. Physics
  • 4. Observability
  • 5. Recursion
  • 6. Hierarchy
slide-34
SLIDE 34

Physics

Dynamics Kinematics

slide-35
SLIDE 35

Kinematic

Wolf

slide-36
SLIDE 36

Dynamic

2D physics simulation Force control

slide-37
SLIDE 37
slide-38
SLIDE 38

Dynamics

Mordatch I, Todorov E & Popovic, Z (2012)

slide-39
SLIDE 39

Theory of Mind

  • 1. Dimensionality
  • 2. Optimality
  • 3. Physics
  • 4. Observability
  • 5. Recursion
  • 6. Hierarchy
slide-40
SLIDE 40

Observability

Partial Observable Fully Observable

slide-41
SLIDE 41

MIT Food Truck Study

Baker, Jara-Ettinger, Saxe & Tenenbaum (2017)

K:Korean L:Lebanese M: Mexican

slide-42
SLIDE 42

Beyond what you can/cannot see ...

slide-43
SLIDE 43

Thoughts are unobservable

Two things awe me most,

the starry sky above me and the moral law within me.

  • Kant
slide-44
SLIDE 44
slide-45
SLIDE 45

Theory of Mind

  • 1. Dimensionality
  • 2. Optimality
  • 3. Physics
  • 4. Observability
  • 5. Recursion
  • 6. Hierarchy
slide-46
SLIDE 46

Recursion

slide-47
SLIDE 47
slide-48
SLIDE 48

Recursion

Straight Recursion

slide-49
SLIDE 49
slide-50
SLIDE 50

Recursion

… the goal is to hide the goal ...

slide-51
SLIDE 51

Nash Equilibrium of Chasing

slide-52
SLIDE 52
slide-53
SLIDE 53

Theory of Mind

  • 1. Dimensionality
  • 2. Optimality
  • 3. Physics
  • 4. Observability
  • 5. Recursion
  • 6. Hierarchy
slide-54
SLIDE 54

Hierarchy

Collection Structure

slide-55
SLIDE 55

Stand

Reach

Move Grasp Walk

HTN + RRT*

slide-56
SLIDE 56

Theory of Mind

  • 1. Dimensionality
  • 2. Optimality
  • 3. Physics
  • 4. Observability
  • 5. Recursion
  • 6. Hierarchy
slide-57
SLIDE 57

Collaborators

Paul Vernaza (CMU) Yibiao Zaho (MIT) Josh Tenenbaum (MIT) Nancy Kanwisher (MIT) Brian Scholl (Yale) Chris Baker (MIT)

slide-58
SLIDE 58

Acknowledgement