Learning Perceptual Causality from Video Amy Fire and Song-Chun Zhu - - PowerPoint PPT Presentation

Learning Perceptual Causality from Video

Amy Fire and Song-Chun Zhu

Center for Vision, Cognition, Learning, and Art

UCLA

1

Ideally: Learn Causality from Raw Video

2

Inference Using Learned Causal Structure

• Answer why events occurred
• Joint STC: Infer misdetections and hidden objects/actions
• Infer triggers, goals, and intents

3

a) Input: Video b) Event Parsing c) STC-Parsing d) Inference Over Time Agent Actions Fluents

Action Hidden Fluent Causal Link

UNKNOWN

Time

ON OFF THIRSTY NOT

Light Agent

Time Time

Drink Flip Switch

But…

OBSERVATION CAUSALITY

(generally)

4

SO…WHERE ARE WE NOW?

5

Vision Research and Causal Knowledge

• Use pre-specified causal relationships for action detection

– E.g., PADS (Albanese, et al. 2010) – Model Newtonian mechanics (Mann, Jepson, and Siskind 1997)

• Use causal measures to aid action detection

– E.g., Prabhakar, et al. 2010

• Use infant perceptions of motion to learn causality

– Using cognitive science (Brand 1997)

• Needed: Learn causality from video, integrating ST learning

strategies at pixel level

6

Causality and Video Data: Often Disjoint

• Learning Bayesian networks

– Constraint satisfaction (Pearl 2009) – Bayesian formulations (Heckerman 1995) – Intractable on vision sensors

• Commonsense reasoning (Mueller 2006) – first order logic.

– Do not allow for ambiguity/probabilistic solutions

• MLNs (Richardson and Domingos 2006)

– Intractable – Used for action detection (Tran and Davis 2008)

• KB formulas not learned

7

MOVING FORWARD: OUR PROPOSED SOLUTION

8

Cognitive Science as a Gateway: Perceptual Causality

• Causal Induction from Observation in Infancy

– Agentive actions are causes (Saxe, Tenenbaum, and Carey 2005) – Co-occurrence of events and effects (Griffiths and Tenenbaum 2005) – Temporal lag between the two is short (Carey 2009) – Cause precedes effect (Carey 2009)

• Note: NOT the same as necessary and sufficient causes

9

e-specified e-specified Measure co-occurrence cr : Action ¬Action Effect c0 c1 ¬Effect c2 c3 − ✏ − fluent classifiers. − λ λ ✓ · ◆ ’s amyfir .fir

MODIFIED GOAL: LEARN AND INFER PERCEPTUAL CAUSALITY

10

What are the effects? Fluent changes.

11

t

Door Opens Door Closed Inertially Door Closes Light On Inertially

Light

ON OFF

Door

OPEN CLOSED

Light Off Inertially Light Turns Off Door Open Inertially Light Turns On

• Fluents are time-varying statuses of objects

– Mueller – Commonsense Reasoning 2006

What are the causes? Actions.

• Probabilistic Graphical Representation for Causality

– And-Or Graph

12

And-Nodes Compose Single Causes (multiple sub-actions)

A

Unlock Door Push Door

a a

Or-Nodes Give Alternative Causes

Open Other Side

A a

Causal AOG

13

Light fluent

• n
• ff

a01 a01

OR OR AND AND

Connecting Temporal to Causal and Spatial

14

fluent1

• bject1

A

parent(A) children(A) terminates

• bject2

R(A) fluent2 S-AOG fragment T-AOG fragment C-AOG fragment

f1 f2 O2 O1

Grounding on Pixels: Connecting S/T/C-AOG

15

parent(O) children(O) fluents

f1

actions terminates grounding templates(O)

O

S-AOG fragment

f2 f3

A

A1 A2 A3 T-AOG fragment C-AOG fragment

T1(o)

Grounded

• n Pixels

LEARNING PERCEPTUAL CAUSALITY

Preliminary Theory

16

Principled Approach: Information projection

DellaPietra, DellaPietra,Lafferty, 97 Zhu, Wu, Mumford, 97

17

True contingency table distribution Initial distribution Repeat

       

 

I E I E p

f p 2 2 2

: cr cr   

Select cause/effect relationship that makes p1 closest (KL) to p to preserve learning history while maximizing Info. Gain

KL f || p

( ) = KL f || p+ ( )+ KL p+ || p ( )

max KL f || p

( )- KL f || p+ ( )

é ë ù û= maxKL p+ || p

( )        

 

I E I E p

f p 1 1 1

: cr cr   

Match the statistics of the contingency table

Learning Pursuit: Add Causal Relations

• Model Pursuit
• Proposition 1: Find parameters

– Model formed by min KL (p+ || p), matching statistics

• Proposition 2: Pursue cr.

18

f p p p p p

k 

     



 

1

Ep+ cr

+

( ) = Ef cr

+

( )

   

 

   

  cr , exp 1  pg p z pg p

(On ST-AOG)

cr cr Effect

• Effect

Action co c2

• Action

c1 c3

l+,i = log hi h0 ×f0 fi æ è ç ö ø ÷

cr

+ = argmax cr

KL p+ || p

( ) = argmax

cr

KL f || h

( )

hi is ci under p fi is ci from f

Selection from ST-AOG

2 1 2 1

A A A A A

f f f f f   

1

A

f

) ( F T 

A

A1 A2

2

A

f

1

A

f

) ( F T 

A

A1 A2

p

p  1

2 1

) 1 (

A A A

f p pf f   

Or-node And-node

Performance vs. TE

20

Performance vs. Hellinger χ2

21

Increasing Misdetections

(Simulation)

22

0% misdetection 10% misdetection 20% misdetection

STC-Parsing Demo

23

Looking Forward:

• Finish learning the C-AOG
• Increase reasoning capacity of the C-AOG
• Integrate experiment design

24