Its Moving! A Probabilistic Model for Causal Motion Segmentation in - - PowerPoint PPT Presentation
Its Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videos Erik Learned-Miller with Manju Narayana Pia Bideau U NIVERSITY OF M ASSACHUSETTS , A MHERST College of Computer and Information Sciences U
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Erik Learned-Miller with Manju Narayana Pia Bideau
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
2
Uploaded by “Houssem” https://www.youtube.com/watch?v=adufPBDNCKo
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
3
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Keypoint matching—”sparse”
Track points through multiple frames Cluster tracks in subspaces Typically non-causal e.g. Vidal (2010), Fragkiadaki et al. (2012), Keuper et al. (2015).
Optical Flow
Dense tracking information enforces local smoothness and appearance constraints slower Narayana and Learned-Miller (ICCV13), Papazoglou and Ferrari (ICCV13), Zamalieva et al. (ECCV 2014).
7
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
8
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
representation
segmented labels
9
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
10
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
What makes flow complicated?
Camera motion along optical axis Camera rotation Dependence on scene depth Noise in optical flow Multiple moving objects
11
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Definition: Flow Angle Field The angle portion of each vector in an optical flow field.
12
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
(Sintel Optical Flow Database)
13
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
For a translating camera and a rigid scene: The angle field depends ONLY
14
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
For a translating camera and a rigid scene: The angle field depends ONLY
15
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
for Scene 1 angle for Scene 1
16
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
for Scene 2 angle for Scene 2
17
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
for Scene 1 angle for Scene 1
18
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
for Scene 2 angle for Scene 2
19
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Translational “canonical” flow angle fields
Optical flow angle fields for different translations
20
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
21
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
22
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Estimate which translations are present Estimate which pixels belong to each translation
segmented labels
23
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
24
Translation portion of
Angle field of translation portion of
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
25
Translation Angle Field
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
26
Translation Angle Field Best fit to canonical Translation Angle Field
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
27
Translation Angle Field Best fit to canonical Translation Angle Field
closed form solution, by Bruss and Horn, 1983
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
28
Translation Angle Field Best fit to canonical Translation Angle Field Residual error
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
29
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Generative model of flow orientations, Narayana et al. (ICCV 2013)
30
flow angle pixel coordinates segment 3D-direction associated with each segment
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Canonical angle fields result from translational motion. But how do we
Deal with camera rotation? Deal with noisy optical flow?
31
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Example with rotational camera motion
32
Total
Angle field of total
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Example with rotational camera motion
33
Total
Angle field of total
Foreground tree does not fit canonical angle field
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
34
Does there exist a rotation field R, such that when R is subtracted from optical flow, the resulting flow angle field is close to a canonical translation angle field? flow angle
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
35
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
36
translation optimization
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
37
translation optimization Our own implementation: simple optimization in Matlab, calling Bruss and Horn as inner loop.
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
38
Angle field with rotation Angle field without rotation Estimated Rotation “Compensated flow” Original Opt. Flow
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
39
translation field ideal translation field error image
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
40
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Initialization: Segment first frame Main body: Segment current frame given posteriors from previous frame
1. Compute optical flow 2. Compute motion component priors 3. Compute camera rotation, “subtract off” 4. Compute motion component likelihoods 5. Compute motion component posteriors
41
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Initialization: Segment first frame Main body: Segment current frame given posteriors from previous frame
1. Compute optical flow 2. Compute motion component priors 3. Compute camera rotation, “subtract off” 4. Compute motion component likelihoods 5. Compute motion component posteriors
42
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Sun, Roth, and Black. “Secrets of Optical Flow Estimation and their Principles.” CVPR 2010.
43
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Initialization: Segment first frame Main body: Segment current frame given posteriors from previous frame
1. Compute optical flow 2. Compute motion component priors 3. Compute camera rotation, “subtract off” 4. Compute motion component likelihoods 5. Compute motion component posteriors
44
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Computing the Motion Component Priors
45
For each of k motion components Start with posterior from previous frame. “Flow forward” using optical flow from previous frame. Blur to reflect uncertainty in optical flow. Normalize at each pixel location to create proper prior.
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Initialization: Segment first frame Main body: Segment current frame given posteriors from previous frame
1. Compute optical flow 2. Compute motion component priors 3. Compute camera rotation, “subtract off” 4. Compute motion component likelihoods 5. Compute motion component posteriors
46
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Use prior for background component to select pixels for nested optimization:
47
Use background prior to do weighted sum over background pixels.
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Next, at each pixel, estimate:
Priors for each motion component Likelihoods
48
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Initialization: Segment first frame Main body: Segment current frame given posteriors from previous frame
1. Compute optical flow 2. Compute motion component priors 3. Compute camera rotation, “subtract off” 4. Compute motion component likelihoods 5. Compute motion component posteriors
49
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
50
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
51
What does 2-D Gaussian noise do to the measured flow angle?
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Modeling noisy flow angles with von Mises
52
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
53
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
54
One consequence: Very small optical flow vectors are uninformative
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Initialization: Segment first frame Main body: Segment current frame given posteriors from previous frame
1. Compute optical flow 2. Compute motion component priors 3. Compute camera rotation, “subtract off” 4. Compute motion component likelihoods 5. Compute motion component posteriors
55
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Bayes rule (at each pixel position):
56
conditional angle likelihood motion component prior motion component posterior
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Initialization: Segment first frame Main body: Segment current frame given posteriors from previous frame
1. Compute optical flow 2. Compute motion component priors 3. Compute camera rotation, “subtract off” 4. Compute motion component likelihoods 5. Compute motion component posteriors
57
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
How to compute camera rotation and translation with no background prior? How to compute posteriors with no motion component priors?
58
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
59
Use RANdom SAmple Consensus (RANSAC)
10 superpixels used to estimate rotation 5000 trials For 3 corners to be part of region Choose rotation with fewest outliers
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
60
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
61
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
62
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
63
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
64
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
65
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
66
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
67
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
68
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
Rotational flows: isomorphic to SO3
69
UNIVERSITY OF MASSACHUSETTS, AMHERST • College of Computer and Information Sciences
70