Object Tracking Computer Vision Fall 2018 Columbia University - - PowerPoint PPT Presentation

▶
object tracking
SMART_READER_LITE
LIVE PREVIEW

Object Tracking Computer Vision Fall 2018 Columbia University - - PowerPoint PPT Presentation

Sep 24, 2022 103 likes •664 views

Object Tracking Computer Vision Fall 2018 Columbia University Homework 5 Released last night Due November 26th Start it today no extensions! Optical Flow Optical flow field: assign a flow vector to each pixel Visualize:

slide-1
SLIDE 1

Object Tracking

Computer Vision Fall 2018 Columbia University

slide-2
SLIDE 2

Homework 5

  • Released last night
  • Due November 26th
  • Start it today— no extensions!
slide-3
SLIDE 3

Optical Flow

  • Optical flow field: assign a flow vector to each pixel
  • Visualize: flow magnitude as saturation,

  • rientation as hue

Ground-truth flow field Visualization code [Baker et al. 2007] Input two frames

slide-4
SLIDE 4
  • Brightness/color is constant
  • Small motions
  • Also assume neighboring pixels have same motion

x y t

I u I v I + + =

Optical Flow Constraint

slide-5
SLIDE 5

Solving the aperture problem

  • How to get more equations for a pixel?
  • Spatial coherence constraint: pretend the pixel’s

neighbors have the same (u,v)

  • If we use a 5x5 window, that gives us 25 equations

per pixel

Slide credit: Steve Seitz

slide-6
SLIDE 6

Solving the aperture problem

Problem: we have more equations than unknowns

  • The summations are over all pixels in the K x K window
  • This technique was first proposed by Lucas & Kanade (1981)

Solution: solve least squares problem

  • minimum least squares solution given by solution (in d) of:

Slide credit: Steve Seitz

slide-7
SLIDE 7

Solving the aperture problem

Problem: we have more equations than unknowns

  • The summations are over all pixels in the K x K window
  • This technique was first proposed by Lucas & Kanade (1981)

Solution: solve least squares problem

  • minimum least squares solution given by solution (in d) of:

Slide credit: Steve Seitz

slide-8
SLIDE 8

Aperture Problem

Which way did the line move?

slide-9
SLIDE 9

Aperture Problem

Which way did the line move?

slide-10
SLIDE 10

Motion Fields

Zoom out Zoom in Pan right to left

slide-11
SLIDE 11

Can we do more? Scene flow

Combine spatial stereo & temporal constraints Recover 3D vectors of world motion

Stereo view 1 Stereo view 2

t t-1

3D world motion vector per pixel z x y

slide-12
SLIDE 12

Scene flow example for human motion

Estimating 3D Scene Flow from Multiple 2D Optical Flows, Ruttle et al., 2009

slide-13
SLIDE 13

Scene Flow

[Estimation of Dense Depth Maps and 3D Scene Flow from Stereo Sequences, M. Jaimez et al., TU Munchen]

https://www.youtube.com/watch?v=RL_TK_Be6_4 https://vision.in.tum.de/research/sceneflow

slide-14
SLIDE 14

Motion Analysis

slide-15
SLIDE 15

Motion Magnification

slide-16
SLIDE 16

Motion Magnification

slide-17
SLIDE 17

Motion Magnification

slide-18
SLIDE 18

Motion Magnification

slide-19
SLIDE 19

Learning optic flow

Synthetic Training data

Fischer et al. 2015. https://arxiv.org/abs/1504.06852

slide-20
SLIDE 20

Time

slide-21
SLIDE 21

Time

What color is that pixel?

slide-22
SLIDE 22

Temporal Coherence of Color

RGB Color Channels Quantized Color

slide-23
SLIDE 23

Obvious exceptions…

slide-24
SLIDE 24

Obvious exceptions…

Edward Adelson, 1995

slide-25
SLIDE 25

Obvious exceptions…

slide-26
SLIDE 26

Color is mostly temporally coherent

slide-27
SLIDE 27

Self-supervised Tracking

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

Reference Frame Gray-scale Video

slide-28
SLIDE 28

What color is this?

slide-29
SLIDE 29

Where to copy color?

slide-30
SLIDE 30

Want to be safe!

slide-31
SLIDE 31

Where to copy color?

slide-32
SLIDE 32

Color can be robust to

  • cclusion
slide-33
SLIDE 33

Input Frame

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-34
SLIDE 34

Target Colors Input Frame Reference Frame Reference Colors

Colorize by Pointing

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-35
SLIDE 35

Target Colors Input Frame Reference Frame Reference Colors

fj fi Aij

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-36
SLIDE 36

Target Colors Input Frame Reference Frame Reference Colors

min

f

L cj, X

i

Aijci ! where Aij = exp

  • f T

i fj

  • P

k exp

  • f T

k fj

  • fj

fi Aij

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-37
SLIDE 37

Target Colors Input Frame Reference Frame Reference Colors

min

f

L cj, X

i

Aijci ! where Aij = exp

  • f T

i fj

  • P

k exp

  • f T

k fj

  • fj

fi Aij ci

ˆ cj =

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-38
SLIDE 38

Target Colors Input Frame Reference Frame Reference Colors

min

f

L cj, X

i

Aijci ! where Aij = exp

  • f T

i fj

  • P

k exp

  • f T

k fj

  • fj

fi Aij ci cj

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-39
SLIDE 39
slide-40
SLIDE 40
slide-41
SLIDE 41

Lumière Brothers

Inventors of motion picture, 1895 Inventors of first practical color camera, 1903

slide-42
SLIDE 42

Georges Méliès

“Discovered” special effects, 1898

slide-43
SLIDE 43

Video Colorization

Ground T Reference Frame Gray-scale Video Predicted Color

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-44
SLIDE 44

Video Colorization

Reference Frame Gray-scale Video Predicted Color Ground T

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-45
SLIDE 45

Input Frame Reference Frame

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

Tracking Emerges!

slide-46
SLIDE 46

Tracking Emerges!

Input Frame Reference Frame Predicted Mask Reference Mask

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-47
SLIDE 47

Input Frame Reference Frame Predicted Mask Reference Mask

min

f

L cj, X

i

Aijci ! where Aij = exp

  • f T

i fj

  • P

k exp

  • f T

k fj

  • ˆ

cj =

fj fi Aij ci

ˆ cj

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-48
SLIDE 48

Only the first frame is given. Colors indicate different instances.

Segment Tracking Results

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-49
SLIDE 49

Only the first frame is given. Colors indicate different instances.

Segment Tracking Results

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-50
SLIDE 50

Pose Tracking Results

Only the skeleton in the first frame is given.

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-51
SLIDE 51

Tracking Performance

Average Performance (Segment Overlap) 20 40 60 80 Frame Number 2 9 16 23 30 37 44 51 58 64

Identity Optic Flow Colorization

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-52
SLIDE 52

Tracking Performance

Scale-Variation Shape Complexity Appearance Change Heterogeneus Object Out-of-view Interacting Objects Motion Blur Occlusion Fast Motion Dynamic Background Low Resolution Deformation Edge Ambiguity Out-of-Plane Rotation Background Clutter Camera-Shake Average Performance (Segment Overlap) 12.5 25 37.5 50

Identity Optic Flow Colorization Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-53
SLIDE 53

Visualizing Embeddings

Project embedding to 3 dimensions and visualize as RGB O r i g i n a l V i d e

  • E

m b e d d i n g V i s u a l i z a t i

  • n

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018

slide-54
SLIDE 54

Colorization and tracking fail together

Reference Colors Reference Mask Predicted Mask Predicted Colors

Vondrick, Shrivastava, Fathi, Guadarrama, Murphy. ECCV 2018