Coherent(Spatiotemporal(Filtering, - - PowerPoint PPT Presentation

▶
coherent spatiotemporal filtering upsampling and
SMART_READER_LITE
LIVE PREVIEW

Coherent(Spatiotemporal(Filtering, - - PowerPoint PPT Presentation

Sep 25, 2022 156 likes •1.05k views

Coherent(Spatiotemporal(Filtering, Upsampling(and(Rendering(of(RGBZ(Videos Christian)Richardt 1,2 )))))))Carsten)Stoll 1 )))))))Neil)A.)Dodgson 2 Hans:Peter)Seidel 1 )))))))Christian)Theobalt 1 1 2

slide-1
SLIDE 1

Christian)Richardt1,2)))))))Carsten)Stoll1)))))))Neil)A.)Dodgson2 Hans:Peter)Seidel1)))))))Christian)Theobalt1

Coherent(Spatiotemporal(Filtering, Upsampling(and(Rendering(of(RGBZ(Videos

1 2

slide-2
SLIDE 2

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos O

slide-3
SLIDE 3

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Introduction(&(motivation

  • ur&work&tackles&the&noisy,&low4resolution&depth&data:

filter&colour&+&depth&to&upsample&and&denoise&depth& capturing&colour&+&depth&enables&a&variety&of compelling,&previously&impossible&video&effects prototype&video&camera&+&video&processing&algorithms =&effective&and&robust&capture&of&RGBZ&video result:&dynamic,&temporally&coherent&scene&geometry,& calculated&at&interactive&frame&rates

P

slide-4
SLIDE 4

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(geometry(capture

four&main&approaches&to& capture&dynamic&geometry:

  • 1. photometric&stereo&/

shape4from4shading

Q

[Malzbender&et&al.&2006]

slide-5
SLIDE 5

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(geometry(capture

four&main&approaches&to& capture&dynamic&geometry:

  • 1. photometric&stereo&/

shape4from4shading

  • 2. active&stereo&/

structured&light

Q

[Lanman&and&Taubin&2009]

slide-6
SLIDE 6

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(geometry(capture

four&main&approaches&to& capture&dynamic&geometry:

  • 1. photometric&stereo&/

shape4from4shading

  • 2. active&stereo&/

structured&light

  • 3. structure4from4motion

&&stereo&vision

Q

[Scharstein&and&Szeliski&2003]

slide-7
SLIDE 7

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(geometry(capture

four&main&approaches&to& capture&dynamic&geometry:

  • 1. photometric&stereo&/

shape4from4shading

  • 2. active&stereo&/

structured&light

  • 3. structure4from4motion

&&stereo&vision

  • 4. time4of4flight&cameras

Q

[Iddan&and&Yahav&2001]

slide-8
SLIDE 8

Markov&random&fields

[Diebel&and&Thrun&2006]

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(depth(upsampling

R

slide-9
SLIDE 9

Markov&random&fields

[Diebel&and&Thrun&2006]

spatial4depth&super4resolution

[Yang&et&al.&2007]

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(depth(upsampling

R

slide-10
SLIDE 10

Markov&random&fields

[Diebel&and&Thrun&2006]

spatial4depth&super4resolution

[Yang&et&al.&2007]

joint4bilateral&upsampling

[Kopf&et&al.&2007]

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(depth(upsampling

R

slide-11
SLIDE 11

Markov&random&fields

[Diebel&and&Thrun&2006]

spatial4depth&super4resolution

[Yang&et&al.&2007]

joint4bilateral&upsampling

[Kopf&et&al.&2007]

noise4aware&filtering

[Chan&et&al.&2008]

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(depth(upsampling

R

slide-12
SLIDE 12

Markov&random&fields

[Diebel&and&Thrun&2006]

spatial4depth&super4resolution

[Yang&et&al.&2007]

joint4bilateral&upsampling

[Kopf&et&al.&2007]

noise4aware&filtering

[Chan&et&al.&2008]

time4of4flight&+&stereo

[Beder&et&al.&2007,&Zhu&et&al.&2008]

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(depth(upsampling

R

time4of4flight stereo&vision fused&depth&map [Zhu&et&al.&2008]

slide-13
SLIDE 13

Markov&random&fields

[Diebel&and&Thrun&2006]

spatial4depth&super4resolution

[Yang&et&al.&2007]

joint4bilateral&upsampling

[Kopf&et&al.&2007]

noise4aware&filtering

[Chan&et&al.&2008]

time4of4flight&+&stereo

[Beder&et&al.&2007,&Zhu&et&al.&2008]

upsampling&dynamic&range&data

[Dolson&et&al.&2010]

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(depth(upsampling

R

slide-14
SLIDE 14

NPR&camera

[Raskar&et&al.&2004]

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(depthDbased(stylisation

S

slide-15
SLIDE 15

NPR&camera

[Raskar&et&al.&2004]

_.`4D&video&stylisation

[Snavely&et&al.&2006]

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(depthDbased(stylisation

S

slide-16
SLIDE 16

NPR&camera

[Raskar&et&al.&2004]

_.`4D&video&stylisation

[Snavely&et&al.&2006]

photometric&surface&enhancement

[Malzbender&et&al.&2006]

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(depthDbased(stylisation

S

slide-17
SLIDE 17

NPR&camera

[Raskar&et&al.&2004]

_.`4D&video&stylisation

[Snavely&et&al.&2006]

photometric&surface&enhancement

[Malzbender&et&al.&2006]

Images&with&normals

[Toler4Franklin&et&al.&2007]

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(depthDbased(stylisation

S

slide-18
SLIDE 18

NPR&camera

[Raskar&et&al.&2004]

_.`4D&video&stylisation

[Snavely&et&al.&2006]

photometric&surface&enhancement

[Malzbender&et&al.&2006]

Images&with&normals

[Toler4Franklin&et&al.&2007]

context4aware&light&source

[Wang&et&al.&2010]

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Related(work(–(depthDbased(stylisation

S

slide-19
SLIDE 19

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Prototype(camera(hardware

depth&sensor: MESA&Imaging&SRfggg hij&×&hff&resolution& video&camera: PointGrey&Flea_ hg_f&×&ijl&resolution hardware&synchronised

T

slide-20
SLIDE 20

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Microsoft(Kinect

low4cost&IR4based&active&stereo&+&colour&camera&in&one&case

  • ur&approach&is&also&applicable&to&the&Kinect

but&our&prototype&gives&us&full&hardware&+&software&control

U

slide-21
SLIDE 21

Resolution&mismatch&

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Points(to(address

V

hij&×&hff hg_f&×&ijl

slide-22
SLIDE 22

Resolution&mismatch& Video&alignment

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Points(to(address

V

slide-23
SLIDE 23

Resolution&mismatch& Video&alignment Noisy&depth&data

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Points(to(address

V

slide-24
SLIDE 24

Resolution&mismatch& Video&alignment Noisy&depth&data Half4occlusions

background depth camera video camera f

  • r

e g r

  • u

n d half occlusions

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Points(to(address

V

slide-25
SLIDE 25

RGBZ%video%effects video%processing%pipeline

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos WX

video( alignment invalidation &(fillDin geometry( filtering video(abstraction strokeDbased(rendering background(segmentation stereoscopic(JD video(relighting

slide-26
SLIDE 26

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Video(alignment

WO

depth&map colour&image

slide-27
SLIDE 27

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Video(alignment

WO

depth&map colour&image

slide-28
SLIDE 28

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Video(alignment

WO

depth&map colour&image

slide-29
SLIDE 29

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Video(alignment

WO

depth&map colour&image

slide-30
SLIDE 30

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Video(alignment

WO

depth&map colour&image

slide-31
SLIDE 31

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Video(alignment

WO

depth&map colour&image

slide-32
SLIDE 32

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Video(alignment

WO

depth&map colour&image

slide-33
SLIDE 33

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Video(alignment

WO

depth&map colour&image

slide-34
SLIDE 34

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Video(alignment

WO

depth&map colour&image

slide-35
SLIDE 35

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Video(alignment

WO

depth&map colour&image

slide-36
SLIDE 36

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WP

background foreground depth& camera colour& camera

slide-37
SLIDE 37

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WP

background foreground depth& camera colour& camera

slide-38
SLIDE 38

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WP

background foreground depth& camera colour& camera

slide-39
SLIDE 39

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WP

background foreground depth& camera colour& camera

slide-40
SLIDE 40

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WP

background foreground depth& camera colour& camera

slide-41
SLIDE 41

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WP

background foreground depth& camera colour& camera

slide-42
SLIDE 42

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WP

background foreground depth& camera colour& camera

slide-43
SLIDE 43

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WP

background foreground depth& camera colour& camera

slide-44
SLIDE 44

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WP

background foreground depth& camera colour& camera

slide-45
SLIDE 45

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WQ

slide-46
SLIDE 46

aligned&geometry&(before&invalidation)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WQ

slide-47
SLIDE 47

invalidated&geometry&(in&orange)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WR

slide-48
SLIDE 48

ig._&ms q_.l&ms single4resolution&fill4in&(σs&=&_i)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WS

slide-49
SLIDE 49

single4resolution&fill4in&(σs&=&hg)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WT

hg.f&ms hf.l&ms

slide-50
SLIDE 50

single4resolution&fill4in&(σs&=&hg)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WT

hg.f&ms hf.l&ms

slide-51
SLIDE 51

single4resolution&fill4in&(σs&=&hg)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WT

hg.f&ms hf.l&ms

slide-52
SLIDE 52

hs.i&ms h`.h&ms

  • ur&multi4resolution&fill4in&(n&=&s,&g&=&s,&σs&=&s)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WU

slide-53
SLIDE 53

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WV

colour&image&(level&k&=&g) invalidated&(level&k&=&g)

slide-54
SLIDE 54

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WV

colour&image&(level&k&=&h) colour&image&(level&k&=&g) invalidated&(level&k&=&g) invalidated&(level&k&=&h)

slide-55
SLIDE 55

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WV

colour&image&(level&k&=&_) colour&image&(level&k&=&h) colour&image&(level&k&=&g) invalidated&(level&k&=&g) invalidated&(level&k&=&h) invalidated&(level&k&=&_)

slide-56
SLIDE 56

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WV

colour&image&(level&k&=&_) colour&image&(level&k&=&h) colour&image&(level&k&=&g) invalidated&(level&k&=&g) invalidated&(level&k&=&h) filled4in&(level&k&=&_)

slide-57
SLIDE 57

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WV

colour&image&(level&k&=&h) colour&image&(level&k&=&g) invalidated&(level&k&=&g) invalidated&(level&k&=&h)

slide-58
SLIDE 58

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WV

colour&image&(level&k&=&h) colour&image&(level&k&=&g) invalidated&(level&k&=&g) invalidated&(level&k&=&h) sparsely&upsampled&(k&=&h)

slide-59
SLIDE 59

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WV

colour&image&(level&k&=&h) colour&image&(level&k&=&g) invalidated&(level&k&=&g) invalidated&(level&k&=&h) filled4in&(level&k&=&h)

slide-60
SLIDE 60

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Geometry(invalidation(and(fillDin

WV

colour&image&(level&k&=&h) colour&image&(level&k&=&g) invalidated&(level&k&=&g) invalidated&(level&k&=&h) filled4in&(level&k&=&g)

slide-61
SLIDE 61

f

S(x, t) =

X

y∈Nx

w(x, y) · d(y, t) X

y∈Nx

w(x, y)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

SpatialDonly(geometry(filtering

OX

slide-62
SLIDE 62

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

SpatialDonly(geometry(filtering

OX

f

S(x, t) =

X

y∈Nx

w(x, y) · d(y, t)

slide-63
SLIDE 63

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

SpatialDonly(geometry(filtering

OX

f

S(x, t) =

X

y∈Nx

wc(x, y) · wd(x, y) · ws(x, y) · d(y, t) · d(y, t)

slide-64
SLIDE 64

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

SpatialDonly(geometry(filtering

OX

f

S(x, t) =

X

y∈Nx

wc(x, y) · wd(x, y) · ws(x, y) · d(y, t) · d(y, t) wc(x, y) = exp ⇣ gc · ki(x, t) i(y, t)k2 / 2σ2

c

⌘ colour& weight

slide-65
SLIDE 65

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

SpatialDonly(geometry(filtering

OX

f

S(x, t) =

X

y∈Nx

wc(x, y) · wd(x, y) · ws(x, y) · d(y, t) · d(y, t) wc(x, y) = exp ⇣ gc · ki(x, t) i(y, t)k2 / 2σ2

c

⌘ colour& weight wd(x, y) = exp ⇣ − |d(x, t) − d(y, t)|2 / 2σ2

d

⌘ distance& weight

slide-66
SLIDE 66

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

SpatialDonly(geometry(filtering

OX

f

S(x, t) =

X

y∈Nx

wc(x, y) · wd(x, y) · ws(x, y) · d(y, t) · d(y, t) wc(x, y) = exp ⇣ gc · ki(x, t) i(y, t)k2 / 2σ2

c

⌘ colour& weight wd(x, y) = exp ⇣ − |d(x, t) − d(y, t)|2 / 2σ2

d

⌘ distance& weight ws(x, y) = exp ⇣ kx yk2 / 2σ2

s

⌘ spatial& weight

slide-67
SLIDE 67

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

SpatialDonly(filtering(results

OW

slide-68
SLIDE 68

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Spatiotemporal(geometry(filtering

OO

spatiotemporal&filter

f

ST (x, t)

slide-69
SLIDE 69

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Spatiotemporal(geometry(filtering

OO

spatiotemporal&filter spatial&filter

f

ST (x, t) = ϕ · f S(x, t)

slide-70
SLIDE 70

f

ST (x, t) = ϕ · f S(x, t) + (1 − ϕ) · f T (x, t) Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Spatiotemporal(geometry(filtering

OO

spatiotemporal&filter temporal&filter spatial&filter

slide-71
SLIDE 71

· f

ST (y, t−1)

f

ST (x, t) = ϕ · f S(x, t) + (1 − ϕ) · f T (x, t)

f

T (x, t) =

X

y∈Nx

w(x, y, x, y)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Spatiotemporal(geometry(filtering

OO

spatiotemporal&filter temporal&filter spatial&filter

slide-72
SLIDE 72

· f

ST (y, t−1)

f

ST (x, t) = ϕ · f S(x, t) + (1 − ϕ) · f T (x, t)

f

T (x, t) =

X

y∈Nx

w(x, y, x, y)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Spatiotemporal(geometry(filtering

OO

spatiotemporal&filter temporal&filter spatial&filter

slide-73
SLIDE 73

f

ST (x, t) = ϕ · f S(x, t) + (1 − ϕ) · f T (x, t)

f

T (x, t) =

X

y∈Nx

wc(x, y)·wd(x, y)·ws(x, y)·wf (y, y)·f

ST (y, t−1) Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Spatiotemporal(geometry(filtering

OO

spatiotemporal&filter temporal&filter spatial&filter

slide-74
SLIDE 74

f

ST (x, t) = ϕ · f S(x, t) + (1 − ϕ) · f T (x, t)

f

T (x, t) =

X

y∈Nx

wc(x, y)·wd(x, y)·ws(x, y)·wf (y, y)·f

ST (y, t−1)

f

S(x, t) =

X

y∈Nx

wc(x, y)·wd(x, y)·ws(x, y)·d(y, t)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Spatiotemporal(geometry(filtering

OO

spatiotemporal&filter temporal&filter spatial&filter

slide-75
SLIDE 75

f

ST (x, t) = ϕ · f S(x, t) + (1 − ϕ) · f T (x, t)

f

T (x, t) =

X

y∈Nx

wc(x, y)·wd(x, y)·ws(x, y)·wf (y, y)·f

ST (y, t−1)

f

S(x, t) =

X

y∈Nx

wc(x, y)·wd(x, y)·ws(x, y)·d(y, t)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Spatiotemporal(geometry(filtering

OO

spatiotemporal&filter temporal&filter spatial&filter

slide-76
SLIDE 76

f

ST (x, t) = ϕ · f S(x, t) + (1 − ϕ) · f T (x, t)

f

T (x, t) =

X

y∈Nx

wc(x, y)·wd(x, y)·ws(x, y)·wf (y, y)·f

ST (y, t−1)

f

S(x, t) =

X

y∈Nx

wc(x, y)·wd(x, y)·ws(x, y)·d(y, t)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Spatiotemporal(geometry(filtering

OO

spatiotemporal&filter temporal&filter spatial&filter

slide-77
SLIDE 77

f

ST (x, t) = ϕ · f S(x, t) + (1 − ϕ) · f T (x, t)

f

T (x, t) =

X

y∈Nx

wc(x, y)·wd(x, y)·ws(x, y)·wf (y, y)·f

ST (y, t−1)

f

S(x, t) =

X

y∈Nx

wc(x, y)·wd(x, y)·ws(x, y)·d(y, t)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Spatiotemporal(geometry(filtering

OO

spatiotemporal&filter temporal&filter spatial&filter

slide-78
SLIDE 78

f

ST (x, t) = ϕ · f S(x, t) + (1 − ϕ) · f T (x, t)

f

T (x, t) =

X

y∈Nx

wc(x, y)·wd(x, y)·ws(x, y)·wf (y, y)·f

ST (y, t−1)

f

S(x, t) =

X

y∈Nx

wc(x, y)·wd(x, y)·ws(x, y)·d(y, t)

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Spatiotemporal(geometry(filtering

OO

spatiotemporal&filter temporal&filter spatial&filter

wf (y, y) = exp ⇣ ky yk2 / 2σ2

f

⌘

flow&weight

slide-79
SLIDE 79

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Spatiotemporal(filtering(results

OP

slide-80
SLIDE 80

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

RGBZ(video(effects

video&relighting geometry4based&video&abstraction stroke4based&video&rendering background&segmentation stereoscopic&sD&rendering

OQ

slide-81
SLIDE 81

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Video(relighting

OR

slide-82
SLIDE 82

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

GeometryDbased(video(abstraction

OS

slide-83
SLIDE 83

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

StrokeDbased(video(rendering

OT

slide-84
SLIDE 84

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Background(segmentation

OU

slide-85
SLIDE 85

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Stereoscopic(JD(rendering

OV

slide-86
SLIDE 86

unreliable&optical&flow&can&lead&to&smearing&artefacts assumption&of&coincident&colour&+&depth&edges ‘texture&copy’&artefacts&in&the&distance&map edges&with&small&colour&differences&not&preserved&well depth&detail&limited&by&time4of4flight&camera&resolution joint4bilateral&filter&not&guaranteed&to&be&optimal: new&values&are&a&linear&combination&of&existing&values

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Limitations

PX

slide-87
SLIDE 87

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos

Future(work

improve&preservation&of&features could&refine&results&using&shape4from4shading formulate&optical&flow&that&respects&depth&discontinuities would&prevent&‘smearing’&artefacts&in&the&distance&map commodification&of&RGBZ&video&cameras&and&effects: miniaturisation&of&camera&hardware improvements&in&hardware&performance& algorithmic&optimisations

PW

slide-88
SLIDE 88

Summary

introduced&a&novel&set&of&efficient&and&effective&depth& filtering&and&upsampling&techniques&for&RGBZ&videos: a&fast&fill4in&procedure&for&unreliable&geometry a&multi4lateral&spatiotemporal&filtering&approach illustrated&the&benefits&of&RGBZ&video&for&effects source&code&and&data&sets&are&available&on&our&project&page&at http://richardt.name/rgbz2camera/

Christian)Richardt)et)al. http://richardt.name/rgbz:camera/ Eurographics)2012,)Cagliari,)Italy Coherent)Spatiotemporal)Filtering,)Upsampling)and)Rendering)of)RGBZ)Videos PO

Hire4me!4I’m4looking4for4a4postdoc4from4October4>?@>.