Sampling Based Scene-Space Video Processing Felix Klose, Oliver - - PDF document

sampling based scene space video processing
SMART_READER_LITE
LIVE PREVIEW

Sampling Based Scene-Space Video Processing Felix Klose, Oliver - - PDF document

5/24/19 Sampling Based Scene-Space Video Processing Felix Klose, Oliver Wang, Jean-Charles Bazin, Marcus Magnor, Alexander Sorkine-Hornung Overview Scene space video processing : pixels are processed according to their 3D positions What


slide-1
SLIDE 1

5/24/19 1

Sampling Based Scene-Space Video Processing

Felix Klose, Oliver Wang, Jean-Charles Bazin, Marcus Magnor, Alexander Sorkine-Hornung

Overview

  • Scene space video processing: pixels are processed according to

their 3D positions

  • What is scene space?
  • Why is scene space processing advantageous?
slide-2
SLIDE 2

5/24/19 2

Challenge

Visual output quality depends on quality of scene space information.

Idea

Do scene-space video processing on casually captured input video by using sample based framework instead of full 3D reconstruction.

slide-3
SLIDE 3

5/24/19 3

Approach

Goal: compute all output pixel colors, Of(p).

Approach

For each Of (p), draw a set of samples Sf (p) from I.

s ∈ ℝ7

r g b x y z f

Filtering: Φ , ∈ - ℝ. → ℝ0

slide-4
SLIDE 4

5/24/19 4

Preprocessing

  • 1. Use commercially available tools to compute camera calibration

parameters for each input frame.

  • 2. Use simple multi view stereo algorithm or Kinect sensor to derive

dense depth information.

Step 1: Sample Gathering

  • Goal: collect multiple observations of the same scene point visible

to an output pixel, p.

  • What is the straightforward way to do this?
  • Why not do this?

1 min. long 60 #$% 1 '() 30 fps 30 +,-'$# 1 #$% × 720p 960 × 720 2(3$4# 1 +,-'$ × = 1,244,160,000 total samples

slide-5
SLIDE 5

5/24/19 5

Step 1: Sample Gathering.

  • px, py = pixel location
  • near, far are depth values for near and far clipping planes
  • Co = output camera matrix
  • l = 3 pixels

For each output pixel p, a physical camera integrates information over a frustum-shaped 3D volume V in scene-space.

Step 1: Sample Gathering

All pixels that project into VO must reside in VJ.

slide-6
SLIDE 6

5/24/19 6

Step 1: Sample Gathering

  • HOWEVER, not all all pixels that reside in VJ project into VO.
  • Why is this?

J

  • Rasterize all pixels q in VJ
  • Check if q projected back into O lies within VO
  • Accept each pixel q that lies within VO

Step 2: Filtering

  • Some pixels are less trustable – why?
  • w(s) : application specific weighting function
  • W : the sum of all weights
slide-7
SLIDE 7

5/24/19 7

Denoising

  • Denoise by averaging multiple observations at same scene point.
  • Why not just set w(s) = 1?

Of = If

If(p)

Denoising

  • sref : sample that originates from Input projection into scene space

r g b x y z f r 40 … … … … … … g … 40 … … … … … b … … 40 … … … … x … … … 10 … … … y … … … … 10 … … z … ... … … … 10 … f … … … … … … 6

slide-8
SLIDE 8

5/24/19 8

Denoising Super Resolution

  • Assumption: each scene point is most clearly recorded when it is
  • bserved from as close as possible
  • pl and pr: left and right pixel edge locations
  • C: camera matrix
  • sf : sample’s frame
slide-9
SLIDE 9

5/24/19 9

Super Resolution Deblurring

  • ∇"#$ % ∶ gradient operator for the frame that the sample s
  • riginated from.
  • down-weights contribution from blurry frames
  • σrgb = 200, σxyz = 10, σf = 20
slide-10
SLIDE 10

5/24/19 10

Deblurring Inpainting

  • Requires a user specified mask M where:
  • M(p) = 1 means pixel should be removed.
  • M(p) = 0 otherwise
  • Don’t have reference sref.
  • Weighting function:.
slide-11
SLIDE 11

5/24/19 11

Inpainting Computational scene-space shutters

slide-12
SLIDE 12

5/24/19 12

Computational scene-space shutters

Where ξ(sf): box function in a typical camera With reasonable depth values:

Computational scene-space shutters

slide-13
SLIDE 13

5/24/19 13

Virtual aperture

  • a0: Thinnest point of cone
  • z0: focal point
  • as: slope of cone

a(z) = a0 +| z0−z|as

Virtual aperture

slide-14
SLIDE 14

5/24/19 14