HDR videos acquisition dr. Francesco Banterle - - PowerPoint PPT Presentation

▶
hdr videos acquisition
SMART_READER_LITE
LIVE PREVIEW

HDR videos acquisition dr. Francesco Banterle - - PowerPoint PPT Presentation

Mar 11, 2023 26 likes •583 views

HDR videos acquisition dr. Francesco Banterle francesco.banterle@isti.cnr.it How to capture? Videos are challenging: We need to capture multiple frames at different exposure times and everything moves How to capture?

slide-1
SLIDE 1

HDR videos acquisition

  • dr. Francesco Banterle

francesco.banterle@isti.cnr.it

slide-2
SLIDE 2

How to capture?

  • Videos are challenging:
  • We need to capture multiple frames at different

exposure times

  • … and everything moves
slide-3
SLIDE 3

How to capture?

  • Different technologies based on exposure

bracketing:

  • beam-splitter; i.e. many sensors one lens
  • stereo/multi-view HDR capturing
  • varying exposure per pixel; i.e. bayer pattern
  • varying shutter speed
slide-4
SLIDE 4

Multi-sensors cameras

  • Idea: to use more sensors to capture the same

scene

  • The light path is divided using beam splitters:
  • careful alignment
slide-5
SLIDE 5

Multi-sensors cameras

slide-6
SLIDE 6

Multi-sensors cameras

  • Debayering after HDR-merging
  • why not before?
  • It can corrupt colors in saturated regions
  • It makes less visible sub-pixel misalignments
  • f sensors
slide-7
SLIDE 7

Multi-sensors cameras

Figure 1: HDR image acquired with our proposed system. On the left we show the final image acquired with our camera and merged with

“A Versatile HDR Video Production System”. Michael D. Tocci, Chris Kiser, Nora Tocci, Pradeep Sen. ACM SIGGRAPH 2011 Papers program.

slide-8
SLIDE 8

Multi-sensors cameras

  • Advantages:
  • no ghosts
  • no misalignments
  • Disadvantages:
  • high costs: sensors + calibration
  • fixed dynamic range that can be captured
  • reconstruction before debayering: complex reconstruction

algorithms

slide-9
SLIDE 9

Multi-cameras systems

  • Idea: to use more cameras in a rig to capture the

same scene:

  • each camera has a different shutter-speed/ISO
  • A synchronization system is required
slide-10
SLIDE 10

Multi-cameras systems

Camera Linear pattern Square pattern

slide-11
SLIDE 11

Multi-cameras systems: Geometric Calibration

  • Geometric calibration of each camera:
  • Intrinsic parameters: optical center, focale,

pixel size in mm, field of view (angle), and aspect ratio.

  • Extrinsic parameters; world position: position

and rotation

slide-12
SLIDE 12

Multi-cameras systems: Alignment

  • There is the need to align other images onto a

reference image (well-exposed one again!)

  • How?
  • Compute disparity map
  • Warp images
slide-13
SLIDE 13

Multi-cameras systems: Disparity Computation

SSD(u, v, d) =

n

X

k=−n m

X

l=−m

✓ I1(u + k, v + l) − I2(u + k + d, v + l) ◆2 do(u, v) = arg min

d SSD(u, v, d)

Note: typically n = m

slide-14
SLIDE 14

I1 I2

Multi-cameras systems: Disparity Computation

slide-15
SLIDE 15

I1 I2

Multi-cameras systems: Disparity Computation

slide-16
SLIDE 16

I1 I2

Multi-cameras systems: Disparity Computation

slide-17
SLIDE 17

I1 I2

Multi-cameras systems: Disparity Computation

slide-18
SLIDE 18

I1 I2

Multi-cameras systems: Disparity Computation

slide-19
SLIDE 19

I1 I2

Multi-cameras systems: Disparity Computation

slide-20
SLIDE 20

I1 I2

Multi-cameras systems: Disparity Computation

slide-21
SLIDE 21

I1 I2

Multi-cameras systems: Disparity Computation

slide-22
SLIDE 22

Multi-cameras systems: disparity computation

http://vision.middlebury.edu/stereo/data/

slide-23
SLIDE 23

Multi-cameras systems: disparity computation

http://vision.middlebury.edu/stereo/data/

slide-24
SLIDE 24

Multi-cameras systems: warping

http://vision.middlebury.edu/stereo/data/

slide-25
SLIDE 25

Multi-cameras systems: warping

http://vision.middlebury.edu/stereo/data/

slide-26
SLIDE 26

Multi-cameras systems: warping

http://vision.middlebury.edu/stereo/data/

slide-27
SLIDE 27

Multi-cameras systems: warping

http://vision.middlebury.edu/stereo/data/

slide-28
SLIDE 28

Multi-cameras systems: warping

http://vision.middlebury.edu/stereo/data/

slide-29
SLIDE 29

Multi-sensors cameras

  • Advantages:
  • no ghosts
  • Disadvantages:
  • misalignments + occlusions
  • high costs: sensors + sync
  • fixed dynamic range that can be captured
slide-30
SLIDE 30

Varying exposure per pixel

  • Idea: to apply bayer pattern not only for RGB colors

but also to exposure

  • Two possible solutions:
  • varying gain
  • a mask with varying neutral density filters:
  • shutter time is not modified!
slide-31
SLIDE 31

Varying exposure per pixel

interleaved rows checkboard pattern

slide-32
SLIDE 32

Varying exposure per pixel

slide-33
SLIDE 33

Varying exposure per pixel

slide-34
SLIDE 34

Varying exposure per pixel: reconstruction

Zo Zr ˆ Z

slide-35
SLIDE 35

Varying exposure per pixel: reconstruction

Zo Zr ˆ Z

slide-36
SLIDE 36

Varying exposure per pixel: reconstruction

  • How can reconstruction be carried out?
  • Linear interpolation can lead to artifacts
  • Cubic interpolation; close to ideal sinc:

Zr(x, y) =

3

X

i=0 3

X

j=0

f(1.5 − i, 1.5 − j)Zo(x − 1.5 + i, y − 1.5 + j)

Kernel Signal Reconstructed

slide-37
SLIDE 37
  • Let’s see the matrix form:

Varying exposure per pixel: reconstruction

Zr = FZo ˆ Z = FZo Zo = F−ˆ Z F− = FT (FFT )−1

slide-38
SLIDE 38

Varying exposure per pixel

slide-39
SLIDE 39

Varying exposure per pixel

slide-40
SLIDE 40

Varying exposure per pixel

slide-41
SLIDE 41

Varying exposure per pixel

  • Advantages:
  • low cost hardware: programmable videocameras; e.g. Canon

DSLR with Magic Lantern

  • no ghosts
  • no misalignments
  • Disadvantages:
  • limited to 2-3 exposure images
  • masks may be expensive to manufacture and difficult to align to

an existing bayer pattern

slide-42
SLIDE 42

Varying Shutter Speed

  • Idea: to program the shutter speed or ISO; i.e.

varying it at each frame

  • Requirements:
  • high frame rate videocamera
  • programmable hardware
slide-43
SLIDE 43

Varying Shutter Speed

Courtesy of Jonas Unger

time 0 time 1 time 2

slide-44
SLIDE 44

Varying Shutter Speed: reconstruction

  • There is the need to align other images onto a

reference image (well-exposed one again!)

  • How?
  • Compute Motion Estimation
  • Warp images
slide-45
SLIDE 45

Varying Shutter Speed: Motion Estimation

frame t frame t+1 It(i, i) = It+1(i + u, i + v) u v

slide-46
SLIDE 46

Varying Shutter Speed: Motion Estimation

SSD(i, j, u, v) =

n

X

k=−n m

X

l=−m

✓ I1(i + k, j + l) − I2(i + k + u, j + l + v) ◆2 OFo(i, j) = arg min

u,v SSD(i, j, u, v)

Note: this is a generalization of the disparity problem

slide-47
SLIDE 47

Varying Shutter Speed: Motion Estimation

Image courtesy of Jonas Unger

slide-48
SLIDE 48

per block motion estimation

slide-49
SLIDE 49

Varying Shutter Speed: Warp

Image courtesy of Jonas Unger

slide-50
SLIDE 50

Varying Shutter Speed: Warp

Image courtesy of Jonas Unger

slide-51
SLIDE 51

Varying Shutter Speed: Warp

Image courtesy of Jonas Unger

slide-52
SLIDE 52

Varying Shutter Speed: Warp

Image courtesy of Jonas Unger

slide-53
SLIDE 53

Varying Shutter Speed: Warp

Image courtesy of Jonas Unger

slide-54
SLIDE 54

Varying Shutter Speed: Warp

  • Advantages:
  • low cost hardware: high frame rate and programmable

videocameras; e.g. Canon DSLR with Magic Lantern

  • Disadvantages:
  • limited to 2-3 exposure images
  • moving camera and scene:
  • camera alignment
  • moving scene
slide-55
SLIDE 55

Questions?