Chapter 4 Multiframe Super-resolution Image Reconstruction 1 - - PDF document

▶
chapter 4 multiframe super resolution image reconstruction
SMART_READER_LITE
LIVE PREVIEW

Chapter 4 Multiframe Super-resolution Image Reconstruction 1 - - PDF document

Jul 25, 2023 227 likes •500 views

Chapter 4 Multiframe Super-resolution Image Reconstruction 1 Multi-frame SRIR (Video Enhancement) Low resolution video Our method LR image Source: https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html 2 1 Multi-frame SRIR

slide-1
SLIDE 1

1

1

Multiframe Super-resolution Image Reconstruction Chapter 4

LR image

2

Multi-frame SRIR (Video Enhancement)

Low resolution video Our method

Source: https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html

slide-2
SLIDE 2

2

3

Input Video High-definition Video

Multi-frame SRIR (Video Enhancement)

  • 1. High-resolution Image Reconstruction
  • 2. Video Still Enhancement Models
  • Classical
  • Tight-frame
  • Low-rank
  • 3. Experiments
  • 1. High-resolution Image Reconstruction
  • 2. Video Still Enhancement Models
  • Classical
  • Tight-frame
  • Low-rank
  • 3. Experiments

4

Outline

slide-3
SLIDE 3

3

5

Not one, but many lens— compound eyes

Super-resolution Image Reconstruction (SRIR)

6

Super-resolution Image Reconstruction (SRIR)

slide-4
SLIDE 4

4

7

low-resolution pixel a b c d given intensity = (a+b+c+d)/4 high-resolution pixel LR 1 d LR 2 LR 3 LR 4 d

Modeling of SRIR

8

Modeling of SRIR

slide-5
SLIDE 5

5

9

Modeling of SRIR

10

Modeling of SRIR

slide-6
SLIDE 6

6

11 11

Outline

  • 1. High-resolution Image Reconstruction
  • 2. Video Still Enhancement Models
  • Classical
  • Tight-frame
  • Low-rank
  • 3. Experiments

Bilinear interpolation from 1 frame Tight-frame method using 21 frames

12 12 12

A 352-by-288 video from a video recorder

[C., Shen, & Xia, ACHA 07]

Video Still Enhancement

30 frames/second frames are not aligned at exactly half-pixel length

slide-7
SLIDE 7

7

13

Reference frame t

displacement

Aligning the Frames

14

High- resolution pixels Ideal low- resolution pixel position Displaced low- resolution pixel

Sub-pixel Displacement

slide-8
SLIDE 8

8

15

Modeling of Video Still Enhancement

16

Modeling of SRIR

slide-9
SLIDE 9

9

17

Classical Approach [Tsai & Huang, 84]

18 18

Outline

  • 1. High-resolution Image Reconstruction
  • 2. Video Still Enhancement Models
  • Classical
  • Tight-frame
  • Low-rank
  • 3. Experiments
slide-10
SLIDE 10

10

19

Low- resolution pixel High- resolution pixels

4 1 2 1 4 1 2 1 1 2 1 4 1 2 1 4 1

Tightframe Approach [C2 + S2, SISC (2003)]

Averaging process = a lowpass filter with refinement mask

20

Piecewise Linear Tight Frame

slide-11
SLIDE 11

11

21

From LR Images to HR Image

22

Key Observation

slide-12
SLIDE 12

12

23

Tightframe SRIR

24 24

Outline

  • 1. High-resolution Image Reconstruction
  • 2. Video Still Enhancement Models
  • Classical
  • Tight-frame
  • Low-rank
  • 3. Experiments
slide-13
SLIDE 13

13

25

Low-rank Approach [ArXiv: 170406196]

26

Motion Matrices in 1D

slide-14
SLIDE 14

14

27

Decomposition of the Motion Matrices

28

Creating Low Rank Structure

slide-15
SLIDE 15

15

29

Variational Formulation

30

Motion Matrices in 2D

slide-16
SLIDE 16

16

Creating Low Rank Structure

32

Low-rank Super-resolution Model

slide-17
SLIDE 17

17

33

Nuclear Norm [Candes, Recht, 09; Recht, Fazel, Parrilo, 10]

34

Motion Estimation

slide-18
SLIDE 18

18

35

Local Motion Estimation [C. Gilliam and T. Blu, 2015]

36

Global Motion Estimation

[C., Shen and Xia, ACHA, 2007]

slide-19
SLIDE 19

19

37 37

Outline

  • 1. High-resolution Image Reconstruction
  • 2. Video Still Enhancement Models
  • Classical
  • Tight-frame
  • Low-rank
  • 3. Experiments

21 frames with tightframe

38 38

Bookshelf Video Upsampled by 2

Single frame with bilinear interpolation 21 frames with nuclear norm 21 frames with TV regularization

slide-20
SLIDE 20

20 tightframe (2007)

39 39

Disk Video Upsampled by 2

MAP (2015) nuclear sparse direct’l (‘10) LR video reference frame

Source: https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html

tightframe

40 40

Text Video Upsampled by 2

MAP nuclear sparse directional LR video reference frame

Source: https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html

slide-21
SLIDE 21

21 tightframe

41 41

Alpaca Video Upsampled by 2

MAP nuclear sparse directional LR video reference frame

Source: https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html

tightframe

42 42

Alpaca Video Upsampled by 4

MAP nuclear sparse directional LR video reference frame

Source: https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html

slide-22
SLIDE 22

22 reference image

43 43

Alpaca Image Upsampled by 4

nuclear tightframe

44 44

EIA Video Upsampled by 4

MAP nuclear sparse directional LR video

Source: https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html

reference frame

slide-23
SLIDE 23

23 reference image

45 45

EIA Image Upsampled by 4

nuclear sparse directional

46 46

EIA Image Upsampled by 4

nuclear L1-robust, Farsiu et al. (2004) 10 x 10 (.jpg)

slide-24
SLIDE 24

24 tightframe

47 47

Book-shelf Upsampled by 2

MAP nuclear sparse directional LR video reference frame tightframe

48 48

Bookshelf Video Upsampled by 2

MAP nuclear sparse directional reference frame LR video

slide-25
SLIDE 25

25 true image tightframe

49 49

Boat Video Upsampled by 2

MAP nuclear sparse directional LR video 25.76dB 20.88dB 26.94dB 18.66dB

50

interpolate/inpaint interpolate/inpaint

Single Frame Upsampling

interpolate/inpaint

t t

slide-26
SLIDE 26

26

51

Input Video Qian et al. Siggraph 09: Upsampled by bicubic Level 6 Tightframe

52

Qian et al. Siggraph 09 Upsampled by bicubic Level 6 Tightframe

No thresholding = completely parameter free

Input Video

slide-27
SLIDE 27

27

53

Concluding Remarks

53

  • R. Zhao and R. Chan, arXiv:1704.06196