Image Processing Framework FELIX HEIDE 1,2 MARKUS STEINBERGER 3 - - PowerPoint PPT Presentation

β–Ά
image processing framework
SMART_READER_LITE
LIVE PREVIEW

Image Processing Framework FELIX HEIDE 1,2 MARKUS STEINBERGER 3 - - PowerPoint PPT Presentation

FlexISP: A Flexible Camera Image Processing Framework FELIX HEIDE 1,2 MARKUS STEINBERGER 3 YUN-TA TSAI 1 NASA ROUF 1,2 DAWID PAJK 1 DIKPAL REDDY 1 ORAZIO GALLO 1 JING LIU 4 WOLFGANG HEIDRICH 5 KAREN EGIAZARIAN 1,6 JAN KAUTZ 1 KARI PULLI 1 1


slide-1
SLIDE 1

FELIX HEIDE1,2 MARKUS STEINBERGER3 YUN-TA TSAI1 NASA ROUF1,2 DAWID PAJΔ„K1 DIKPAL REDDY1 ORAZIO GALLO1 JING LIU4 WOLFGANG HEIDRICH5 KAREN EGIAZARIAN1,6 JAN KAUTZ1 KARI PULLI1

FlexISP: A Flexible Camera Image Processing Framework

1 NVIDIA 2 UBC 3 TU GRAZ 4 UCSC 5 KAUST 6 TUT

slide-2
SLIDE 2

CAMERA IMAGE PROCESSING

slide-3
SLIDE 3
slide-4
SLIDE 4

4 Image Signal Processor (ISP)

slide-5
SLIDE 5

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-6
SLIDE 6

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-7
SLIDE 7

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-8
SLIDE 8

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-9
SLIDE 9

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-10
SLIDE 10

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-11
SLIDE 11

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-12
SLIDE 12

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-13
SLIDE 13

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-14
SLIDE 14

IMAGE PROCESSING PIPELINE

Demosaic Denoise Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-15
SLIDE 15

IMAGE PROCESSING PIPELINE

Demosaic Denoise Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-16
SLIDE 16

IMAGE PROCESSING PIPELINE

Demosaic Denoise Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-17
SLIDE 17

Long Exposure Short Exposure Long Exposure Short Exposure

NOVEL FEATURES

slide-18
SLIDE 18

Single Exposure Interleaved Exposure

NOVEL FEATURES

slide-19
SLIDE 19

NOVEL FEATURES

slide-20
SLIDE 20
slide-21
SLIDE 21

Refocusing Depth

slide-22
SLIDE 22
slide-23
SLIDE 23
slide-24
SLIDE 24

Kepler GPU

slide-25
SLIDE 25

OBJECTIVES

NO PIPELINE

FLEXIBILITY

PERFORMANCE

slide-26
SLIDE 26

PROPOSED SOLUTION

FLEXISP Bayer Burst iHDR Others

…

slide-27
SLIDE 27

x B D z

IMAGE FORMATION

slide-28
SLIDE 28

x B D z

z = DBx

IMAGE FORMATION

slide-29
SLIDE 29

z = A x

IMAGE FORMATION

B D

slide-30
SLIDE 30

LINEAR SYSTEM

π‘π‘ π‘•π‘›π‘—π‘œπ‘¦ 𝑨 βˆ’ 𝐡𝑦 2

2

slide-31
SLIDE 31

LINEAR SYSTEM

errors w.r.t. observation x A

π‘π‘ π‘•π‘›π‘—π‘œπ‘¦ 𝑨 βˆ’ 𝐡𝑦 2

2

slide-32
SLIDE 32

LINEAR SYSTEM

errors w.r.t. observation x A

π‘π‘ π‘•π‘›π‘—π‘œπ‘¦ 𝑨 βˆ’ 𝐡𝑦 2

2

slide-33
SLIDE 33

LINEAR SYSTEM

regularization

π‘π‘ π‘•π‘›π‘—π‘œπ‘¦ 𝑨 βˆ’ 𝐡𝑦 2

2 + Ξ»(𝑦)

slide-34
SLIDE 34

TOTAL VARIATION

(Rudin et al. β€˜92)

slide-35
SLIDE 35

CROSS-CHANNEL

(Heide et al. β€˜13)

slide-36
SLIDE 36

COLLABORATIVE DENOISING

slide-37
SLIDE 37

COLLABORATIVE DENOISING

slide-38
SLIDE 38

LINEAR SYSTEM

Total variation Cross-channel

+ +

Collaborative denoising

π‘π‘ π‘•π‘›π‘—π‘œπ‘¦ 𝑨 βˆ’ 𝐡𝑦 2

2 + Ξ»(𝑦)

slide-39
SLIDE 39

IMPORTANCE OF PRIORS

slide-40
SLIDE 40

MORE ON PRIORS…

TV [Rudin et al. β€˜92] EPLL [Zoran and Weiss β€˜11] Curvelets [CandΓ¨s and Donoho β€˜99]

slide-41
SLIDE 41

MORE ON PRIORS…

slide-42
SLIDE 42

LINEAR SYSTEM

Convex Optimization (Mostly)

π‘π‘ π‘•π‘›π‘—π‘œπ‘¦ 𝑨 βˆ’ 𝐡𝑦 2

2 + Ξ»(𝑦)

slide-43
SLIDE 43

CONVERGENCE

slide-44
SLIDE 44

RESULTS

slide-45
SLIDE 45

Bayer Burst iHDR

CAMERA SYSTEMS

slide-46
SLIDE 46

Bayer Burst iHDR

CAMERA SYSTEMS

slide-47
SLIDE 47

Bayer Burst iHDR

CAMERA SYSTEMS A

slide-48
SLIDE 48
slide-49
SLIDE 49
slide-50
SLIDE 50
slide-51
SLIDE 51
slide-52
SLIDE 52
slide-53
SLIDE 53
slide-54
SLIDE 54

Bayer Burst iHDR

CAMERA SYSTEMS

slide-55
SLIDE 55

Bayer Burst iHDR

CAMERA SYSTEMS

slide-56
SLIDE 56

Bayer Burst iHDR

CAMERA SYSTEMS A

slide-57
SLIDE 57
slide-58
SLIDE 58
slide-59
SLIDE 59
slide-60
SLIDE 60
slide-61
SLIDE 61
slide-62
SLIDE 62
slide-63
SLIDE 63
slide-64
SLIDE 64

Bayer Burst iHDR

CAMERA SYSTEMS

slide-65
SLIDE 65

Bayer Burst iHDR

CAMERA SYSTEMS

slide-66
SLIDE 66

Bayer Burst iHDR

CAMERA SYSTEMS A

slide-67
SLIDE 67
slide-68
SLIDE 68
slide-69
SLIDE 69
slide-70
SLIDE 70
slide-71
SLIDE 71
slide-72
SLIDE 72
slide-73
SLIDE 73

PERFORMANCE

0.82 7.45 2.13 Image burst (0.4MPx16) iHDR (13MP) Demosaic (5MP) TIME IN SECONDS

GTX Titan

slide-74
SLIDE 74

PERFORMANCE

16.7 174.4 40.5 Image burst (0.4MPx16) iHDR (13MP) Demosaic (5MP) TIME IN SECONDS

Tegra K1

slide-75
SLIDE 75

MORE ON PRIORS…

slide-76
SLIDE 76

PERFORMANCE (BM3D)

16.7 174.4 40.5 Image burst (0.4MPx16) iHDR (13MP) Demosaic (5MP) TIME IN SECONDS

Tegra K1

slide-77
SLIDE 77

PERFORMANCE (AVERAGING NLM)

6.6 33.2 6.7 Image burst (0.4MPx16) iHDR (13MP) Demosaic (5MP) TIME IN SECONDS

Tegra K1

slide-78
SLIDE 78
slide-79
SLIDE 79
slide-80
SLIDE 80

CONCLUSION

slide-81
SLIDE 81

Bayer Burst iHDR Color Camera Array

+ others such as: deconvolution, super-resolution, JPEG compression and de-blocking.

slide-82
SLIDE 82

Bayer Burst AOHDR Color Camera Array FLEXISP

A

slide-83
SLIDE 83

THANK YOU!