Image Processing Framework FELIX HEIDE 1,2 MARKUS STEINBERGER 3 - - PowerPoint PPT Presentation

▶

image processing framework

Image Processing Framework FELIX HEIDE 1,2 MARKUS STEINBERGER 3 - - PowerPoint PPT Presentation

Mar 29, 2024 478 likes •1.33k views

FlexISP: A Flexible Camera Image Processing Framework FELIX HEIDE 1,2 MARKUS STEINBERGER 3 YUN-TA TSAI 1 NASA ROUF 1,2 DAWID PAJK 1 DIKPAL REDDY 1 ORAZIO GALLO 1 JING LIU 4 WOLFGANG HEIDRICH 5 KAREN EGIAZARIAN 1,6 JAN KAUTZ 1 KARI PULLI 1 1

slide-1

SLIDE 1

FELIX HEIDE1,2 MARKUS STEINBERGER3 YUN-TA TSAI1 NASA ROUF1,2 DAWID PAJĄK1 DIKPAL REDDY1 ORAZIO GALLO1 JING LIU4 WOLFGANG HEIDRICH5 KAREN EGIAZARIAN1,6 JAN KAUTZ1 KARI PULLI1

FlexISP: A Flexible Camera Image Processing Framework

1 NVIDIA 2 UBC 3 TU GRAZ 4 UCSC 5 KAUST 6 TUT

slide-2

SLIDE 2

CAMERA IMAGE PROCESSING

slide-3

SLIDE 3

slide-4

SLIDE 4

4 Image Signal Processor (ISP)

slide-5

SLIDE 5

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-6

SLIDE 6

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-7

SLIDE 7

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-8

SLIDE 8

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-9

SLIDE 9

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-10

SLIDE 10

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-11

SLIDE 11

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-12

SLIDE 12

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-13

SLIDE 13

IMAGE PROCESSING PIPELINE

Denoise Demosaic Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-14

SLIDE 14

IMAGE PROCESSING PIPELINE

Demosaic Denoise Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-15

SLIDE 15

IMAGE PROCESSING PIPELINE

Demosaic Denoise Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-16

SLIDE 16

IMAGE PROCESSING PIPELINE

Demosaic Denoise Bad Pixel Correction

Image Enhancing

Tone Mapping Lens Correction Black Level Metering

Exposure / Focus Adjustment

slide-17

SLIDE 17

Long Exposure Short Exposure Long Exposure Short Exposure

NOVEL FEATURES

slide-18

SLIDE 18

Single Exposure Interleaved Exposure

NOVEL FEATURES

slide-19

SLIDE 19

NOVEL FEATURES

slide-20

SLIDE 20

slide-21

SLIDE 21

Refocusing Depth

slide-22

SLIDE 22

slide-23

SLIDE 23

slide-24

SLIDE 24

Kepler GPU

slide-25

SLIDE 25

OBJECTIVES

NO PIPELINE

FLEXIBILITY

PERFORMANCE

slide-26

SLIDE 26

PROPOSED SOLUTION

FLEXISP Bayer Burst iHDR Others

…

slide-27

SLIDE 27

x B D z

IMAGE FORMATION

slide-28

SLIDE 28

x B D z

z = DBx

IMAGE FORMATION

slide-29

SLIDE 29

z = A x

IMAGE FORMATION

B D

slide-30

SLIDE 30

LINEAR SYSTEM

𝑏𝑠𝑕𝑛𝑗𝑜𝑦 𝑨 − 𝐵𝑦 2

2

slide-31

SLIDE 31

LINEAR SYSTEM

errors w.r.t. observation x A

𝑏𝑠𝑕𝑛𝑗𝑜𝑦 𝑨 − 𝐵𝑦 2

2

slide-32

SLIDE 32

LINEAR SYSTEM

errors w.r.t. observation x A

𝑏𝑠𝑕𝑛𝑗𝑜𝑦 𝑨 − 𝐵𝑦 2

2

slide-33

SLIDE 33

LINEAR SYSTEM

regularization

𝑏𝑠𝑕𝑛𝑗𝑜𝑦 𝑨 − 𝐵𝑦 2

2 + λ(𝑦)

slide-34

SLIDE 34

TOTAL VARIATION

(Rudin et al. ‘92)

slide-35

SLIDE 35

CROSS-CHANNEL

(Heide et al. ‘13)

slide-36

SLIDE 36

COLLABORATIVE DENOISING

slide-37

SLIDE 37

COLLABORATIVE DENOISING

slide-38

SLIDE 38

LINEAR SYSTEM

Total variation Cross-channel

+ +

Collaborative denoising

𝑏𝑠𝑕𝑛𝑗𝑜𝑦 𝑨 − 𝐵𝑦 2

2 + λ(𝑦)

slide-39

SLIDE 39

IMPORTANCE OF PRIORS

slide-40

SLIDE 40

MORE ON PRIORS…

TV [Rudin et al. ‘92] EPLL [Zoran and Weiss ‘11] Curvelets [Candès and Donoho ‘99]

slide-41

SLIDE 41

MORE ON PRIORS…

slide-42

SLIDE 42

LINEAR SYSTEM

Convex Optimization (Mostly)

𝑏𝑠𝑕𝑛𝑗𝑜𝑦 𝑨 − 𝐵𝑦 2

2 + λ(𝑦)

slide-43

SLIDE 43

CONVERGENCE

slide-44

SLIDE 44

RESULTS

slide-45

SLIDE 45

Bayer Burst iHDR

CAMERA SYSTEMS

slide-46

SLIDE 46

Bayer Burst iHDR

CAMERA SYSTEMS

slide-47

SLIDE 47

Bayer Burst iHDR

CAMERA SYSTEMS A

slide-48

SLIDE 48

slide-49

SLIDE 49

slide-50

SLIDE 50

slide-51

SLIDE 51

slide-52

SLIDE 52

slide-53

SLIDE 53

slide-54

SLIDE 54

Bayer Burst iHDR

CAMERA SYSTEMS

slide-55

SLIDE 55

Bayer Burst iHDR

CAMERA SYSTEMS

slide-56

SLIDE 56

Bayer Burst iHDR

CAMERA SYSTEMS A

slide-57

SLIDE 57

slide-58

SLIDE 58

slide-59

SLIDE 59

slide-60

SLIDE 60

slide-61

SLIDE 61

slide-62

SLIDE 62

slide-63

SLIDE 63

slide-64

SLIDE 64

Bayer Burst iHDR

CAMERA SYSTEMS

slide-65

SLIDE 65

Bayer Burst iHDR

CAMERA SYSTEMS

slide-66

SLIDE 66

Bayer Burst iHDR

CAMERA SYSTEMS A

slide-67

SLIDE 67

slide-68

SLIDE 68

slide-69

SLIDE 69

slide-70

SLIDE 70

slide-71

SLIDE 71

slide-72

SLIDE 72

slide-73

SLIDE 73

PERFORMANCE

0.82 7.45 2.13 Image burst (0.4MPx16) iHDR (13MP) Demosaic (5MP) TIME IN SECONDS

GTX Titan

slide-74

SLIDE 74

PERFORMANCE

16.7 174.4 40.5 Image burst (0.4MPx16) iHDR (13MP) Demosaic (5MP) TIME IN SECONDS

Tegra K1

slide-75

SLIDE 75

MORE ON PRIORS…

slide-76

SLIDE 76

PERFORMANCE (BM3D)

16.7 174.4 40.5 Image burst (0.4MPx16) iHDR (13MP) Demosaic (5MP) TIME IN SECONDS

Tegra K1

slide-77

SLIDE 77

PERFORMANCE (AVERAGING NLM)

6.6 33.2 6.7 Image burst (0.4MPx16) iHDR (13MP) Demosaic (5MP) TIME IN SECONDS

Tegra K1

slide-78

SLIDE 78

slide-79

SLIDE 79

slide-80

SLIDE 80

CONCLUSION

slide-81

SLIDE 81

Bayer Burst iHDR Color Camera Array

+ others such as: deconvolution, super-resolution, JPEG compression and de-blocking.

slide-82

SLIDE 82

Bayer Burst AOHDR Color Camera Array FLEXISP

A

slide-83

SLIDE 83

THANK YOU!