Photons to Pixels The Imaging Pipeline Mark Buckler The Vision - - PowerPoint PPT Presentation

▶
photons to pixels
SMART_READER_LITE
LIVE PREVIEW

Photons to Pixels The Imaging Pipeline Mark Buckler The Vision - - PowerPoint PPT Presentation

Sep 02, 2023 392 likes •623 views

Photons to Pixels The Imaging Pipeline Mark Buckler The Vision Pipeline Photons Image Sensor Raw Data Image Signal Processor (ISP) Human Readable Image (JPEG, PNG, etc) GPU/CPU/ASIC Vision task result (bounding boxes, segmentation, etc)

slide-1
SLIDE 1

Photons to Pixels

The Imaging Pipeline Mark Buckler

slide-2
SLIDE 2

The Vision Pipeline

Image Sensor Image Signal Processor (ISP) GPU/CPU/ASIC

Photons Raw Data Human Readable Image (JPEG, PNG, etc) Vision task result (bounding boxes, segmentation, etc)

slide-3
SLIDE 3

The Image Sensor (excluding optics)

https://arxiv.org/pdf/1705.04352.pdf

slide-4
SLIDE 4

The Image Sensor

https://www.globalspec.com/learnmore/video_imaging_equipment/video_cameras_accessories/cmos_cameras

slide-5
SLIDE 5

The Bayer Pattern

https://en.wikipedia.org/wiki/Image_sensor

slide-6
SLIDE 6

The ISP

https://arxiv.org/pdf/1705.04352.pdf

slide-7
SLIDE 7

The ISP: Visual effect

ISP

Raw Data JPEG

https://arxiv.org/pdf/1705.04352.pdf

slide-8
SLIDE 8

The ISP: Demosaicing

http://www.ok.sc.e.titech.ac.jp/~mtanaka/research.html

slide-9
SLIDE 9
  • The most computationally expensive step in the ISP by far
  • Especially important for low light (shot noise)
  • Must balance need for smoothing aberrations with need for sharp edges

The ISP: Denoising

https://en.wikipedia.org/wiki/Total_variation_denoising

slide-10
SLIDE 10

The ISP: Color Mapping & White Balancing

https://www.cambridgeincolour.com/tutorials/white-balance.htm

slide-11
SLIDE 11

The ISP: Gamut Mapping

  • Mapping between color systems with more or less representable range

Showing a CIE 1931 chromaticity diagram of:

  • Human eye gamut (grey + colored)
  • Computer monitor gamut (colored)

Each image sensor has its own gamut

https://en.wikipedia.org/wiki/Gamut

slide-12
SLIDE 12

The ISP: Tone Mapping (global/gamma compression)

  • PDF of natural light is log-normal.

Terrible for linear quantization!

  • Tone mapping computes log,

resulting in normal distribution

https://en.wikipedia.org/wiki/Gamma_correction

slide-13
SLIDE 13

The ISP: Tone Mapping (global/gamma compression)

http://www.markbuckler.com/pdf/iccv-2017-slides.pdf

slide-14
SLIDE 14

The ISP: Tone Mapping (local/HDR)

https://rawpedia.rawtherapee.com/File:Rt407-ba-tonemapping-hdr-cropped.jpg

slide-15
SLIDE 15

The ISP: Compression (JPEG)

https://www.cl.cam.ac.uk/teaching/1011/R08/jpeg/acs10-jpeg.pdf

slide-16
SLIDE 16

JPEG: Color space conversion

https://www.cl.cam.ac.uk/teaching/1011/R08/jpeg/acs10-jpeg.pdf

slide-17
SLIDE 17

JPEG: The Discrete Cosine Transform

https://www.cl.cam.ac.uk/teaching/1011/R08/jpeg/acs10-jpeg.pdf

slide-18
SLIDE 18

JPEG: The Discrete Cosine Transform

https://www.cl.cam.ac.uk/teaching/1011/R08/jpeg/acs10-jpeg.pdf

slide-19
SLIDE 19

JPEG: Quantize

http://www.robertstocker.co.uk/jpeg/jpeg_new_10.htm

slide-20
SLIDE 20

JPEG: Quantize

Notice the blurring around the characters in this image? That blurring is directly because of this quantization step

http://www.robertstocker.co.uk/jpeg/jpeg_new_10.htm

slide-21
SLIDE 21

JPEG: Encode

Zig-zag run length encoding

http://www.robertstocker.co.uk/jpeg/jpeg_new_10.htm

slide-22
SLIDE 22

The End!

ISP