Image Processing Todays Class Image Representations: Matrices - - PowerPoint PPT Presentation

▶
image processing today s class
SMART_READER_LITE
LIVE PREVIEW

Image Processing Todays Class Image Representations: Matrices - - PowerPoint PPT Presentation

Nov 14, 2022 181 likes •961 views

CS6501: Deep Learning for Visual Recognition Image Processing Todays Class Image Representations: Matrices Image Representations: RGB, HSV, etc Image Processing: Brightness Image Filtering: Mean Filter, Median Filter Image Filtering:

slide-1
SLIDE 1

CS6501: Deep Learning for Visual Recognition

Image Processing

slide-2
SLIDE 2

Today’s Class

Image Representations: Matrices Image Representations: RGB, HSV, etc Image Processing: Brightness Image Filtering: Mean Filter, Median Filter Image Filtering: Convolutions Blurring, Sharpening, Computing Gradients Walkthrough Image Processing Primer

slide-3
SLIDE 3
  • Instructor: Vicente Ordóñez
  • Email: vicente@virginia.edu
  • Website: http://vicenteordonez.com/deeplearning/
  • Class Location: Olsson Hall 005
  • Class Times: Monday-Wednesday 3:30pm and 4:45pm
  • Piazza:

https://piazza.com/virginia/spring2020/cs6501003/home

  • Office Hours: Tuesdays 3 to 5pm (Rice 310)

3

About the Course

CS6501-003: Deep Learning for Visual Recognition

slide-4
SLIDE 4

Paola Cascante-Bonilla (pc9za@virginia.edu) Hours: Fridays 2 to 4pm (Rice 442)

4

Teaching Assistants

Ziyan Yang (tw8cb@virginia.edu) Office Hours: Thursdays 3 to 5pm (Rice 442)

slide-5
SLIDE 5

Grading

5

  • Assignments: 400pts (4 assignments)

(100pts + 100pts + 100pts + 100pts)

  • Course Project: 400pts

Groups of up to 3 students (more only if justified)

  • Paper Reading Summaries: 100pts
  • Class Paper Presentation: 100pts (groups of mostly 2 students)
slide-6
SLIDE 6

Also…

  • Assignment 1 released on course website
  • You should have completed the pytorch/jupyter/Google

Colaboratory tutorial and the Numpy / Image Processing tutorial.

slide-7
SLIDE 7

Additionally you will need for your Course Project:

Free credits for students! $50. g2.2xlarge: 4GB ($0.65 / hour) p2.xlarge: 12GB ($0.90 / hour)

slide-8
SLIDE 8

Or maybe, even better:

GTX 1080 Ti (11GB): $700 GTX 1080 (8GB): $500 GTX 1060 (6GB): $150-$250

slide-9
SLIDE 9

I am also working to get you set up on

GTX 1080 Ti (11GB): $700 GTX 1080 (8GB): $500 GTX 1060 (6GB): $150-$250

slide-10
SLIDE 10

What we see What a computer sees

Source: S. Narasimhan

Reminder of what is an image for a computer.

slide-11
SLIDE 11

Images as Functions

z = #(%, ')

slide-12
SLIDE 12

Images as Functions

z = #(%, ')

  • The domain of x and y is [0, img-width) and [0 and img-height)
  • x, and y are discretized into integer values.
slide-13
SLIDE 13

Light

  • What determines the color of a pixel?

Figure from Szeliski

slide-14
SLIDE 14

The Retina

Cross-section of eye Ganglion cell layer Bipolar cell layer Receptor layer Pigmented epithelium Ganglion axons Cross section of retina

[What the Frog's Eye Tells the Frog's Brain]

Cones cone-shaped less sensitive

  • perate in high light

color vision Rods rod-shaped highly sensitive

  • perate at night

gray-scale vision

slide-15
SLIDE 15

Electromagnetic Spectrum

http://www.yorku.ca/eye/photopik.htm

Human Luminance Sensitivity Function

slide-16
SLIDE 16

Basic Image Processing

! "!

" > 1

slide-17
SLIDE 17

Basic Image Processing

! "!

0 < " < 1

slide-18
SLIDE 18

Color Images as Tensors

!ℎ#$$%& ' ℎ%()ℎ* ' +(,*ℎ

slide-19
SLIDE 19

Color Images as Tensors

!ℎ#$$%& ' ℎ%()ℎ* ' +(,*ℎ Channels are usually RGB: Red, Green, and Blue Other color spaces: HSV, HSL, LUV, XYZ, Lab, CMYK, etc

slide-20
SLIDE 20

Color spaces: RGB

0,1,0 0,0,1 1,0,0

Image from: http://en.wikipedia.org/wiki/File:RGB_color_solid_cube.png

Some drawbacks

  • Strongly correlated channels
  • Non-perceptual

Default color space R

(G=0,B=0)

G

(R=0,B=0)

B

(R=0,G=0)

Slide by James Hays

slide-21
SLIDE 21

Color spaces: HSV

Intuitive color space

H

(S=1,V=1)

S

(H=1,V=1)

V

(H=1,S=0)

Slide by James Hays

slide-22
SLIDE 22

Color spaces: L*a*b*

“Perceptually uniform”* color space

L

(a=0,b=0)

a

(L=65,b=0)

b

(L=65,a=0)

Slide by James Hays

slide-23
SLIDE 23

Most information in intensity

Only color shown – constant intensity

Slide by James Hays

slide-24
SLIDE 24

Most information in intensity

Only intensity shown – constant color

Slide by James Hays

slide-25
SLIDE 25

Most information in intensity

Original image

Slide by James Hays

slide-26
SLIDE 26

Image filtering

slide-27
SLIDE 27

Image filtering

Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

slide-28
SLIDE 28

Image filtering

Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

slide-29
SLIDE 29

Image filtering: e.g. Mean Filter

slide-30
SLIDE 30

Image filtering: e.g. Mean Filter

slide-31
SLIDE 31

Image filtering: e.g. Median Filter

Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

slide-32
SLIDE 32

Image filtering: Convolution operator

Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

!(#, %) ' #, % = )

*

)

+

! ,, - .(# − ,, % − -)

slide-33
SLIDE 33

http://www.cs.virginia.edu/~vicente/recognition/animation.gif (filter, kernel)

slide-34
SLIDE 34

Image filtering: Convolution operator e.g. mean filter

Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

!(#, %)

!(#, %) =

1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9

slide-35
SLIDE 35

Image filtering: Convolution operator e.g. mean filter

Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

!(#, %)

!(#, %) =

1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9

slide-36
SLIDE 36

1 1 1 1 1 1 1 1 1

Slide credit: David Lowe (UBC)

] , [ g × ×

Example: box filter

slide-37
SLIDE 37

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

Credit: S. Seitz

] , [ ] , [ ] , [

,

l n k m f l k g n m h

l k

+ + =å

[.,.] h [.,.] f

Image filtering

1 1 1 1 1 1 1 1 1

] , [ g × ×

slide-38
SLIDE 38

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 10 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

[.,.] h [.,.] f

Image filtering

1 1 1 1 1 1 1 1 1

] , [ g × ×

Credit: S. Seitz

] , [ ] , [ ] , [

,

l n k m f l k g n m h

l k

+ + =å

slide-39
SLIDE 39

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 10 20 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

[.,.] h [.,.] f

Image filtering

1 1 1 1 1 1 1 1 1

] , [ g × ×

Credit: S. Seitz

] , [ ] , [ ] , [

,

l n k m f l k g n m h

l k

+ + =å

slide-40
SLIDE 40

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 10 20 30 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

[.,.] h [.,.] f

Image filtering

1 1 1 1 1 1 1 1 1

] , [ g × ×

Credit: S. Seitz

] , [ ] , [ ] , [

,

l n k m f l k g n m h

l k

+ + =å

slide-41
SLIDE 41

10 20 30 30 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

[.,.] h [.,.] f

Image filtering

1 1 1 1 1 1 1 1 1

] , [ g × ×

Credit: S. Seitz

] , [ ] , [ ] , [

,

l n k m f l k g n m h

l k

+ + =å

slide-42
SLIDE 42

10 20 30 30 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

[.,.] h [.,.] f

Image filtering

1 1 1 1 1 1 1 1 1

] , [ g × ×

Credit: S. Seitz

?

] , [ ] , [ ] , [

,

l n k m f l k g n m h

l k

+ + =å

slide-43
SLIDE 43

10 20 30 30 50 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

[.,.] h [.,.] f

Image filtering

1 1 1 1 1 1 1 1 1

] , [ g × ×

Credit: S. Seitz

?

] , [ ] , [ ] , [

,

l n k m f l k g n m h

l k

+ + =å

slide-44
SLIDE 44

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 10 20 30 30 30 20 10 20 40 60 60 60 40 20 30 60 90 90 90 60 30 30 50 80 80 90 60 30 30 50 80 80 90 60 30 20 30 50 50 60 40 20 10 20 30 30 30 30 20 10 10 10 10

[.,.] h [.,.] f

Image filtering

1 1 1 1 1 1 1 1 1

] , [ g × ×

Credit: S. Seitz

] , [ ] , [ ] , [

,

l n k m f l k g n m h

l k

+ + =å

slide-45
SLIDE 45

What does it do?

  • Replaces each pixel with

an average of its neighborhood

  • Achieve smoothing effect

(remove sharp features)

1 1 1 1 1 1 1 1 1

Slide credit: David Lowe (UBC)

] , [ g × ×

Box Filter

slide-46
SLIDE 46

Image filtering: e.g. Mean Filter

slide-47
SLIDE 47

Image filtering: Convolution operator Important filter: gaussian filter (gaussian blur)

Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

!(#, %)

!(#, %) =

1/16 1/8 1/16 1/8 1/4 1/8 1/16 1/8 1/16

slide-48
SLIDE 48
  • Weight contributions of neighboring pixels by nearness

0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022 0.013 0.003

5 x 5, s = 1

Slide credit: Christopher Rasmussen

Important filter: Gaussian

slide-49
SLIDE 49

Image filtering: Convolution operator e.g. gaussian filter (gaussian blur)

Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

slide-50
SLIDE 50

Practical matters

  • What about near the edge?
  • the filter window falls off the edge of the image
  • need to extrapolate
  • methods:
  • clip filter (black)
  • wrap around
  • copy edge
  • reflect across edge

Source: S. Marschner

slide-51
SLIDE 51

Practice with linear filters

1 Original

?

Source: D. Lowe

slide-52
SLIDE 52

Practice with linear filters

1 Original Filtered (no change)

Source: D. Lowe

slide-53
SLIDE 53

Practice with linear filters

1 Original

?

Source: D. Lowe

slide-54
SLIDE 54

Practice with linear filters

1 Original Shifted left By 1 pixel

Source: D. Lowe

slide-55
SLIDE 55

Practice with linear filters

Original 1 1 1 1 1 1 1 1 1 2

  • ?

(Note that filter sums to 1)

Source: D. Lowe

slide-56
SLIDE 56

Practice with linear filters

Original 1 1 1 1 1 1 1 1 1 2

  • Sharpening filter
  • Accentuates differences

with local average

Source: D. Lowe

slide-57
SLIDE 57

Sharpening

Source: D. Lowe

slide-58
SLIDE 58

Key properties of linear filters

Linearity:

imfilter(I, f1 + f2) = imfilter(I,f1) + imfilter(I,f2)

Shift invariance: same behavior regardless of pixel location

imfilter(I,shift(f)) = shift(imfilter(I,f))

Any linear, shift-invariant operator can be represented as a convolution

Source: S. Lazebnik

slide-59
SLIDE 59

– Enhance images

  • Denoise, resize, increase contrast, etc.

– Extract information from images

  • Texture, edges, distinctive points, etc.

– Detect patterns

  • Template matching

– Deep Convolutional Networks

Image filtering: Convolution operator

slide-60
SLIDE 60

Image filtering: Convolution operator Important Filter: Sobel operator

Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

!(#, %)

!(#, %) =

1

  • 1

2

  • 2

1

  • 1
slide-61
SLIDE 61

Other filters

  • 1

1

  • 2

2

  • 1

1 Vertical Edge (absolute value)

Sobel

Slide by James Hays

slide-62
SLIDE 62

Other filters

  • 1
  • 2
  • 1

1 2 1 Horizontal Edge (absolute value)

Sobel

Slide by James Hays

slide-63
SLIDE 63

Sobel operators are equivalent to 2D partial derivatives of the image

  • Vertical sobel operator – Partial derivative in X (width)
  • Horizontal sobel operator – Partial derivative in Y (height)
  • Can compute magnitude and phase at each location.
  • Useful for detecting edges
slide-64
SLIDE 64

https://en.wikipedia.org/wiki/Sobel_operator

slide-65
SLIDE 65

Sobel filters are (approximate) partial derivatives

  • f the image

!"($, &) !$ = lim

,→.

" $ + ℎ, & − "($, &) ℎ "($, &) be your input image, then the partial derivative is: Let !"($, &) !$ = lim

,→.

" $ + ℎ, & − "($ − ℎ, &) 2ℎ Also:

slide-66
SLIDE 66

But digital images are not continuous, they are discrete

Δ"#[%, '] = #[% + 1, '] − #[%, '] #[%, '] be your input image, then the partial derivative is: Let Δ"#[%, '] = #[% + 1, '] − #[% − 1, '] Also:

slide-67
SLIDE 67

But digital images are not continuous, they are discrete

Δ"#[%, '] = #[% + 1, '] − #[%, '] #[%, '] be your input image, then the partial derivative is: Let Δ"#[%, '] = #[% + 1, '] − #[% − 1, '] Also:

  • 1

1

  • 1

1

k(x, y) = k(x, y) =

slide-68
SLIDE 68

Sobel Operators Smooth in Y and then Differentiate in X

1

  • 1

k(x, y) =

1 2 1

* =

1

  • 1

2

  • 2

1

  • 1

Similarly to differentiate in Y

slide-69
SLIDE 69

Image Features

slide-70
SLIDE 70

What are Image Features?

Image 0.22 0.30 0.13 0.24 0.31 0.15 0.35 0.48 Color histogram? Maximum color on sub-areas of the image? Any statistics on the input image? The output of some image processing on the input image?

slide-71
SLIDE 71

Why are they useful?

Image

0.22 0.30 0.13 0.24 0.31 0.15 0.35 0.48

Machine Learning Model Predictions As inputs to a machine learning model

slide-72
SLIDE 72

Why are they useful?

0.22 0.30 0.13 0.24 0.31 0.15 0.35 0.48

To compare images (i.e. retrieve similar images)

0.24 0.34 0.23 0.27 0.63 0.15 0.25 0.48

Distance function (e.g. Euclidean distance)

slide-73
SLIDE 73

Image Features: Color

Photo by: marielito

slide by Tamara L. Berg

slide-74
SLIDE 74

Image Features: Color

slide-75
SLIDE 75

Color often not a powerful feature

However, these are all images of people but the colors in each image are very different.

slide-76
SLIDE 76

Suggested Reading

  • [What the Frog's Eye Tells the Frog's Brain]

[Csurka's Bag of Keypoints. ECCV 2004]

slide-77
SLIDE 77

Questions?

77