Image Operator Learning and Applications Igor S. Montagner Nina S. - - PowerPoint PPT Presentation

▶
image operator learning
SMART_READER_LITE
LIVE PREVIEW

Image Operator Learning and Applications Igor S. Montagner Nina S. - - PowerPoint PPT Presentation

May 04, 2023 586 likes •2.3k views

SIBGRAPI 2016 TUTORIAL Image Operator Learning and Applications Igor S. Montagner Nina S. T. Hirata Roberto Hirata Jr. Department of Computer Science Institute of Mathematics and Statistics University of So Paulo (USP) October / 2016

slide-1
SLIDE 1

SIBGRAPI 2016 – TUTORIAL

Image Operator Learning

and Applications

Igor S. Montagner Nina S. T. Hirata Roberto Hirata Jr.

Department of Computer Science Institute of Mathematics and Statistics

University of São Paulo (USP)

October / 2016

slide-2
SLIDE 2

Observed input Expected output

Given pairs of images as the ones

  • n the left...

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 2

slide-3
SLIDE 3

Observed input Expected output

Ψ =? Given pairs of images as the ones

  • n the left...

Question:

– how to design an image

  • perator Ψ that transforms

the input image to the expected output image?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 3

slide-4
SLIDE 4

What about this transformation ?

Ψ = ? = ⇒

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 4

slide-5
SLIDE 5

And this one?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 5

slide-6
SLIDE 6

This one?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 6

slide-7
SLIDE 7

Or this one?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 7

slide-8
SLIDE 8

Historical overview

1960s: Mathematical morphology was introduced by G. Matheron and J. Serra 1980s: Book by J. Serra (1982) Nonlinear filters (median, order statistic, stack), morphological filters 1990s: E. R. Dougherty - Design of optimal morphological filters 1994: Junior Barrera - Vision Lab created at IME/USP We started working on designing morphological operators from training images at that time !!

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 8

slide-9
SLIDE 9

Today (almost 25 years since then)

Huge advances in image processing, computer vision, machine learning, hardware and computing technologies High-level × low-level processing – Deep learning (high level image understanding) – Image operator learning (low level processing) The second is a building block for the first. Thus, very important.

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 9

slide-10
SLIDE 10

Plan for this tutorial

  • 1. Concepts and fundamentals
  • 2. Optimal operators
  • 3. Learning image operators
  • 4. Practice
  • 5. Application examples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 10

slide-11
SLIDE 11

Plan for this tutorial

  • 1. Concepts and fundamentals

– you will know what an image operator is, and important properties that define a class of operators adequate for the type of processing we are interested on

  • 2. Optimal operators
  • 3. Learning image operators
  • 4. Practice
  • 5. Application examples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 11

slide-12
SLIDE 12

Plan for this tutorial

  • 1. Concepts and fundamentals

– you will know what an image operator is, and important properties that define a class of operators adequate for the type of processing we are interested on

  • 2. Optimal operators

– you will know how to characterize what a good image

  • perator is
  • 3. Learning image operators
  • 4. Practice
  • 5. Application examples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 12

slide-13
SLIDE 13

Plan for this tutorial

  • 1. Concepts and fundamentals

– you will know what an image operator is, and important properties that define a class of operators adequate for the type of processing we are interested on

  • 2. Optimal operators

– you will know how to characterize what a good image

  • perator is
  • 3. Learning image operators

– you will know how to learn image operators from training data

  • 4. Practice
  • 5. Application examples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 13

slide-14
SLIDE 14

Plan for this tutorial

  • 1. Concepts and fundamentals

– you will know what an image operator is, and important properties that define a class of operators adequate for the type of processing we are interested on

  • 2. Optimal operators

– you will know how to characterize what a good image

  • perator is
  • 3. Learning image operators

– you will know how to learn image operators from training data

  • 4. Practice

– you will have the opportunity to do exercises and run code

  • 5. Application examples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 14

slide-15
SLIDE 15

Plan for this tutorial

  • 1. Concepts and fundamentals

– you will know what an image operator is, and important properties that define a class of operators adequate for the type of processing we are interested on

  • 2. Optimal operators

– you will know how to characterize what a good image

  • perator is
  • 3. Learning image operators

– you will know how to learn image operators from training data

  • 4. Practice

– you will have the opportunity to do exercises and run code

  • 5. Application examples

– you will see some application examples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 15

slide-16
SLIDE 16

Fundamentals on image

  • perators

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 16

slide-17
SLIDE 17

How to model image transformations ?

Image domain E (subset of Z2) Image f : E → K (K is the set of gray levels) KE set of all images defined on E with gray-levels in K Image operator Ψ : KE → KE [Ψ(f )](p) value of the processed image at p

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 17

slide-18
SLIDE 18

Image operators (mathematical morphology view)

Morphological operators are built by composing basic elementary operators. Examples of basic operators – erosion – dilation – hit-miss, ... Structuring elements: parameter of the operators Simple shapes such as squares, discs, line segments, to locally probe an image and extract geometric structural information Fundamentals: We will present concepts restricted to binary images, but they hold for gray-scale images too

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 18

slide-19
SLIDE 19

Binary image operators = set operators

Binary image f : E → {0, 1} Binary image = set p ∈ Sf ⇐ ⇒ f (p) = 1 Operations on images = operations on sets Set relations and operations: ⊆, ∪, ∩, ·c Set translation: Ap = {a + p : a ∈ A} Set reflection: ˇ A = {−a : a ∈ A} Set interval: [A, B] = {X : A ⊆ X ⊆ B}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 19

slide-20
SLIDE 20

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 20

slide-21
SLIDE 21

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

How does the erosion work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 21

slide-22
SLIDE 22

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 22

slide-23
SLIDE 23

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 23

slide-24
SLIDE 24

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 24

slide-25
SLIDE 25

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 25

slide-26
SLIDE 26

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 26

slide-27
SLIDE 27

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 27

slide-28
SLIDE 28

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 28

slide-29
SLIDE 29

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 29

slide-30
SLIDE 30

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 30

slide-31
SLIDE 31

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 31

slide-32
SLIDE 32

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 32

slide-33
SLIDE 33

Basic operator: dilation

δB(S) = {p ∈ E : ˇ Bp ∩ S = ∅}

S δB(S)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 33

slide-34
SLIDE 34

Basic operator: dilation

δB(S) = {p ∈ E : ˇ Bp ∩ S = ∅}

How does the dilation work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 34

slide-35
SLIDE 35

Basic operator: dilation

δB(S) = {p ∈ E : ˇ Bp ∩ S = ∅}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 35

slide-36
SLIDE 36

Basic operator: dilation

δB(S) = {p ∈ E : ˇ Bp ∩ S = ∅}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 36

slide-37
SLIDE 37

Basic operator: dilation

δB(S) = {p ∈ E : ˇ Bp ∩ S = ∅}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 37

slide-38
SLIDE 38

Basic operator: dilation

δB(S) = {p ∈ E : ˇ Bp ∩ S = ∅}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 38

slide-39
SLIDE 39

Basic operator: dilation

δB(S) = {p ∈ E : ˇ Bp ∩ S = ∅}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 39

slide-40
SLIDE 40

Basic operator: dilation

δB(S) = {p ∈ E : ˇ Bp ∩ S = ∅}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 40

slide-41
SLIDE 41

Basic operator: dilation

δB(S) = {p ∈ E : ˇ Bp ∩ S = ∅}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 41

slide-42
SLIDE 42

Basic operator: dilation

δB(S) = {p ∈ E : ˇ Bp ∩ S = ∅}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 42

slide-43
SLIDE 43

Basic operator: dilation

δB(S) = {p ∈ E : ˇ Bp ∩ S = ∅}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 43

slide-44
SLIDE 44

Basic operator: dilation

δB(S) = {p ∈ E : ˇ Bp ∩ S = ∅}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 44

slide-45
SLIDE 45

Basic operator: dilation

δB(S) = {p ∈ E : ˇ Bp ∩ S = ∅}

S δB(S)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 45

slide-46
SLIDE 46

Basic operator: erosion and dilation

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 46

slide-47
SLIDE 47

Another basic operator: hit-miss H(A,Bc)(S) = {p ∈ E : Ap ⊆ S and Bc

p ⊆ Sc} = εA(S) ∩ εBc(Sc) A Bc

places where A hits S places where Bc hits the background (or, equiva- lently, misses S) places detected by the hit- miss operator

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 47

slide-48
SLIDE 48

Short break for an exercise

Exercise 1

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 48

slide-49
SLIDE 49

Solution to exercise 1 – structuring element A

A Bc In red, positions x at which Ax fits in the foreground

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 49

slide-50
SLIDE 50

Solution to exercise 1 – structuring element Bc

A Bc In red, positions x at which Bc

x fits in the background

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 50

slide-51
SLIDE 51

Solution to exercise 1 – Intersection

A Bc Result: the holes

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 51

slide-52
SLIDE 52

Many useful operators can be built by composing these and

  • ther simple operators

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 52

slide-53
SLIDE 53

Example of an operator: Contour detection

f − εB(f )

S εB(S) S − εB(S)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 53

slide-54
SLIDE 54

Example of an operator: Contour detection

f − εB(f )

S εB(S) S − εB(S)

Which properties of this operator are interesting ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 54

slide-55
SLIDE 55

Example of an operator: Contour detection

f − εB(f )

S εB(S) S − εB(S)

Which properties of this operator are interesting ?

  • Translation invariance
  • Local definition

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 55

slide-56
SLIDE 56

Translation invariance

[Ψ(f )]p = Ψ(fp)

Ψ(S)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 56

slide-57
SLIDE 57

Translation invariance

[Ψ(f )]p = Ψ(fp)

Ψ(S) [Ψ(S)]p

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 57

slide-58
SLIDE 58

Translation invariance

[Ψ(f )]p = Ψ(fp)

Ψ(S) [Ψ(S)]p Sp

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 58

slide-59
SLIDE 59

Translation invariance

[Ψ(f )]p = Ψ(fp)

Ψ(S) [Ψ(S)]p Sp Ψ(Sp)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 59

slide-60
SLIDE 60

Local definition

Ψ is locally defined if there is a window W such that:

p ∈ Ψ(S) ⇐ ⇒ p ∈ Ψ(S ∩ W ′)

for every p ∈ E, S ∈ KE, and W ′ ⊇ W

The red pixel is a contour point

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 60

slide-61
SLIDE 61

Local definition

Ψ is locally defined if there is a window W such that:

p ∈ Ψ(S) ⇐ ⇒ p ∈ Ψ(S ∩ W ′)

for every p ∈ E, S ∈ KE, and W ′ ⊇ W

The red pixel is a contour point

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 61

slide-62
SLIDE 62

Local definition

Ψ is locally defined if there is a window W such that:

p ∈ Ψ(S) ⇐ ⇒ p ∈ Ψ(S ∩ W ′)

for every p ∈ E, S ∈ KE, and W ′ ⊇ W

The red pixel is a contour point

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 62

slide-63
SLIDE 63

Local definition

Ψ is locally defined if there is a window W such that:

p ∈ Ψ(S) ⇐ ⇒ p ∈ Ψ(S ∩ W ′)

for every p ∈ E, S ∈ KE, and W ′ ⊇ W

The red pixel is a contour point

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 63

slide-64
SLIDE 64

Window operator or W-operator

W-operator: translation invariance + local definition There is a local function ψ that uniquely characterizes Ψ

S Ψ(S) z z

Ψ(S)(z) = ψ

  • SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.)

64

slide-65
SLIDE 65

Back to contour detection

Contour detection operator: S − εB(S) – translation-invariant – locally defined Local function: To decide if a pixel is a(n internal) contour point or not, it suffices to check if it is in the foreground and if there is at least one pixel adjacent to it in the background. Considering 4-adjacency, a cross- window is sufficient

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 65

slide-66
SLIDE 66

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 66

slide-67
SLIDE 67

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 00000 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 67

slide-68
SLIDE 68

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 00000 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 68

slide-69
SLIDE 69

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 00011 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 69

slide-70
SLIDE 70

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 00011 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 70

slide-71
SLIDE 71

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 00101 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 71

slide-72
SLIDE 72

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 00101 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 72

slide-73
SLIDE 73

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 01001 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 73

slide-74
SLIDE 74

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 01001 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 74

slide-75
SLIDE 75

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 00000 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 75

slide-76
SLIDE 76

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 00000 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 76

slide-77
SLIDE 77

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 00011 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 77

slide-78
SLIDE 78

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 00011 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 78

slide-79
SLIDE 79

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 00111 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 79

slide-80
SLIDE 80

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 00111 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 80

slide-81
SLIDE 81

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 11111 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 81

slide-82
SLIDE 82

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 11111 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 82

slide-83
SLIDE 83

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 01101 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 83

slide-84
SLIDE 84

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 01101 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 84

slide-85
SLIDE 85

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 01001 X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 85

slide-86
SLIDE 86

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output 01001

and so on

X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 86

slide-87
SLIDE 87

Local function of the contour detection operator

x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 window array

Input Output X ψ(X) 00000 00001 00010 00011 00100 1 00101 1 00110 1 00111 1 · · · · · · 01001 · · · · · · 01101 1 · · · · · · 11011 · · · · · · 11110 1 11111

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 87

slide-88
SLIDE 88

Short break for another exercise

Exercise 2

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 88

slide-89
SLIDE 89

Solution to exercise 2 – structuring element A

A Bc

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 89

slide-90
SLIDE 90

Solution to exercise 2 – structuring element Bc

A Bc

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 90

slide-91
SLIDE 91

Solution to exercise 2 – Intersection

A Bc

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 91

slide-92
SLIDE 92

Representation of an operator How to represent an image operator ?

If Ψ is a W-operator it suffices to know ψ : {0, 1}W → {0, 1}. Kernel of Ψ

K(Ψ) = {X ⊆ W : ψ(X) = 1}

  • What is the importance of the kernel ?
  • Do we need to explicitly represent the kernel ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 92

slide-93
SLIDE 93

Representation of an operator How to represent an image operator ?

If Ψ is a W-operator it suffices to know ψ : {0, 1}W → {0, 1}. Kernel of Ψ

K(Ψ) = {X ⊆ W : ψ(X) = 1}

  • What is the importance of the kernel ?
  • Do we need to explicitly represent the kernel ?

Kernels are related to a canonical representation of W-operators

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 93

slide-94
SLIDE 94

Representation of operators

Hit-miss in an important piece in the representation of W-operators A hit-miss operator is equivalent to an interval

X is detected by hit-miss operator H(A,Bc) ⇐ ⇒ X ∈ [A, B]

W A Bc B

The window images detected by H(A,Bc) or in [A, B] are the same:

(all templates that correspond top contour points)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 94

slide-95
SLIDE 95

Decomposition theorem

Example: intervals for the contour detection operator , , , , , , ,

{ }

  • Interval operator:

Λ(A,B)(X) = 1 ⇐ ⇒ X ∈ [A, B]

Basis of Ψ: B(Ψ) = maximal intervals in K(Ψ) Minimal decomposition theorem

ψ(X) = max

[A,B]∈B(Ψ) Λ(A,B)(X)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 95

slide-96
SLIDE 96

What would be the basis for the extreme point case ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 96

slide-97
SLIDE 97

What would be the basis for the extreme point case ?

left endpoint right endpoint bottom endpoint top endpoint

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 97

slide-98
SLIDE 98

Summary of the fundamentals

You acquired knowledge on – image operators – W-operators, a broad class of image operators – characterization by local functions – basis representation: supremum of interval operators – geometrical interpretation We should also mention: – binary case: equivalent to Boolean functions – Similar results hold for gray-scale image operators

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 98

slide-99
SLIDE 99

Learning image operators

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 99

slide-100
SLIDE 100

Goal of image operator learning

Given observed images f and respective expected transformation g, we would like to find Ψ such that Ψ(f ) is a good approximation of g MAE (mean absolute error) MAEΨ = E

  • |[Ψ(f )](p) − g(p)|
  • Empirical MAE

Err = 1 |E|

  • p∈E
  • [Ψ(f )](p) − g(p)
  • Average of pixel-wise absolute difference

Binary images: pixel error rate

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 100

slide-101
SLIDE 101

Optimal W-operators

Window: W Input-output pairs of images: (f , g) Observations: (X, y) X is a window image y is the corresponding output value Optimal operator: (in the case of binary operators) ψ(X) =      1, if P(1|X) > P(0|X), 0, if P(0|X) > P(1|X), 0 or 1 if P(0|X) = P(1|X) = 0.5 Probabilities P(y|X) can be estimated from training images

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 101

slide-102
SLIDE 102

Counting occurrences of (X, y)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 102

slide-103
SLIDE 103

Counting occurrences of (X, y) ( 000 , 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 103

slide-104
SLIDE 104

Counting occurrences of (X, y) ( 001 1 , 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 104

slide-105
SLIDE 105

Counting occurrences of (X, y) ( 010 1 , 1)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 105

slide-106
SLIDE 106

Counting occurrences of (X, y) ( 100 1 , 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 106

slide-107
SLIDE 107

Example

X ψ(X) 27 9 3 3 7 7 1 3 8 11 0/1 1 1 1 1

Optimal decision

ψ(X) =      0, if p(0|X) > p(1|X), 1, if p(0|X) < p(1|X), ×, if p(0|X) = p(1|X).

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 107

slide-108
SLIDE 108

Short break for a new exercise

Exercise 3

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 108

slide-109
SLIDE 109

Solution to exercise 3

2 8 2 10 2 44 36 10 9 11 2 12 2

X 1

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 109

slide-110
SLIDE 110

Image operator design

in the framework of

Machine Learning

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 110

slide-111
SLIDE 111

Machine learning – Supervised classification

http://www.nltk.org/book/ch06.html

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 111

slide-112
SLIDE 112

Learning the local function

S Ψ(S) z z

Ψ(S)(z) = ψ

  • The local function can be understood as a classifier

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 112

slide-113
SLIDE 113

Training samples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 113

slide-114
SLIDE 114

Training samples

Positive example

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 114

slide-115
SLIDE 115

Training samples

Positive example Negative example

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 115

slide-116
SLIDE 116

Feature extraction (to form the training set)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 116

slide-117
SLIDE 117

Feature extraction (to form the training set) ( 000 , 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 117

slide-118
SLIDE 118

Feature extraction (to form the training set) ( 001 1 , 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 118

slide-119
SLIDE 119

Feature extraction (to form the training set) ( 010 1 , 1)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 119

slide-120
SLIDE 120

Feature extraction (to form the training set) ( 100 1 , 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 120

slide-121
SLIDE 121

Overview

Training W−operator S I Window W

  • W defines the feature space
  • Training data is extracted from input-output images

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 121

slide-122
SLIDE 122

The problem is not solved yet ...

Challenge: tradeoff between window size and generalization error

0.01 0.012 0.014 0.016 0.018 0.02 0.022 0.024 0.026 0.028 0.03 20 40 60 80 100 120 140 MAE Window size 1 training image 5 training images

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 122

slide-123
SLIDE 123

Two-level training

1 2 3 W W W same target pixel distinct neighborhood three results pattern for the second level classifier target pixel

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 123

slide-124
SLIDE 124

Input

= ⇒

Results of 7

  • perators

= ⇒

Superimposed results

(darker = less votes)

⇓

Combination result

(better than the individual ones)

slide-125
SLIDE 125
slide-126
SLIDE 126

Why two-level training scheme is good?

Windows in the first level training

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 126

slide-127
SLIDE 127

Why two-level training scheme is good?

Windows in the first level training Their union is a larger window

Two-level training (indirectly) considers a larger neighborhood, without increasing window size.

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 127

slide-128
SLIDE 128

Application examples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 128

slide-129
SLIDE 129

Application examples

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 129

slide-130
SLIDE 130

Application examples

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 130

slide-131
SLIDE 131

Application examples

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 131

slide-132
SLIDE 132

Application examples

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 132

slide-133
SLIDE 133

Application examples

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 133

slide-134
SLIDE 134

Application examples

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 134

slide-135
SLIDE 135

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 135

slide-136
SLIDE 136

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 136

slide-137
SLIDE 137

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 137

slide-138
SLIDE 138

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 138

slide-139
SLIDE 139

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 139

slide-140
SLIDE 140

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 140

slide-141
SLIDE 141

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 141

slide-142
SLIDE 142

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 142

slide-143
SLIDE 143

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 143

slide-144
SLIDE 144

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 144

slide-145
SLIDE 145

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 145

slide-146
SLIDE 146

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 146

slide-147
SLIDE 147

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 147

slide-148
SLIDE 148

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 148

slide-149
SLIDE 149

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 149

slide-150
SLIDE 150

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 150

slide-151
SLIDE 151

Handwritten musical scores

Staff line removal:

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 151

slide-152
SLIDE 152

Synthetically generated musical scores

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 152

slide-153
SLIDE 153

PRACTICAL SESSION

TRIOSLib

Laboratory session exercises 1 to 4 http://bit.ly/2dtL5en

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 153

slide-154
SLIDE 154

TRIOSlib

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 154

slide-155
SLIDE 155

TRIOSlib: research library containing implementations of many WOperator learning techniques. – Integrates with Numpy, Scipy, scikit-learn. – Extensible design: new techniques implemented without changing the library. – Error measuring, persistence on disk, parallel implementations, ... – Used in many published papers. Available at https://sourceforge.net/projects/trioslib and installable using pip and conda.

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 155

slide-156
SLIDE 156

Important classes: – trios.Imageset: list of triples (input,output,mask). mask=None means that every pixel is processed. – trios.WOperator: executes learning process; – trios.FeatureExtractor:

– raw pixels (RAWFeatureExtractor and RAWBitFeatureExtractor) – two-level patterns (CombinationPattern) – aperture

– trios.Classifier:

– Boolean function minimization (ISI) – models from scikit-learn (SKClassifier)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 156

slide-157
SLIDE 157

Recent advances

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 157

slide-158
SLIDE 158

Window determination

Empirical approach: Based on user experience; trial and error Feature selection based methods

Window points ∼ features / Window design ∼ feature selection

– Interaction information: C. S. Santos, N. S. Hirata, and R. Hirata Jr., “An information

theory framework for two-stage binary image operator design,” Pattern Recognition Letters, 2010

– RELIEF (feature selection algorithm): I. Montagner, R. Hirata Jr., and N. S. T.

Hirata, Ä machine learning based method for staff removal,” ICPR 2014

Measures to rank windows: discriminative power of windows – Entropy based ranking: M. Dornelles and N. S. T. Hirata, “Selection of Windows for

W-Operator Combination from Entropy Based Ranking,” SIBGRAPI 2015

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 158

slide-159
SLIDE 159

Operator selection for combination

– Given a domain window W defined as a m × n = N rectangle, there are 2N possible subwindows. – Train a very strongly regularized sparse combination using all possible windows. – Fast algorithms for linear sparse models(LASSO, L1-SVM). Iterative solution is possible by checking optimality conditions and random sampling of the operator space.

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 159

slide-160
SLIDE 160

Processing gray scale images

n = |W| There are: – 22n binary W-operators – kkn gray-level W-operators Computational cost (memory, processing) and statistical precision are challenging issues Some possible approaches: – constrain operator space: two-level, stack filters, aperture – reduce window image variability Some details next

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 160

slide-161
SLIDE 161

Gray-scale from binary

Threshold decomposition

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 161

slide-162
SLIDE 162

Example of stack filter

Gray-scale median can be computed as the sum of the medians of the cross-sections

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 162

slide-163
SLIDE 163

Test image and result

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 163

slide-164
SLIDE 164

Aperture Operators

Class of operators to transform grayscale images. Number of possible binary operators: 22|W|. Number of possible grayscale operators: KK|W|. Now we have to restrict W and K.

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 164

slide-165
SLIDE 165

Aperture Operators

Besides translation invariance in the domain

  • x

t h(x)

W

−t

x

  • h (x)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 165

slide-166
SLIDE 166

Aperture Operators

we add translation invariance in grayscale

−t

W

u

−z

x h (x)

  • u−z | K

x

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 166

slide-167
SLIDE 167

Aperture Operators

Translation invariance in the domain and in the range, locally defined by a window W and a range K = {−k, . . . , 0, . . . , k} Ψ(f )(t) = ζO(ut) + ψ(ut

−ζI(ut)|K)

where ζO and ζI are functions defined from KW to N.

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 167

slide-168
SLIDE 168

Kernel approximation

Main issue in training is generalization. Kernel SVMs offer good generalization, but require building a Gram matrix K ∈ RM×M. In WOperator learning, M >> 100, 000. Nyström method: – approximate K using a small set of m examples, m << M. – works for any kernel; – exact solution for m = M. More details: Friday 14:00-15:45 at TS10!

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 168

slide-169
SLIDE 169

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 169

slide-170
SLIDE 170

Trios - training image operators from samples

Available: https://sourceforge.net/projects/trioslib

Igor S. Montagner igordsm@ime.usp.br Nina S. T. Hirata nina@ime.usp.br Roberto Hirata Jr. hirata@ime.usp.br

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 170