ADVANCED TOPICS ON VIDEO PROCESSING Image Spatial Processing Image - - PowerPoint PPT Presentation

ADVANCED TOPICS ON VIDEO PROCESSING

Image Spatial Processing Image Spatial Processing

FILTERING EXAMPLES FILTERING EXAMPLES

FOURIER INTERPRETATION FOURIER INTERPRETATION

FILTERING EXAMPLES FILTERING EXAMPLES

FOURIER INTERPRETATION FOURIER INTERPRETATION

FILTERING EXAMPLES FILTERING EXAMPLES

FILTERING EXAMPLES FILTERING EXAMPLES

FOURIER INTERPRETATION FOURIER INTERPRETATION

FILTERING EXAMPLES FILTERING EXAMPLES

FOURIER INTERPRETATION FOURIER INTERPRETATION

LINEAR AND NON LINEAR OPERATIONS LINEAR AND NON LINEAR OPERATIONS

Median Filter: (6 8 9 9 10 11 12 13 15) = 10 Average of nearest neighbours: Minimum = 6; Maximum: 15 Median Filter: (6, 8, 9, 9,10, 11, 12, 13, 15) 10 Average of nearest neighbours:

LOW PASS GAUSSIAN FILTER LOW PASS GAUSSIAN FILTER

MEDIAN FILTERING MEDIAN FILTERING

HI PASS FILTERING FOR HIGH FREQUENCIES

EXAMPLE EXAMPLE

1D DERIVATIVES 1D DERIVATIVES

GRADIENT METHODS GRADIENT METHODS

 1D example

Si

 1D example

ye s

Yes Yes

>Thres-

No No

Thres hold?

No edge No edge

2D CASE 2D CASE

Th fi d i i i b i d b h di

• The first derivative is substituted by the gradient

     

   f x y f x y x i f x y y i

x y

, ,  ,     

• Omnidirectional detector

B d |ƒ( )| i t i b h i

 Based on|ƒ(x,y)|: isotropic behaviour

• Directional detector

 Based on an oriented derivative:

 ex.: a possible horizontal edge detector is |y|

2D APPROACH 2D APPROACH

EDGE THINNING EDGE THINNING

1) if f has a local horizontal max but not a vertical one in ( ) then that point is an edge point if (x0,y0), then, that point is an edge point if

2( value) f f K K typical    

 

0, 0

( ) ( , )

2( value)

x y x y

K K typical x y   

2) if f has a local vertical max but not a horizontal one in (x0,y0), then, that point is an edge point if

2(typical value) f f K K y x      

0, 0 0,

( ) ( )

• x y

x y

y x  

DIRECTIONAL CASE DIRECTIONAL CASE

.

EXAMPLE: ISOTROPIC CASE EXAMPLE: ISOTROPIC CASE

.

DISCRETIZATION DISCRETIZATION

The gradient operator can be discretized as: The gradient operator can be discretized as:

 

2 2

, ) , ( ) (            y x f y x f f  

 

, ) , ( ) , (                y y f x y f y x f  

2 2 1 2 2 1 2 1

) , ( ) , ( ) , ( n n f n n f n n f

y x

  

Which is based on a discretization of directional derivatives: derivatives:

) , ( ) , ( ) , (

2 1 2 1 2 1

n n f n n f n n f

y x

  

FINITE IMPULSE RESPONSE MODEL

Discrete operators for derivative estimation can be estimated as

FINITE IMPULSE RESPONSE MODEL

sc ete ope ato s o de at e est at o ca be est ated as FIR filters. ) ( * ) ( ) ( n n h n n f n n f  ) , ( * ) , ( ) , (

2 1 2 1 2 1

n n h n n f n n f

y y

 ) , ( ) , ( ) , (

2 1 2 1 2 1

n n h n n f n n f

x x

 ) ( n n f ) ( n n f ) , (

2 1 n

n hx ) , (

2 1 n

n f ) , (

2 1 n

n f x

DISCRETE DIFFERENTIAL OPERATORS DISCRETE DIFFERENTIAL OPERATORS

• Pixel difference: luminance difference between to

neighbour pixels along orthogonal directions. neighbour pixels along orthogonal directions.

) 1 , ( ) , ( ) , (    k j f k j f k j f x

Separable filters

) , 1 ( ) , ( ) , ( k j f k j f k j f y   

Separable filters

     1           1 1

x

h          1 1

y

h            

EXAMPLE: PIXEL DIFFERENCE C

SEPARATED PIXEL DIFFERENCE SEPARATED PIXEL DIFFERENCE

• If farther pixels are chosen there is a higher noise

rejection, and there is no phase translation in edge definition.

) 1 , ( ) 1 , ( ) , (     k j f k j f k j fx

) 1 ( ) 1 ( ) ( k j f k j f k j f     ) , 1 ( ) , 1 ( ) , ( k j f k j f k j f y 

         1           1 1

x

h          1

y

h         1

EX.: SEPARATED PIXEL DIFFERENCE S C

ROBERTS EDGE EXTRACTION ROBERTS EDGE EXTRACTION

) 1 , 1 ( ) , ( ) , (     k j f k j f k j f x ) , 1 ( ) 1 , ( ) , ( k j f k j f k j f y    

        1 1 h        1 1 h        1

x

h        1

y

h

EX.: ROBERTS METHOD O S O

PREWITT METHOD PREWITT METHOD

Estimation can be improved involving more samples for te Estimation can be improved involving more samples for te gradient operator 3x3

     

    f 

                 

1 1 1 1 1 1 1 , 1 1 , 1 *

2 1 2 1

                   f f f f n n f n n f x f K   = vertical low pass* horizontal high pass

           

1 , 1 1 , 1 , 1 , 1

2 1 2 1 2 1 2 1

          n n f n n f n n f n n f vertical low pass horizontal high pass

     

1 , 1 1 , 1 *

2 1 2 1

               n n f n n f f K  

                 

1 , 1 1 , 1 1 , 1 ,

2 1 2 1 2 1 2 1

              n n f n n f n n f n n f y  = vertical high pass* horizontal low pass

GRADIENT ESTIMATION

• Gradient modulus = the value of the higher directional derivative

GRADIENT ESTIMATION

g

• Gradient phase = orientation of the higher directional derivative

Squared lattice =eight possible directions     1 1     1 1

 

          1 1 1 1 ,

2 1 n

n h

 

            1 1 1 1 ,

2 1 n

n h E NE       1 1 1   1 1

 

              1 1 1 ,

2 1 n

n h

 

              1 1 1 1 ,

2 1 n

n h N NW     1 1 1     1 1

GRADIENT ESTIMATION

    1 1

GRADIENT ESTIMATION

 

         1 1 1 1 ,

2 1 n

n h

 

            1 1 1 1 ,

2 1 n

n h W SW       1 1     1 1      1 1 1     1 1

 

           1 1 1 1 1 1 ,

2 1 n

n h

 

            1 1 1 1 ,

2 1 n

n h S SE     1 1 1     1 1

EX.: PREWITT METHOD 3X3 O 3 3

SOBEL METHOD SOBEL METHOD

S di i f P i fil

 Same dimensions of Prewitt filter  Different weight for central point in different

directions.

  1 1        1 2 1              1 1 2 2

r

h          1 2 1

c

h

 Sobel for exagonal grids.

    1 1            2 2 1 1 h        1 1

EX.: SOBEL METHOD SO O

COMPARISON COMPARISON

Roberts Sobel Prewitt Sobel Prewitt

FREI CHEN OPERATOR FREI-CHEN OPERATOR

I t i t i il t P itt

 Isotropic operator similar to Prewitt  different weight for the central point in the 4

directions.

 The gradient has the same value for horizontal,

vertical and diagonal edges.          2 2 1 1 h           1 2 1 h       1 1 2 2

r

h        1 2 1

c

h

EX.: METODO DI FREI-CHEN O O C

EXTENDED OPERATORS EXTENDED OPERATORS

A li it f th f ti d th d i th i k i

 A limit for the aforementioned methods is their weakness in

accurate edge detection when SNR is very low.

 A possible solution in to extend their size: the result will be a  A possible solution in to extend their size: the result will be a

less accurate edge positioning but noise rejection will be higher. higher.

PREWITT METHOD 7X7 PREWITT METHOD 7X7

 Extension of Prewitt 3X3

       1 1 1 1 1 1 1 1 1 1 1 1

 Extension of Prewitt 3X3  Normalized impulse response:

                    1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

r

h                   1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

r

       1 1 1 1 1 1

EX.: PREWITT 7X7 METHOD O

ABDOU 7X7 METHOD ABDOU 7X7 METHOD

I filt k th t i li d i l i ht

 Is a filter mask that gives a linear decreasing sample weight as

they are farther from the edge. Its behaviour is close to a truncated pyramid truncated pyramid.

 The normalized impulse response is:

            1 2 2 2 2 1 1 1 1 1 1 1                1 2 3 3 2 1 1 2 3 3 2 1 1 2 2 2 2 1

r

h               1 2 2 2 2 1 1 2 3 3 2 1

r

         1 1 1 1 1 1

EX.: ABDOU 7X7 METHOD OU O

FURTHER EXTENDED OPERATORS FURTHER EXTENDED OPERATORS

 It is possible to obtain extended gradient filters for low SNR

t s poss b e to obta e te ded g ad e t te s o

• S

conditions convolving a 3x3 operator with a low-pass filter.

) , ( * ) , ( ) , ( k j h k j h k j h

PG G



 HG (j,k) is one of the previously considered filters, HPB (j,k) is

the impulse response for a low-pass filter.

EXAMPLE

 

EXAMPLE

Prewitt 3X3 convoled with:

h        1 9 1 1 1 1 1 1 *       9 1 1 1

...and we get the Smoothed Prewitt 5X5

  1 1 1 1           1 1 1 1 2 2 2 2 h              3 3 3 3 2 2 2 2         1 1 1 1

LAPLACIAN BASED METHODS LAPLACIAN BASED METHODS

 1D Case

1D Case

Fi d i g f th d d i ti di g t

 Find zero-crossing of the second derivative, corresponding to

inflection points.

LAPLACIAN LAPLACIAN

 For the 2D case the 2nd order differential operator

is the Laplacian

2 2 2 2 2

) , ( ) , ( )) , ( ( ) , ( y x f y x f y x f y x f           

Isotropic operator

2 2

y x  

Isotropic operator More sensible to noise with respect to gradient

 False edges can be generated due to noise  False edges can be generated due to noise.

Thinner edges are produced.

ALGORITHM: CASE 2D ALGORITHM: CASE 2D

G di t

O x y ( , )

Gradient estimation

Yes Yes

2



O x y ( , ) 

) , (

2

y x f  edge N N

= 0 ?

2



threshold

f x y ( , )

No No

ZERO-CROSSING WITHOUT THRESHOLD Sobel vs. Laplacian Sobel vs. Laplacian

LAPLACIAN DISCRETIZATION LAPLACIAN DISCRETIZATION

Can be seen as the convolution of f(n1,n2) with the impulse h( ) f li t response h(n1,n2) of a linear system.

4 NEIGHBOURS METHOD 4 NEIGHBOURS METHOD

S bl li d filt

 Separable normalized filter

 Unit gain for continuous component  The sign of h(n

h(n n ) can be changed itho t changes in

 The sign of h(n

h(n1,n2) can be changed without changes in the final result (since we are looking for zeros of laplacian)

    1                      2 1 4 1 1 2 1 4 1 ) , (

2 1 n

n h               1 1 1 4 4             1 1 4 1 4 1  

EX.: 4 NEIGHBOURS METHOD G OU S O

LAPLACIAN DISCRETIZATION LAPLACIAN DISCRETIZATION

 The laplacian can be approximated with finite

differences

) , ( ) , 1 ( ) , ( ) , ( k j f k j f y x f y x f     

differences

) , ( ) , 1 ( ) , ( k j f k j f y x f x

x

   ) (

2 f

 ) 1 ( ) ( 2 ) 1 ( ) , 1 ( ) , ( ) , ( ) , (

2 2

k f k f k f k j f k j f k j f x y x f

x x xx

       ) , 1 ( ) , ( 2 ) , 1 ( k j f k j f k j f     

DISCRETIZATION EXAMPLES DISCRETIZATION EXAMPLES Prewitt method

 Not separable filter

h n n ( , )

1 2

1 8 1 1 1 1 8 1            

8 Neighbours method

( )

1 2

8 1 1 1         

8 Neighbours method

 Similar to Prewitt but with a separable formulation

                              1 4 1 2 1 2 1 2 2 2 1 1 1 1 1 2 1 1 2 1 1 ) ( n n h                             2 1 2 1 4 1 8 1 1 1 2 2 2 8 1 2 1 1 2 1 8 ) , (

2 1 n

n h

EX.: 8 NEIGHBOURS METHOD 8 G OU S O

EX.: PREWITT NOT SEPARABLE O S

NOISE PRESENCE NOISE PRESENCE

 When noise is significant these filters could not be accurate for

diagonal edges. The Prewitt filter can work even in regions with high density of edges.           2 4 2 1 2 1 8 1 ) , (

2 1 n

n h

 Since ege are directional and noise can generate luminance

variations zero crossing for laplacian could find non correct         1 2 1 8 variations, zero-crossing for laplacian could find non-correct edges.

EX.: LAPLACIAN FOR DIAGONAL EDGES C O GO G S

SUPER RESOLUTION (LAPLACIAN) SUPER-RESOLUTION (LAPLACIAN)

First method.

 Given two neighbour pixels, mark as possible edge point

the intrapixels points if the laplacian values in the two i l h diff t i pixels have different signs.

 Assume as effective edge the point, among them, with the

largest gradient largest gradient.

 Apply this analysis to all the pixels couples. LAPLACIAN Sign analysis Magnitude comparison

     

2 3 4

I I I I I I                  

8 7 6 1 5

I I I I I I          

SUPERRESOLUTION SUPERRESOLUTION

Second method: analytical approach

Approximate the continuous form of function f(n1,n2) with a

2D polynomial in order to describe the laplacian in an analytical way. analytical way.

Polynomial example:

2 2 2 2 2 2

ˆ

 where Ki are the weights obtained from the discrete image. 2 2 9 2 8 2 7 2 6 5 2 4 3 2 1

) , ( c r K rc K c r K c K rc K r K c K r K K c r F         

 r and c then become continuous variables

r and c then become continuous variables associtated to a discrete image matrix.

     ( ) , ( ) W r c W 1 2 1 2

 Polynomial formulation can be found with small efforts.

SUPERRESOLUTION SUPERRESOLUTION

M th d 3 dg fitti g

 Method 3: edge fitting

Based on the comparison with an edge model based on

th i g l ti the image correlation.

 < s x ( )      a per x < x a + h per x x

 



L x0 2

 Exemple: comparison with a step function

 





 

L x

dx x s x f E

2

) ( ) (

the edge is accepted if the MSE is below a threshold.

SUPERRESOLUTION SUPERRESOLUTION

f ( i )    a for (x cos +y sin )< ( , ) a+h for (x cos +y sin ) s x y           

 Case 2D

Compare f(x,y) with a 2D step, where  and

Compare f(x,y) with a 2D step, where  and  specify the distance and the angle, in polar coordinates of the edge point from the center of the considered circular the center of the considered circular region.

The edge is accepted if the MSE is below a

The edge is accepted if the MSE is below a Threshold

 

2

( , ) ( , ) MSE f x y s x y dxdy  



circle

GAUSSIAN FILTERING GAUSSIAN FILTERING

Why we should use a gaussian function?

• Since the Fourier transform of a Gaussian is still e gaussian,

g ,

• The cut-off frequency can be expressed as a function of the width of the

impulse response

• It has a low aliasing

It has a low aliasing

• The filter is separable and isotropic at the same time

2 /

) ( ) ( ) ( ) , (

2 2



    x

e h y h x h y x h The noise sensitivity (numerous zero crossing) decrease as the

2

2 ) (  

 x

h y ( g) filter strength (width) increase.

“LOG” OPERATOR LOG OPERATOR

Th l i fil h i bl ff

 The low pass gaussian filter has a variable cut-off

frequency.   

2 2

y x          

2

2 exp ) , ( y x y x h                   2 ) ( exp 2 ) , (

2 2 2 2 2 y x y x

H     2

 It follows that the standard deviation  is inversely

ti l t th filt idth proportional to the filter width.

“LOG” operator “LOG” operator

 

) , ( * ) , ( ) , (

2

y x h y x f y x g  

LOG operator LOG operator

• Laplacian and filtering are interchangeable since both of them

are linear

 

) , ( * ) , ( ) , (

2

y x h y x f y x g    

2 2 2 2 2

                

2 2 2 4 2 2 2 2 2

2 exp 2 ) , ( y x y x y x h

 

 

2 2 2 2 2 2

2 2 exp 2 ) , (

y x y x

y x h                     2    

DIFFERENCE OF GAUSSIANS

The LOG, Laplacian of a Gaussian corresponds to the

DIFFERENCE OF GAUSSIANS

e OG, ap ac a o a Gauss a co espo ds o e derivative of a gaussian with respect to 22 The laplacian can be approximated with the difference of two gaussian filters with different .

DOG APPLICATION DOG APPLICATION