Edge Preserving Filtering Median Filter Bilateral Filter Shai - - PowerPoint PPT Presentation

Edge Preserving Filtering Median Filter Bilateral Filter

Shai Avidan Tel-Aviv University

Slide Credits (partial list)

• Rick Szeliski
• Steve Seitz
• Alyosha Efros
• Yacov Hel-Or
• Marc Levoy
• Bill Freeman
• Fredo Durand
• Sylvain Paris

A Gentle Introduction to Bilateral Filtering and its Applications

“Fixing the Gaussian Blur”: the Bilateral Filter

Sylvain Paris – MIT CSAIL

Box Average

average

input square neighborhood

• utput

normalized box function sum over all pixels q intensity at pixel q result at pixel p

Equation of Box Average

• S

I B I BA

q q p

q p ) ( ] [

Square Box Generates Defects

• Axis-aligned streaks
• Blocky results

input

• utput

unrelated pixels unrelated pixels related pixels

Box Profile

pixel position pixel weight

Strategy to Solve these Problems

• Use an isotropic (i.e. circular) window.
• Use a window with a smooth falloff.

box window Gaussian window

Gaussian Blur

average

input per-pixel multiplication

• utput

*

input

box average

Gaussian blur

normalized Gaussian function

Equation of Gaussian Blur

• S

I G I GB

q q p

q p || || ] [

• Same idea: weighted average of pixels.

1

unrelated pixels unrelated pixels uncertain pixels uncertain pixels related pixels

Gaussian Profile

pixel position pixel weight

• 2

2

2 exp 2 1 ) (

• x

x G

size of the window

Spatial Parameter

• S

I G I GB

q q p

q p || || ] [

• small

large input limited smoothing strong smoothing

Blur Comes from Averaging across Edges

* * *

input

• utput

Same Gaussian kernel everywhere.

Bilateral Filter No Averaging across Edges

* * *

input

• utput

The kernel shape depends on the image content.

[Aurich 95, Smith 97, Tomasi 98]

space weight

not new

range weight

I new

normalization factor

new

Bilateral Filter Definition: an Additional Edge Term

• S

I I I G G W I BF

q q q p p p

q p | | || || 1 ] [

r s

• Same idea: weighted average of pixels.

Illustration a 1D Image

• 1D image = line of pixels
• Better visualized as a plot

pixel intensity pixel position

space

Gaussian Blur and Bilateral Filter

space range normalization

Gaussian blur

• S

I I I G G W I BF

q q q p p p

q p | | || || 1 ] [

r s

• Bilateral filter

[Aurich 95, Smith 97, Tomasi 98] space space range p p q q

• S

I G I GB

q q p

q p || || ] [

Space and Range Parameters

• space s : spatial extent of the kernel, size of

the considered neighborhood.

• range r : “minimum” amplitude of an edge
• S

I I I G G W I BF

q q q p p p

q p | | || || 1 ] [

r s

How to Set the Parameters

Depends on the application. For instance:

• space parameter: proportional to image size

– e.g., 2% of image diagonal

• range parameter: proportional to edge

amplitude

– e.g., mean or median of image gradients

• independent of resolution and exposure

A Few More Advanced Remarks

Bilateral Filter Crosses Thin Lines

• Bilateral filter averages across

features thinner than ~2s

• Desirable for smoothing: more pixels = more robust
• Different from diffusion that stops at thin lines

close-up kernel

Iterating the Bilateral Filter

• Generate more piecewise-flat images
• Often not needed in computational

photo.

] [

) ( ) 1 ( n n

I BF I

input

1 iteration

2 iterations

4 iterations

Bilateral Filtering Color Images

• S

I I I G G W I BF

q q q p p p

q p | | || || 1 ] [

r s

• S

G G W I BF

q q q p p p

C C C q p || || || || 1 ] [

r s

• For gray-level images

For color images

intensity difference color difference

The bilateral filter is The bilateral filter is extremely easy to adapt to your need. extremely easy to adapt to your need.

scalar 3D vector (RGB, Lab)

input

• utput

Applications

• Image denoising
• HDR Compression

– Key idea: – Break image into base and detail layers – Compress base – Recompose image

Fast Implementation: 3D Kernel

• Idea: represent image data such that the weights

depend only on the distance between points

[Paris and Durand 06]

pixel intensity pixel position

1D image Plot I = f ( x ) far in range close in space

1st Step: Re-arranging Symbols

• S

S

I I G G W I I I G G W I BF

q q p p q q q p p p

q p q p | | || || | | || || 1 ] [

r s r s

• 1

| | || || | | || || ] [

r s r s

• S

S

I I G G W I I I G G I BF W

q q p p q q q p p p

q p q p

• Multiply first equation by Wp

1st Step: Summary

• Similar equations
• No normalization factor anymore
• Don’t forget to divide at the end
• 1

| | || || | | || || ] [

r s r s

• S

S

I I G G W I I I G G I BF W

q q p p q q q p p p

q p q p

2nd Step: Higher-dimensional Space p p

space range

• “Product of two Gaussians” = higher dim.

Gaussian

2nd Step: Higher-dimensional Space p p

space range

• 0 almost everywhere, I at “plot location”

2nd Step: Higher-dimensional Space p p

• 0 almost everywhere, I at “plot location”
• Weighted average at each point = Gaussian blur

2nd Step: Higher-dimensional Space p p

• 0 almost everywhere, I at “plot location”
• Weighted average at each point = Gaussian blur
• Result is at “plot location”

• New num. scheme:
• simple operations
• complex space

• Strategy:

downsampled convolution Conceptual view, not exactly the actual algorithm

The Algorithm

NEW IDEA : ‘Joint’ or ‘Cross’ Bilateral’ Petschnigg(2004) and Eisemann(2004)

Bilateral two kinds of weights NEW : get them from two kinds of images.

• Smooth image A pixels locally, but
• Limit to ‘similar regions’ of image B

Why do this?

To get ‘best of both images’

Ordinary Bilateral Filter

Bilateral two kinds of weights, one image A :

• S

A A A G G W A BF

q q q p p p

q p | | || || 1 ] [

r s

• c

c

s s

Image A:

• f(x)

f(x) x x

‘Joint’ or ‘Cross’ Bilateral Filter

NEW: two kinds of weights, two images

• S

A B B G G W A BF

q q q p p p

q p | | || || 1 ] [

r s

• c

c

s s

A: Noisy, dim

(ambient image)

c c

s s

B: Clean,strong

(Flash image)

Image A: Warm, shadows, but too Noisy

(too dim for a good quick photo) (too dim for a good quick photo)

No-flash

Image B: Cold, Shadow-free, Clean

(flash: simple light, ALMOST no shadows) (flash: simple light, ALMOST no shadows)

MERGE BEST OF BOTH: apply ‘Cross Bilateral’ or ‘Joint Bilateral’

(it really is much better!)

Video Enhancement Using Per Pixel Exposures (Bennett, 06)

From this video: ASTA: Adaptive S Spatio- T Temporal Accumulation Filter

ASTA

Replace pixel difference with a general dissimilarity measure D(x,x)=0, D(x,y)=D(y,x)

Shot noise

A new dissimilarity measure

Instead of comparing pixel intensities, look at their local spatial neighborhood

• Raw Video Frame:

(from FIFO center)

• Histogram stretching;

(estimate gain for each pixel)

• ‘Mostly Temporal’ Bilateral Filter:

– Average recent similar values, – Reject outliers (avoids ‘ghosting’), spatial avg as needed – Tone Mapping

The Process for One Frame

The Process for One Frame

• Raw Video Frame:

(from FIFO center)

• Histogram stretching;

(estimate gain for each pixel)

• ‘Mostly Temporal’ Bilateral Filter:

– Average recent similar values, – Reject outliers (avoids ‘ghosting’), spatial avg as needed – Tone Mapping

The Process for One Frame

• Raw Video Frame:

(from FIFO center)

• Histogram stretching;

(estimate gain for each pixel)

• ‘Mostly Temporal’ Bilateral Filter:

– Average recent similar values, – Reject outliers (avoids ‘ghosting’), spatial avg as needed – Tone Mapping

(color: # avg’ pixels)

The Process for One Frame

• Raw Video Frame:

(from FIFO center)

• Histogram stretching;

(estimate gain for each pixel)

• ‘Mostly Temporal’ Bilateral Filter:

– Average recent similar values, – Reject outliers (avoids ‘ghosting’), spatial avg as needed – Tone Mapping

Bilateral Filter Variant: Mostly Temporal

• FIFO for Histogram-stretched video

– Carry gain estimate for each pixel; – Use future as well as previous values;

• Expanded Bilateral Filter Methods:

– Static scene? Temporal-only avg. works well – Motion? Bilateral rejects outliers: no ghosts!

• Generalize: ‘Dissimilarity’ (not just || Ip – Iq ||2)
• Voting: spatial filter de-noises motion

Multispectral Bilateral Video Fusion (Bennett,07)

• Result:

– Produces watchable result from unwatchable input – – VERY VERY robust; accepts almost any dark video; – Exploits temporal coherence to emulate Low-light HDR video, without special equipment