RGB++ How "Side Information" Improves Computational - - PowerPoint PPT Presentation

RGB++

How "Side Information" Improves Computational Photography and Computer Vision (...or how to make better pictures)

Sabine Süsstrunk

Images and Visual Representation Group (IVRG)

This%talk%is%about%linking%Informa3on% Theory%(or%Signal%Processing)%with...

Photography

Source%Coding

Encoder Decoder

S R ˆ S

Source%Coding

Encoder Decoder

ˆ S S

Source%Coding

Encoder Decoder

ˆ S S

System Performance

Source%Coding

Encoder Decoder

Lossless.Source.Coding% ˆ S = S

ˆ S S

System Performance

Source%Coding

Encoder Decoder

Lossy.Source.Coding% ˆ S 6= S

ˆ S S

System Performance

Source:%LighAield

Source Encoder

Lightfield

Decoder

ˆ S S

System Performance

Encoder:%Camera

Source

Lightfield

Decoder

Camera

ˆ S S

System Performance

Decoder:%Display

Display Camera

ˆ S S

Lightfield

System Performance

Es3mated%Source:%Image

Display Image Camera

ˆ S S

Lightfield

System Performance

System%Performance:%you%or%me

Camera Display Image

you or me ˆ S S

Lightfield

Lossless%Source%Coding

Display Image

I.like.it!%

)

you or me

)

Camera

ˆ S S

Lossy%Source%Coding

)

Camera Display Lightfield Image

I.do.NOT.like.it!%

you or me

)

ˆ S S

hEp://www.jigsawexplorer.com/puzzles/disassembledJcameraJjigsawJpuzzle/

Image%Capture

Lightfield Optics Color Filter Array + Silicon Sensor

t

Image%Capture

Lightfield Optics Color Filter Array + Silicon Sensor

= t

Source%Coding

Display Image Camera

ˆ S U

Lightfield

S

Remote%Source%Coding

Encoder Decoder

“Noisy” Observation

U ˆ S S U

Side%Informa3on

Encoder Decoder

Side Information

U ˆ S S

“Noisy” Observation

Side%Informa3on

Encoder Decoder

Side Information

U ˆ S S Y

“Noisy” Observation

Side%Informa3on

Encoder Decoder

Side Information

U ˆ S S Y

“Noisy” Observation

Side%Informa3on

Encoder Decoder

Side Information

U ˆ S S Y Y

“Noisy” Observation

Side%Informa3on

Encoder Decoder

Side Information

U ˆ S S Y Y

“Noisy” Observation

Side%Informa3on

Encoder Decoder

Side Information

U ˆ S S Y Y

“Noisy” Observation

Side%Informa3on%can%improve%system% performance%if%it%is%correlated.with% the%source%signal.

)

Side%Informa3on

Encoder Decoder

Side Information

“Similar”.images!

U ˆ S S Y

“Noisy” Observation

Mul3ple%Exposures

Mul3ple%exposures High%dynamic%range%rendering

L.%Meylan,%D.%Alleysson,%and%S.%Süsstrunk,%A%Model%of%Re3nal%Local%Adapta3on%for%the%Tone%Mapping%of% Color%Filter%Array%Images.%JOSA%A,%2007.

Mul3ple%Illumina3ons

Ambient%Light Flash Flash/No%Flash%Composite

E.%Eisemann%and%F.%Durand.%Flash%Photography%Enhancement%via%Intrinsic%Religh3ng,%ACM%SIGGRAPH,%2004.

Mul3ple%Time%Instances

Mul3ple%Exposures “Good”%Face%Composite

M.%F.%Cohen%and%R.%Szeliski.%The%Moment%Camera.%IEEE%Computer,%2006.

Our%Research

Encoder Decoder

Side Information Noisy Observation

Near?Infrared SemanAcs

U ˆ S S Y

Mul$ple'Spectra

400 nm 700 nm 1100 nm

Visible Spectrum Near-Infrared

Visible (Red, Green, and Blue: RGB)

Mul$ple'Spectra

400 nm 700 nm 1100 nm

Visible Spectrum Near-Infrared

Visible (Red, Green, and Blue: RGB) Near-infrared (NIR)

EPFL%

= Ecole Polytechnique Fédéral de Lausanne = Swiss Federal Institute of Technology, Lausanne

Normal%view%(RGB)...

Enhanced%view%(RGB+NIR)

Visible NIR

Haze

sparticle < λ/10 : Es ∝ E0 λ4

Light.scaBering%in%the%atmosphere%produces%haze% (Raleigh’s%law):%

Visible NIR

Haze

sparticle < λ/10 : Es ∝ E0 λ4

Light.scaBering%in%the%atmosphere%produces%haze% (Raleigh’s%law):%

Intui3on

• The%high.frequencies!of%the%visible%image%can%be%

exchanged!by%the%high%frequencies%of%the%NIR% image.

SpectralJspa3al%correla3on%of%50%RGB/NIR%image%pairs Z.%Sadeghipoor,%Y.%M.%Lu,%and%S.%Süsstrunk,%Correla3onJbased%Joint%Acquisi3on%and%Demosaicing%of% Visible%and%NearJinfrared%Images,%ICIP,%2011.

Algorithm

• Mul3Jresolu3on%analysis%and%fusion%of%both%visible%

luminance%V and%NIR%intensi3es%N (criterion:%contrast):%

L.%Schaul,%C.%Fredembach,%and%S.%Süsstrunk,%Color%image%deJhazing%using%the%nearJinfrared,%IEEE% Interna7onal%Conference%on%Image%Processing%(ICIP),%2009.

Id

k = Ia k − Ia k+1

Ia

k+1

• Analysis%Criterion:

Synthesis

• Keep%the%visible%low%frequency%informa3on,%and%then%

retain%the%highest.contrast%from%either%the%visible%or% the%NIR%image.%

F0 = V a

n

• k=1

(max(V d

k , N d k ) + 1)

L.%Schaul,%C.%Fredembach,%and%S.%Süsstrunk,%Color%image%deJhazing%using%the%nearJinfrared,%IEEE% Interna7onal%Conference%on%Image%Processing%(ICIP),%2009.

Results

USM Gaussian Bilateral Visible

Original Unsharp.Masking

Results

L.%Schaul,%C.%Fredembach,%and%S.%Süsstrunk,%Color%image%deJhazing%using%the%nearJinfrared,%IEEE% Interna7onal%Conference%on%Image%Processing%(ICIP),%2009.

De?hazing.using.NIR

Results

Original

L.%Schaul,%C.%Fredembach,%and%S.%Süsstrunk,%Color%image%deJhazing%using%the%nearJinfrared,%IEEE% Interna7onal%Conference%on%Image%Processing%(ICIP),%2009.

System%Performance?

• R. Fa&al, Single image dehazing,
• Proc. Interna,onal Conference
• n Computer Graphics and

Interac,ve Techniques, 2008.

• K. He, J. Sun, and X. Tang, Single

image haze removal using dark channel prior, Proc. IEEE Conference on Computer Vision and Pa>ern Recogni,on, 2009.

System%Performance

R.%FaEal,%Single%image%dehazing,% Proc.%Interna7onal%Conference

• n%Computer%Graphics%and%

Interac7ve%Techniques,%2008. K.%He,%J.%Sun,%and%X.%Tang,%Single% image%haze%removal%using%dark% channel%prior,%Proc.%%IEEE% Conference%on%Computer%Vision% and%PaFern%Recogni7on,%2009.

• NIR%radia3on%has%different%light%scaEering%and%absorp'on/

reflec'on%behavior%than%visible%light.

Material%Absorp3on/Reflectances

Yellow%Arrow:%same%color%in%RGB,%different%intensity%in%NIR Green%Arrow:%different%colors%in%RGB,%same%intensity%in%NIR%

N.%Salama3,%C.%Fredembach,%and%S.%Süsstrunk,%Material%Classifica3on%Using%Color%and%NIR%Images,%IS&T/SID% 17th%Color%Imaging%Conference,%2009.

T.%Igarashi,%K.%Nishino,%and%S.K.%Nayar,%The%appearance%of%human%skin.%Technical%Report:%CUCSJ024J05,%2005. Hair,%fine%wrinkles

Epidermis Dermis Subcu3s

Hair,%fine%wrinkles Stratum%corneum Basal%Cells Melanosytes%% Collagenous%network Blood%vessels Fat%Cells

Penetra3on%of%radia3on%in%skin

• AbsorpAon%and%scaBering%are%inversely%propor3onal%to%

wavelength.

Intui3on

• If%NIR%penetrates%deeper%into%skin,%then%surface%

imperfec3ons%are!less%visible.

Intui3on

• Unwanted%skin%imperfec3ons%(freckles,%pores,%warts,%

wrinkles)%are%usually%small. Base Detail Original

NIR RGB BF

NIRdetail NIRbase Y Ydetail

BF

Ybase

TRGBJYCC

Ysmooth RGBsmooth

Algorithm

BF=Bilateral%Filter

NIR RGB BF

NIRdetail NIRbase Y Ydetail

BF

Ybase

TRGBJYCC

Ysmooth RGBsmooth

Algorithm

BF=Bilateral%Filter

NIR RGB BF

NIRdetail NIRbase Y Ydetail

BF

Ybase

TRGBJYCC

Ysmooth RGBsmooth

Algorithm

BF=Bilateral%Filter

Skin%Smoothing

Visible NIR Visible + NIR

C.%Fredembach,%N.%Barbuscia,%and%S.%Süsstrunk,%Combining%visible%and%nearJinfrared%images%for%realis3c%skin% smoothing,%IS&T/SID%17th%Color%Imaging%Conference,%2009.

Skin%Smoothing

Visible NIR Visible + NIR

C.%Fredembach,%N.%Barbuscia,%and%S.%Süsstrunk,%Combining%visible%and%nearJinfrared%images%for%realis3c%skin% smoothing,%IS&T/SID%17th%Color%Imaging%Conference,%2009.

Skin%Smoothing

Visible NIR Visible + NIR

C.%Fredembach,%N.%Barbuscia,%and%S.%Süsstrunk,%Combining%visible%and%nearJinfrared%images%for%realis3c%skin% smoothing,%IS&T/SID%17th%Color%Imaging%Conference,%2009.

Skin%Smoothing%

C.%Fredembach,%N.%Barbuscia,%and%S.%Süsstrunk,%Combining%visible%and%nearJinfrared%images%for%realis3c%skin% smoothing,%IS&T/SID%17th%Color%Imaging%Conference,%2009.

System%Performance?

S.%Süsstrunk,%C.%Fredembach,%and%D.%Tamburrino,%Automa3c%skin%enhancement%with%visible%and%nearJ infrared%image%fusion,%ACM%Mul7media,%2010.%%Best.Demo.Award.

Camera%Museum%in%Vevey

Visible Visible%+%NIR

S.%Süsstrunk,%C.%Fredembach,%and%D.%Tamburrino,%Automa3c%skin%enhancement%with%visible%and%nearJ infrared%image%fusion,%ACM%Mul7media,%2010.%%Best.Demo.Award.

Preference:%26.6%%(2,331) Preference:%73.4%%(6,424)

Color%Constancy

Overcast%Sky Es3mated%Color%Temperature: 7500K Sunny%Day Es3mated%Color%Temperature: 5000K Incandescent%Lamp Es3mated%Color%Temperature: 3000K

• C. Fredembach and S. Süsstrunk, Illuminant estimation and detection using near infrared, IS&T/SPIE

Electronic Imaging, Digital Photography V, Vol. 7250, 2009.

Shadow%Detec3on

Visible NIR Binary.Shadow.Mask

• D. Rüfenacht, C. Fredembach, and S. Süsstrunk, “Automatic and Accurate Shadow Detection Using Near-

infrared Information,” submitted to IEEE Transactions on Pattern Recognition and Machine Intelligence, 2013.

Comparison%with%visible%only

Visible Intrinsic!Image Visible!Segmenta7on Visible Visible Intrinsic NIR Visible!Only Visible!+!NIR N.%Salama3%and%S.%Süsstrunk,%MaterialJBased%Object%Segmenta3on%using%NearJInfrared%Informa3on,% IS&T/SID%18th%Color%Imaging%Conference,%2010.%Best.InteracAve.Paper.Award

Image%Database%(477%RGB+NIR)

• 477%image%pairs%for%9%scene%categories:

M.%Brown%and%S.%Süsstrunk,%Mul3Jspectral%SIFT%for%Scene%Category%Recogni3on,%CVPR%2011.

Seman3c%Image%Segmenta3on

N.%Salama3,%D.%Larlus,%G.%Csurka,%and%S.%Süsstrunk,%Seman3c%Image%Segmenta3on%Using%Visible%and%NearJ Infrared%Channels,%in%ECCV’s%4th%Color%and%Photometry%in%Computer%Vision%Workshop%(2012).

Visible NIR V%Seg V+NIR%Seg GT

Why%NearJInfrared?

Near Infrared Blocking Filter (Hot Mirror)

Camera%Design

All%digital%cameras%are%inherently% sensi3ve%to%NIR…

All%digital%cameras%are%inherently% sensi3ve%to%NIR…

...if we remove the hot mirror!

Side%Informa3on

Encoder Decoder

NIR Images

Y U ˆ S S

“Noisy” Observation

RGBN%Camera

Encoder Decoder

Less “Noisy” Observation with NIR

U ˆ S S

Seman3cs

Encoder Decoder

WORDS

U ˆ S S Y

“Noisy” Observation

Which%image%do%you%like%beEer?

sand sunset

Which%image%do%you%like%beEer?

dark snow Image%rendering%depends%on%context

Camera%Modes

“automa3c”%mode “foilage”%mode

Canon%PowerShot%S100

Correla3on?

?

How%do%we%link%image%characteris3cs%with%words?

Wordl%of%the%wikipedia%page%“word”

• MIR%Flickr%database,%1%Million%annotated%images.
• Selec3on%based%on%Flickr’s%“interes3ngness”%score.
• 1%MegaPixel,%assume%sRGB.

gold,%oregoncoast,%fortstevens,%astoria,%outside,% lightroom,%sigma,%1020mm,%nikon,%d40,% diamondclassphotographer,%grass,%yellow,%blue,%sky,% clouds,%singlecloud,%color,%saturated,%happy,%field

Meredith_Farmer (cc)

MJ%Huiskes,%B%Thomee,%MS%Lew.%New%trends%and%ideas%in%visual%concept%detec3on:%the%MIR%Flickr% retrieval%evalua3on%ini3a3ve,%ACM%Mul7media,%2010.

Image%Database

1M%images%+%keywords gold gold 996’688 3312

Sta3s3cal%Framework

Sta3s3cal%Framework

gold gold

stevewhis (cc) laura.bell (cc) paige_eliz (cc) Arty Smokes (cc) TW Collins (cc) raketentim (cc) golfnride (cc) Dunechaser (cc) Tal Bright (cc) 10863752@N00 (cc)

4 6

Sta3s3cal%Framework

gold gold

stevewhis (cc) laura.bell (cc) paige_eliz (cc) Arty Smokes (cc) TW Collins (cc) raketentim (cc) golfnride (cc) Dunechaser (cc) Tal Bright (cc) 10863752@N00 (cc)

percentage%of%yellow%pixels 10% 90% 70% 5% 8% 0% 30% 3% 2% 9%

MannJWhitneyJWilcoxon%ranksum%test cardinali3es

• f%both%sets

F.%%Wilcoxon,%Individual%comparisons%by%ranking%methods,%Biometrics%Bulle3n,%1(6):80–83,%1945

σ2

T = nwnw(nw + nw + 1)

12 µT = nw(nw + nw + 1) 2 nw, nw z = T − µT σT = 30 − 22 4.69 ≈ 1.71

sorted%list: 10% 90% 70% 5% 8% 0% 30% 3% 2% 9% 1 2 3 4 5 6 7 8 rank%index: ranksum: 9 10

T = 4 + 7 + 9 + 10 = 30

Sta3s3cal%Framework

MannJWhitneyJWilcoxon%ranksum%test cardinali3es

• f%both%sets

σ2

T = nwnw(nw + nw + 1)

12 µT = nw(nw + nw + 1) 2 nw, nw z = T − µT σT = 30 − 22 4.69 ≈ 1.71

sorted%list: 10% 90% 70% 5% 8% 0% 30% 3% 2% 9% 1 2 3 4 5 6 7 8 rank%index: ranksum: 9 10

T = 4 + 7 + 9 + 10 = 30

Sta3s3cal%Framework

significantly%more%yellow%pixels%in%gold%images.

z > 0

• CIELAB%histogram

15x15x15%bins.

• %%%values%indicate%

significance%of%a% keyword%w.r.t.%to%a% characteris3c.

gold

z

z

%%%%Distribu3on

A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,%ACM%Mul7media,%2012.

Other%Characteris3cs

Spa3al%lightness%layout. light

−7 −6 −5 −4 −3 −2 −1 1

A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,%ACM%Mul7media,%2012.

Spa3al%chroma%layout. barn

−6 −4 −2 2 4 6 8

Other%Characteris3cs

A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,%ACM%Mul7media,%2012.

Spa3al%Gabor%filter%layout. fireworks

−10 −5 5

Other%Characteris3cs

A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,%ACM%Mul7media,%2012.

Seman3c%Image%Enhancement

Gray%scale%tone% mapping snow gold Color%enhancement% macro Change%depthJofJfield

[Zhuo%and%Sim,%2011]

Seman3c%Enhancement

seman3c processing input

• utput

image component seman3c component gold characteris3cs seman3c component

A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,%ACM%Mul7media,%2012.

Seman3c%Component

50 100 150 200 250 1 2 3 4 5 6 pixel value z value red green blue

significance%values%for%gold

100 200 50 100 150 200 250 input value

• utput value

red green blue identity

global%scale%parameter

f 0 = ⇢ 1/ (1 + Sz) if z ≥ 0 1 + S|z| if z < 0

S

Tone%mapping%func3on f

z

Seman3c%Enhancement

seman3c processing input

• utput

gold characteris3cs

• utput value

image component

A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,%ACM%Mul7media,%2012.

Image%Component

gold weight%map

Gaussian%blurring%kernel (1%%of%image%diagonal)

ω = ⇥ gσ ∗ zw

• col(p)

⇤1

gσ

⇥ · ⇤1

0 normaliza3on%operator

ω

Seman3c%Enhancement

input

• utput

gold characteris3cs

• utput value

seman3c processing

A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,%ACM%Mul7media,%2012.

Seman3c%Enhancement

Enhance%relevant%characteris3cs%in%relevant%regions. input

• utput

gold characteris3cs

• utput value

Iout = (1 − ω) · Iin

in + ω · Itmp

ω

Itmp

A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,%ACM%Mul7media,%2012.

sand

sand

snow

snow

dark

dark

silhouette

silhouette

sunset

sunset

grass

grass

autumn

autumn

strawberry

strawberry

sky

sky

banana

banana

macro

macro

flower

flower

macro

macro

sand

System%Performance?

%%8%keywords,%30%images,%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%,%30%observers %%=28’800%image%comparisons.

S = {0.5, 1, 2, 4}

• riginal

proposed

Amazon%Mechanical%Turk

0.5 1.0 2.0 4.0 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 S approval rate white dark sand snow contrast silhouette portrait light

A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,%ACM%Mul7media,%2012.

Conclusion

• Adding%side.informaAon%to%the%encoding,%decoding%or%

both%can%improve%system.performance,

. and%that%can%be%“like”%or%“dislike”%of%photos.

Conclusion

• Adding%side%informa3on%to%the%encoding,%decoding%or%

both%can%improve%system%performance

. and%that%can%be%like%or%dislike%of%photos

• The%given%examples%are%in%image.enhancement%

(computa3onal%photography),%but%the%same%concept%is% also%applicable%to%computer.vision%applica3ons.

Conclusion

• Considering%todays%capture%devices%(cell%phones,%android%

cameras)%with%mulAple.sensors.and.connected.to.the. internet,%there%is%more%side%informa3on%available%than% ever!

hEp://gizmoreport.com/huaweiJascendJmateJinfo/

Conclusion

• Considering%todays%capture%devices%(cell%phones,%android%

cameras)%with%mulAple.sensors.and.connected.to.the. internet,%there%is%more%side%informa3on%available%than% ever!

hEp://www.newgadgets.de/46122/nikonJcoolpixJs800cJ16JmegapixelJkameraJmitJandroid/

Conclusion

• Side%Informa3on%needs%to%be%correlated%with%the%signal,%

and%that%might%not%always%be%so%evident%with%nonJimage% signals.

. Let%your%imagina3on%rule!

Acknowledgment

• Joint work with

J Nathalie Barbuscia, EPFL J Nicolas Bonnier, OCE now Canon CiSRA J Clément Fredembach, EPFL now Canon CiSRA J Albrecht Lindner, EPFL J Yue M. Lu, EPFL now Harvard J Neda Salamati, EPFL J Zahra Sadeghipoor, EPFL J Lex Schaul, EPFL now ETHZ J Appu Shaji, EPFL J Daniel Tamburrino, EPFL now ECAL

• Funding

J Swiss National Science Foundation J OCE Print Logic, a Canon Group J Hasler Foundation J Xerox Foundation

Thank%you!

ivrg.epfl.ch