RGB++ How "Side Information" Improves Computational - PowerPoint PPT Presentation

Comparison%with%visible%only Visible Intrinsic Intrinsic!Image Visible!Only Visible!+!NIR NIR Visible!Segmenta7on N.%Salama3%and%S.%Süsstrunk,%MaterialJBased%Object%Segmenta3on%using%NearJInfrared%Informa3on,% IS&T/SID%18th%Color%Imaging%Conference ,%2010.% Best.InteracAve.Paper.Award Visible Visible

Image%Database%(477%RGB+NIR) • 477%image%pairs%for%9%scene%categories: M.%Brown%and%S.%Süsstrunk,%Mul3Jspectral%SIFT%for%Scene%Category%Recogni3on,% CVPR %2011.

Seman3c%Image%Segmenta3on Visible NIR V%Seg V+NIR%Seg GT N.%Salama3,%D.%Larlus,%G.%Csurka,%and%S.%Süsstrunk,%Seman3c%Image%Segmenta3on%Using%Visible%and%NearJ Infrared%Channels,%in% ECCV’s%4th%Color%and%Photometry%in%Computer%Vision%Workshop %(2012).

Why%NearJInfrared?

Camera%Design Near Infrared Blocking Filter (Hot Mirror)

All%digital%cameras%are%inherently% sensi3ve%to%NIR…

All%digital%cameras%are%inherently% sensi3ve%to%NIR… ...if we remove the hot mirror!

Side%Informa3on S “Noisy” U Encoder Observation Y NIR Images ˆ S Decoder

RGBN%Camera Less “Noisy” S U Encoder Observation with NIR ˆ S Decoder

Seman3cs S “Noisy” U Encoder Observation Y WORDS ˆ S Decoder

Which%image%do%you%like%beEer? sand sunset

Which%image%do%you%like%beEer? dark snow Image%rendering%depends%on% context

Camera%Modes “automa3c”%mode “foilage”%mode Canon%PowerShot%S100

Correla3on? ? How%do%we%link%image%characteris3cs%with%words? Wordl%of%the%wikipedia%page%“word”

Image%Database • MIR%Flickr%database,%1%Million%annotated%images. • Selec3on%based%on%Flickr’s%“interes3ngness”%score. • 1%MegaPixel,%assume%sRGB. gold ,%oregoncoast,%fortstevens,%astoria,%outside,% lightroom,%sigma,%1020mm,%nikon,%d40,% diamondclassphotographer,%grass,%yellow,%blue,%sky,% clouds,%singlecloud,%color,%saturated,%happy,%field Meredith_Farmer (cc) MJ%Huiskes,%B%Thomee,%MS%Lew.%New%trends%and%ideas%in%visual%concept%detec3on:%the%MIR%Flickr% retrieval%evalua3on%ini3a3ve, %ACM%Mul7media,% 2010.

Sta3s3cal%Framework gold gold 1M%images%+%keywords 996’688 3312

Sta3s3cal%Framework gold gold TW Collins (cc) raketentim (cc) laura.bell (cc) stevewhis (cc) Dunechaser (cc) golfnride (cc) paige_eliz (cc) Arty Smokes (cc) Tal Bright (cc) 10863752@N00 (cc) 4 6

Sta3s3cal%Framework gold gold 0% 3% TW Collins (cc) raketentim (cc) 5% 70% laura.bell (cc) stevewhis (cc) 8% 30% Dunechaser (cc) golfnride (cc) 90% 10% paige_eliz (cc) Arty Smokes (cc) 2% 9% Tal Bright (cc) 10863752@N00 (cc) percentage%of%yellow%pixels

Sta3s3cal%Framework sorted%list: 0% 2% 3% 5% 8% 9% 10% 30% 70% 90% rank%index: 1 5 2 3 4 6 7 8 9 10 ranksum: T = 4 + 7 + 9 + 10 = 30 MannJWhitneyJWilcoxon%ranksum%test µ T = n w ( n w + n w + 1) T = n w n w ( n w + n w + 1) σ 2 2 12 cardinali3es z = T − µ T = 30 − 22 ≈ 1 . 71 n w , n w of%both%sets 4 . 69 σ T F.%%Wilcoxon,%Individual%comparisons%by%ranking%methods,%Biometrics%Bulle3n,%1(6):80–83,%1945

Sta3s3cal%Framework sorted%list: 0% 2% 3% 5% 8% 9% 10% 30% 70% 90% rank%index: 1 5 2 3 4 6 7 8 9 10 ranksum: T = 4 + 7 + 9 + 10 = 30 MannJWhitneyJWilcoxon%ranksum%test µ T = n w ( n w + n w + 1) T = n w n w ( n w + n w + 1) σ 2 2 12 cardinali3es z = T − µ T = 30 − 22 ≈ 1 . 71 n w , n w of%both%sets 4 . 69 σ T significantly%more%yellow%pixels%in% gold %images. z > 0

%%%%Distribu3on z gold • CIELAB%histogram 15x15x15%bins. • %%% values%indicate% z significance%of%a% keyword%w.r.t.%to%a% characteris3c. A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,% ACM%Mul7media ,%2012.

Other%Characteris3cs Spa3al%lightness%layout. light 1 0 − 1 − 2 − 3 − 4 − 5 − 6 − 7 A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,% ACM%Mul7media ,%2012.

Other%Characteris3cs Spa3al%chroma%layout. barn 8 6 4 2 0 − 2 − 4 − 6 A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,% ACM%Mul7media ,%2012.

Other%Characteris3cs Spa3al%Gabor%filter%layout. fi reworks 5 0 − 5 − 10 A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,% ACM%Mul7media ,%2012.

Seman3c%Image%Enhancement Gray%scale%tone% snow mapping gold Color%enhancement% macro Change%depthJofJfield [Zhuo%and%Sim,%2011]

Seman3c%Enhancement input characteris3cs image output component seman3c processing seman3c seman3c gold component component A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,% ACM%Mul7media ,%2012.

Seman3c%Component significance%values%for% gold Tone%mapping%func3on f 250 6 red green 5 200 blue 4 output value z value 150 3 z 2 100 red 1 green 50 blue 0 identity 0 0 50 100 150 200 250 0 100 200 pixel value input value ⇢ 1 / (1 + Sz ) if z ≥ 0 f 0 = 1 + S | z | if z < 0 global%scale%parameter S

Seman3c%Enhancement input characteris3cs image output component seman3c processing 250 200 output value gold 150 100 red green 50 blue identity 0 0 100 200 input value A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,% ACM%Mul7media ,%2012.

Image%Component weight%map gold ω �⇤ 1 ⇥ � ω = col( p ) g σ ∗ z w 0 ⇤ 1 Gaussian%blurring%kernel 0 normaliza3on%operator ⇥ g σ · (1%%of%image%diagonal)

Seman3c%Enhancement input characteris3cs output seman3c processing 250 200 output value gold 150 100 red green 50 blue identity 0 0 100 200 input value A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,% ACM%Mul7media ,%2012.

Seman3c%Enhancement Enhance%relevant%characteris3cs%in%relevant%regions. input characteris3cs output ω I out = (1 − ω ) · I in 250 in + ω · I tmp 200 output value I tmp gold 150 100 red green 50 blue identity 0 0 100 200 input value A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords% with%Applica3ons%in%Seman3c%Image%Enhancement,% ACM%Mul7media ,%2012.

sand

sand

snow

snow

dark

dark

silhouette

silhouette

sunset