Estimating scene typicality from human ratings and image - - PowerPoint PPT Presentation

▶

Dec 04, 2023 47 likes •361 views

Estimating scene typicality from human ratings and image features Krista A. Ehinger, Jianxiong Xiao, Antonio T orralba, Aude Oliva chair kitchen Are there

SLIDE 1

Estimatingscenetypicalityfrom humanratingsandimagefeatures

KristaA.Ehinger,Jianxiong Xiao,AntonioT

rralba,

Aude Oliva

SLIDE 2

“chair”

SLIDE 3

“ kitchen”

SLIDE 4

Arethere“ typical”examplesofscene

categories?

Domoretypicalexamplescontainmoreofthe

visualfeaturesrelevantforscene categorization?

SLIDE 5

SUNDatabase:899scenecategories,130,519images

Xiao,Hays,Ehinger,Oliva,&T

rralba (2010)

SLIDE 6

SceneUnderstanding(SUN)database

Scenes: baseball_field warehouse theater

pera_house

forest Not scenes: equator golden_gate_bridge

utdoors

forest – evergreen forest – deciduous

SLIDE 7

Ratingscenetypicality

706categories
124,901images(22– 2360images/category)
935peoplecontributedratingsthrough

Amazon’sM echanicalTurkservice

SLIDE 8

SLIDE 9

SLIDE 10

SLIDE 11

Ratingscenetypicality

Categoryimages ... 20imagespertrial

SLIDE 12

Ratingscenetypicality

Categoryimages ... 20imagespertrial

SLIDE 13

Ratingscenetypicality

Categoryimages ... 20imagespertrial ... 77,331trials,eachimageappeared1215times

SLIDE 14

Ratingscenetypicality

Categoryimages ... 20imagespertrial ...

14appearances 3“ best” 0“ worst” 12appearances 1“ best” 1“ worst”

... 77,331trials,eachimageappeared1215times

SLIDE 15

Typicalityscore

“ best”votes– k(“worst”votes) Numberofappearances Typicality=

Voteda“ best ” exemplaron8outof 12appearances

0.67 Voteda“ worst ” exemplaron7outof 12appearances

0.52

k=0.9

SLIDE 16

M osttypicalbeaches M osttypicalbedrooms

Examples

SLIDE 17

Leasttypicalbeaches Leasttypicalbedrooms

Examples

SLIDE 18

...

Ratingscenetypicality

Categoryimages 20imagespertrial ...

14appearances 3“ best” 0“ worst” 12appearances 1“ best” 1“ worst”

... Simulation:Computer randomlyselectedimages

SLIDE 19

Typicalityscores:Experimentvs.simulation

SLIDE 20

M osttypical Average

boardwalk

M osttypical Average

closet

SLIDE 21

Arethere“ typical”examplesofscene

categories? – Y

es,thereisarangeoftypicalityinscene categories

SLIDE 22

Arethere“ typical”examplesofscene

categories? – Y

es,thereisarangeoftypicalityinscene categories

Domoretypicalexamplescontainmoreofthe

visualfeaturesrelevantforscene categorization?

SLIDE 23

. . .

Imagefeatures

. . .

“ beach” Aremoretypicalimages classifiedmoreaccurately? Training T est

SLIDE 24

M achinevisionclassifier

Featurehistograms

(color,edges, texturefeatures)

GISTdescriptor
SIFTdescriptor
HOG(histogramsof
rientedgradients)
SSIM(self

similarity)

Geometricclasses

Felzenszwalb,Girshick,M cAllester,&Ramanan (2010);Hoiem,Efros,&Herbert(2007);Lazebnik, Schmid,&Ponce (2006);Oliva&Torralba(2001);Renninger&M alik(2004);Shechtman &Irani (2007)

Onevs.allSVMclassifiertrained

n50randomimagesineach

category,testedon50images

SLIDE 25

Classificationperformance

Classifier = 38% correct Chance = 0.25% correct Humans = 68% correct

SLIDE 26

Average classifier performance = 38% Chance = 0.25%

Sceneclassificationvs.scenetypicality

SLIDE 27

Classificationconfidencevs.scenetypicality

SLIDE 28

Arethere“ typical”examplesofscene

categories? – Y

es,thereisarangeoftypicalityinscene categories

Domoretypicalexamplescontainmoreofthe

visualfeaturesrelevantforscene categorization? – Y

es,moretypicalscenesareclassifiedmore accurately

SLIDE 29

SLIDE 30

SLIDE 31

Thankyou!

ThisworkisfundedbyNSFCAREERaward(0546262),NSFgrants (0705677and1016862),andaNEIgrant(EY02484)toA.O;NSFCAREER award(0747120)toA.T;andGoogleresearchawardstoA.OandA.T . Jianxiong Xiao AntonioT

rralba

Aude Oliva