Tool went online July 1st, 2005 Til Feb 11 th , 2009 Visitors: - PowerPoint PPT Presentation

I MAGE A NNOTATION 1 Yu-Ting Peng

O UTLINE � Current object recognition datasets � Human Computation � LabelMe � ESP � Pickaboom � CAPTCHA&RECAPTCHA 2

C URRENT OBJECT RECOGNITION DATASETS 3

H UMAN - BASED C OMPUTATION � Humans can recognize 30000 entry-level obeject catagories. � Current techniques insufficient- can only recognize a few object catagories 4

W EB - BASED A NNOTATION T OOLS � Provide a way of building large annotated datasets by relying on the collaborative effort of a large population of users � Provide a drawing interface that works on many platforms, is easy to use, and allow instant platforms, is easy to use, and allow instant sharing of the collected data. � Examples � LabelMe � ESP � Peekaboom 5

� Tool went online July 1st, 2005 � Til Feb 11 th , 2009 Visitors: 64771 Available Images: 176180 Annotated Images: 51455 Annotated Images: 51455 Object categories: 4418 6

S TRENGTH � Object class recognition and localization � Drawing polygons � Runs on almost any web browser-Javascript drawing interface � Resulting labels are stored in the XML file Resulting labels are stored in the XML file format-makes the annotations portable and easy to extend. � Matlab toolbox 7

8

3D 9

Q UERYING OBJECTS 10

Q UERYING SCENES 11

S TATISTICS 12

13

W ORDNET � How to choose the label? 14

15

16

E XPERIMENTS IN THE PAPER � percentage of pixels labeled per image. � the number of labeled objects per image. � the average number of control points clicked for each category. � Distributions of object location showing where in Distributions of object location showing where in the image each instance occurs � object sizes, relative to the image size showing what is the typical size that the object has in the LabelMe dataset. � how many parts an object has � Depth ordering 17

C ORRELATION - TOP PAIRS 18

building streetlight door sign person window sidewalk car tree road sky 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 19

sky streetlight sign window person mountain sidewalk car road tree building 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 20

tree plant streetlight sign window person sidewalk car road building sky 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 21

road door streetlight sign person window tree sidewalk sky car building 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 22

car streetlight wheel sign person window tree sidewalk sky road building 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 23

chair person window bookshelf floor lamp lamp wall mouse keyboard screen table 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 24

mouse wall speaker bookshelf cpu mug mug chair mousepad keyboard table screen 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 25

L EAST FREQUENT PAIRS 26

H IGH VARIANCE OF C LICKED POINTS STD Mean 27

LOW VARIANCE OF C LICKED POINTS STD Mean 28

V ARIANCE OF COLOR DISTRIBUTION Variance 29

D ISCUSSION � Quailty control � Text label itself � Statics pictures & Sequence pictures � dataset 30

31

32

33

34

ESP PLAYER 1 PLAYER 2 GUESSING: CAR GUESSING: BOY GUESSING: HAT GUESSING: CAR GUESSING: KID SUCCESS! YOU AGREE ON CAR SUCCESS! 35 YOU AGREE ON CAR

G OOGLE I MAGE L ABELER � The ESP Game has been licensed by Google. 36

T HE L IMITATIONS OF ESP � The ESP Game can label images, but it cannot: � Find the objects being labeled. • Determine the way in which the object appears – does the label “car” refer to the text “car” or an actual car in the image? 37

P EEKABOOM The Guesser guesses: •Flower •Petal •Butterfly The Revealer clicks on parts of the image and shows �������������������������� them to the Guesser. 38

C OMPLETING THE I MAGE C YCLE ESP game server unlabeled labeled images images Peekaboom game server located images 39

H INTS 40

HINTS 41