Improve Tag Suggestions in Image Sharing Sites Position of .. o - PowerPoint PPT Presentation

Using Visual Features to Improve Tag Suggestions in Image Sharing Sites Position of .. o Mathias Lux, mlux@itec.uni-klu.ac.at o Oge Marques, omarques@fau.edu o Arthur Pitman, apitman@edu.uni-klu.ac.at Department for Information Technology, Klagenfurt University, Austria

Agenda http:// www.uni-klu.ac.at ● Motivation ● Proposed Architecture ● Current State ● Preliminary Conclusions 2 ITEC, Klagenfurt University, Austria

Motivation http:// www.uni-klu.ac.at ● 5,000 + uploads per minute on Flickr. o Only 20%-25% are tagged ● Why are not all images tagged? o Benefits of tagging are obvious … o But effort is considered too high … 3 ITEC, Klagenfurt University, Austria

http:// www.uni-klu.ac.at Focus on the annotation process … 4 ITEC, Klagenfurt University, Austria

Motivation II http:// www.uni-klu.ac.at ● Tagging images includes visual information ● Visual information retrieval in “narrow domains” has shown some success o … to bridge the semantic gap ● Tags as narrow domains? o e.g. Ferrari or sunset 5 ITEC, Klagenfurt University, Austria

Assumptions & Process http:// www.uni-klu.ac.at ● User has selected/uploaded a photo ● User has assigned at least one tag ● Our Task: o Find more appropriate tags o Present them to the user o User decides which tags are “good” 6 ITEC, Klagenfurt University, Austria

Our Approach http:// www.uni-klu.ac.at 1. Find possible suggestions (tag based) 2. Find image sets per suggestion 3. Compare input image to different image sets 4. Re-rank the possible suggestions 7 ITEC, Klagenfurt University, Austria

Example: Tag “juggling” http:// www.uni-klu.ac.at juggling + clown juggling + fire juggling + training Input image 8

Architecture http:// www.uni-klu.ac.at 9 ITEC, Klagenfurt University, Austria

Behind the curtains … http:// www.uni-klu.ac.at ● Image sets are “ground truth” for tag suggestion ● Several (arbitrary) features extracted ● Fuzzy classifiers are trained ● Best feature+classifier is selected ● Input image gets classified ● Best matching class is ranked highest, etc. 10 ITEC, Klagenfurt University, Austria

http:// www.uni-klu.ac.at ● Finding tag suggestions statistically ● Download image sets for suggestions ● Extract global image features ● Experiments with classifiers 11

Current state http:// www.uni-klu.ac.at 12 ITEC, Klagenfurt University, Austria

Preliminary Conclusions http:// www.uni-klu.ac.at ● Efficient implementation poses an engineering problem (CBIR, network, …) ● Promising results for some tags o Found several tags considered as noise for our use case: flickrdiamonds, abigfave, 1imageaday , … ● We might find some “good questions” … o How to define a “narrow domain”? o How to find “narrow domains”? o etc. 13 ITEC, Klagenfurt University, Austria

Mathias Lux http:// www.uni-klu.ac.at Klagenfurt University, ITEC Austria Contact o mlux @ itec.uni-klu.ac.at o http://www.itec.uni-klu.ac.at/~mlux o http://www.flickr.com/photos/mathias_l 14 ITEC, Klagenfurt University, Austria