Visipedia Tool Ecosystem for Dataset Curation and Annotation Serge - - PowerPoint PPT Presentation

▶
visipedia tool ecosystem for dataset curation and
SMART_READER_LITE
LIVE PREVIEW

Visipedia Tool Ecosystem for Dataset Curation and Annotation Serge - - PowerPoint PPT Presentation

Apr 03, 2023 426 likes •1.27k views

Visipedia Tool Ecosystem for Dataset Curation and Annotation Serge Belongie Outline Visipedia Project Overview Related Work Bird Datasets ViBE: Visipedia Back End Future Work Outline Visipedia Project Overview

slide-1
SLIDE 1

Visipedia Tool Ecosystem for Dataset Curation and Annotation

Serge Belongie

slide-2
SLIDE 2
  • Visipedia Project Overview
  • Related Work
  • Bird Datasets
  • ViBE: Visipedia Back End
  • Future Work

Outline

slide-3
SLIDE 3
  • Visipedia Project Overview
  • Related Work
  • Bird Datasets
  • ViBE: Visipedia Back End
  • Future Work

Outline

slide-4
SLIDE 4

What Is Visipedia?

http://en.wikipedia.org/wiki/Bird

  • A user-generated encyclopedia of visual knowledge
  • An effort to associate articles with large quantities
  • f well-organized, intuitive visual concepts
slide-5
SLIDE 5
  • People will willingly label or organize certain

images if:

○ They are interested in a particular subject matter ○ They have the appropriate expertise

Ring-tailed lemur Thruxton Jackaroo

Motivation

slide-6
SLIDE 6

[BikeRumor.com]

slide-7
SLIDE 7
slide-8
SLIDE 8
slide-9
SLIDE 9

Motivation

  • Construct comprehensive, intuitive

knowledge base of visual objects

  • Provide better text-to-image search and

image-to-article search

slide-10
SLIDE 10
  • Visipedia Project Overview
  • Related Work
  • Bird Datasets
  • ViBE: Visipedia Back End
  • Future Work

Outline

slide-11
SLIDE 11

Related Work: Systems

  • {Leaf,Dog,Bird}snap [Belhumeur et al.]
  • Oxford Flowers [Nilsback & Zisserman]
  • STONEFLY9 [Martínez-Muñoz et al.]
  • omoby [IQEngines.com]
  • 20 Questions game [20q.net]
  • ReCAPTCHA [von Ahn et al.]
  • Wikimedia Commons

*

slide-12
SLIDE 12

Related Work: Methods

  • Relevance Feedback
  • Active Learning
  • Expert Systems
  • Decision Trees
  • Feature Sharing & Taxonomies
  • Parts & Attributes
  • Crowdsourcing & Human

Computation

*

slide-13
SLIDE 13

Motivation: Computer Vision Perspective

  • Need for more training data

○ Beyond the capacity of any one research group ○ Better quality control

  • Need for more realistic data

○ Let people define what tasks are important ○ Study tightly-related categories

slide-14
SLIDE 14

Dealing With a Large Number of Related Classes

  • Standard classification methods fail because:

○ Few training examples per class available ○ Variation between classes is small ○ Variation within a class is often still high

Brewer’s Sparrow Vesper Sparrow

slide-15
SLIDE 15
slide-16
SLIDE 16

*

slide credit: Neeraj Kumar

slide-17
SLIDE 17

Visual 20 Questions

  • “Computer Vision” module = Vedaldi’s VLFeat
  • VQ Geometric Blur, color/gray SIFT spatial pyramid
  • Multiple Kernel Learning
  • Per-Class 1-vs-All SVM
  • 15 training examples per bird species
  • Choose question to maximize expected Information Gain
slide-18
SLIDE 18

Pose Normalized Deep ConvNets

[Van Horn, Branson, Perona, Belongie BMVC 2014 ]

slide-19
SLIDE 19
slide-20
SLIDE 20
  • Visipedia Project Overview
  • Related Work
  • Bird Datasets
  • ViBE: Visipedia Back End
  • Future Work

Outline

slide-21
SLIDE 21

Birds-200 Dataset

6033 images over 200 bird species

slide-22
SLIDE 22

Image Harvesting

  • Flickr: text search on species name
  • MTurk: presence/absence and bounding

boxes

*

slide-23
SLIDE 23
slide-24
SLIDE 24

*

  • Modeling various aspects of annotation:

○ Worker competency – accuracy in labeling ○ Worker expertise – better at labeling some things

than others, based on their strengths

○ Worker bias – how one weighs errors ○ Task difficulty – ambiguous images are universally

hard to label

○ True label – the ground truth label value

  • We leverage the "Multidimensional Wisdom of Crowds"

[Welinder et al. 2010]

The human annotation process

slide-25
SLIDE 25

*

Indigo Bunting

Task: Find the Indigo Bunting

Blue Grosbeak

Types of annotator errors

slide-26
SLIDE 26

*

slide-27
SLIDE 27

*

slide-28
SLIDE 28

*

slide-29
SLIDE 29

*

slide-30
SLIDE 30

*

slide-31
SLIDE 31

*

slide-32
SLIDE 32

*

slide-33
SLIDE 33

*

Image formation process

Object presence or absence Signal seen by ideal

  • bserver

Factors influencing appearance

slide-34
SLIDE 34

*

Entire annotation process

Annotator expertise Annotator noise Annotator bias Image formation

slide-35
SLIDE 35
slide-36
SLIDE 36
slide-37
SLIDE 37

*

Multidimensional ability of annotators

slide-38
SLIDE 38

*

Multidimensional ability of annotators

slide-39
SLIDE 39

*

Multidimensional ability of annotators

slide-40
SLIDE 40

*

Worker “schools of thought”

Ducks Ducks and grebes Ducks, grebes, and geese

slide-41
SLIDE 41

*

  • Models can capture multidimensionality of

annotation process

  • How well does this generalize to continuous

annotations? Different tasks require different reviewing strategies. Predicting quality accurately can reduce the number of labels needed.

Discussion: quality management

slide-42
SLIDE 42
slide-43
SLIDE 43

Attribute Labeling

  • Attributes from whatbird.com
  • 25 visual attributes 288 binary attributes

○ similar to “dichotomous key” in biology

  • MTurk interface

○ {guessing, probably, definitely}

  • 3-5x redundancy factor

*

slide-44
SLIDE 44
slide-45
SLIDE 45
slide-46
SLIDE 46
slide-47
SLIDE 47
slide-48
SLIDE 48

MTurker Label Certainty

slide-49
SLIDE 49

MTurker Feedback

  • “These hits were fun. Will you be posting more of them

anytime soon? Thanks!”

  • “These are Beautiful birds and I am enjoying this hit

collection”

  • “I really enjoy doing your hits, they are fun and interesting.

Thanks.”

  • “Love doing these because I'm a bird watcher.”
  • “the birds are so cute..hope u can send more kind of birds”
  • “I haven't really studied birds, but doing these HITs has

made me realize just how beautiful they are. It has also made me aware of the many different types of birds. Thank you”

  • “I REALLY LOVE THE COLOR OF THE BIRDS.”
  • “Thank you for providing this job. The fact that the images

are beautiful to look at make it a lot more enjoyable to do!”

  • “Enjoyable to do.”
  • Hourly Wage ≈ $1.25

*

slide-50
SLIDE 50
slide-51
SLIDE 51

CCUB Taster25

"Sweet" Taster "Bitter" Taster

slide-52
SLIDE 52

CCUB Taster25 Results

Using the winning ILSVRC '11 approach by [F. Perronnin, et al.], training on 25 images/category

Average Performance: 64.7%

Baseline Performance: The winning ILSVRC '11 approach of Florent Perronnin and Jorge Sanchez.

  • Dense SIFT and Color

Descriptors

  • Aggregated using Fisher

vectors [Perronnin, et al. ECCV 10]

  • Linear SVMs with SGD
  • Same parameters used in

ILSVRC

slide-53
SLIDE 53

CCUB Taster25 Results

Average Performance: 79.4%

Using the winning ILSVRC '11 approach by [F. Perronnin, et al.], training on 50 images/category

slide-54
SLIDE 54

http://birds.cornell.edu/nabirds

slide-55
SLIDE 55
  • Visipedia Project Overview
  • Related Work
  • Bird Datasets
  • ViBE: Visipedia Back End
  • Future Work

Outline

slide-56
SLIDE 56

Vibe Demo

http://visipedia.org http://vibe.visipedia.org

slide-57
SLIDE 57
slide-58
SLIDE 58
slide-59
SLIDE 59
slide-60
SLIDE 60
slide-61
SLIDE 61
slide-62
SLIDE 62
slide-63
SLIDE 63
slide-64
SLIDE 64
slide-65
SLIDE 65
slide-66
SLIDE 66
slide-67
SLIDE 67
slide-68
SLIDE 68
slide-69
SLIDE 69
slide-70
SLIDE 70
  • Visipedia Project Overview
  • Related Work
  • Bird Datasets
  • ViBE: Visipedia Back End
  • Future Work

Outline

slide-71
SLIDE 71

Future Work

  • Beyond Birds
  • Attribute Induction
  • Relevance Feedback
slide-72
SLIDE 72
slide-73
SLIDE 73
slide-74
SLIDE 74

Perceptual Embedding

slide-75
SLIDE 75

visipedia.org

Thank You

  • Caltech: Steve Branson, Grant Van Horn, Pietro Perona
  • UCSD: Catherine Wah
  • Cornell: Jessie Barry, Miyoko Chu
  • BYU: Ryan Farrell
  • Google Focused Research Award
slide-76
SLIDE 76
slide-77
SLIDE 77

Extra Slides

slide-78
SLIDE 78

Computational Pathology

slide-79
SLIDE 79
slide-80
SLIDE 80

Populating Visipedia

  • Populate Wikipedia articles with more visual

data using large quantities of unlabeled data on the web

World wide web Visipedia

slide-81
SLIDE 81

Attribute-Based Classification

  • Train classifiers on

attributes instead of

  • bjects
  • Attributes are shared by

different object classes

  • Attributes provide the

ingredients necessary to recognize each object class

Lampert et al. 2009 Farhadi et al. 2009

slide-82
SLIDE 82

Attribute-Based Classification

  • Number of attributes

is less than number

  • f classes
  • Attribute

classification tasks might be easier

  • Makes it easier to

incorporate human knowledge

www.whatbird.com

slide-83
SLIDE 83