Hierarchy-of-Visual-Words: a Learning-based Approach for Trademark - - PowerPoint PPT Presentation

▶
hierarchy of visual words a learning based approach for
SMART_READER_LITE
LIVE PREVIEW

Hierarchy-of-Visual-Words: a Learning-based Approach for Trademark - - PowerPoint PPT Presentation

Oct 19, 2022 204 likes •422 views

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work Hierarchy-of-Visual-Words: a Learning-based Approach for

slide-1
SLIDE 1

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Hierarchy-of-Visual-Words: a Learning-based Approach for Trademark Image Retrieval

Vítor Lourenço, Gabriela Silva, Leandro A. F. Fernandes

vitorlourenco@id.uff.br

Instituto de Computação Universidade Federal Fluminense

October 31st, 2019

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 1 / 20

slide-2
SLIDE 2

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Introduction and Motivation

  • Trademarks carry the identification, the reputation, and the

quality meanings of the associated product or service

  • In the year of 2018, about 200000 trademarks were deposited

in the Brazilian Intellectual Property Agency

  • Law offices hires hundreds of people to manually analyze each

new trademark image

  • Not scalable
  • Error-prune
  • Costly

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 2 / 20

slide-3
SLIDE 3

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Related Work

  • Content-based trademark image retrieval systems

Wei et al. (2009), Qi et al. (2010), Anuar et al. (2013), Liu et al. (2017)

  • Pros: Geometrically partitions the trademark image and uses

global and local descriptors separately

  • Cons: Does not consider topological relationship between

trademark image’s objects

  • Hierarchical representation of multiobjects images

Alajlan et al. (2006 e 2008), Chen e Weng (2017)

  • Pros: Encodes topological relationship between image’s objects

within a hierarchical structure

  • Cons: Measures similarity between hierarchies on inference

time

  • Bag-based models Silva et al. (2013, 2014, 2017)
  • Pros: Provides bags of graph-like data structures
  • Cons: Uses texture-based representation and does not encode

topological relationship

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 3 / 20

slide-4
SLIDE 4

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Contributions

  • A new learning-based framework for the hierarchical

representation of elements in binary images

  • Its application on trademark image description and retrieval

from image databases

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 4 / 20

slide-5
SLIDE 5

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

The Hierarchy-of-Visual-Words Framework

E V A L U A T I O N

Training Trademark Images

(a)

Trademark Image Binarization

(b)

Shape Extraction

(c)

Shape’s Feature Extraction

(d)

T R A I N I N G . . . . . .

Visual Words Codebook Learning

(e)

< B > < C > < A > < D > < C > < C > < B > < D >

. . .

< C > < TA > < TB > Query Trademark Image

(h)

Trademark Image Binarization

(i)

Shape Extraction

(j)

Shape’s Feature Extraction

(k)

Hierarchical Relationship Encoding

(l)

< B > < A > Similar Images Retrieval

(m)

Hierarchical Relationship Encoding

(f)

Visual Hierarchies Codebook Learning

(g)

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 5 / 20

slide-6
SLIDE 6

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Hierarchy-of-Visual-Words: Training Stage

. . . . . .

< C > < C > < B > < D >

. . .

< C >

Trademark Image Binarization Shape Extraction Shape's Feature Extraction Hierarchical Relationship Encoding

< B > < C > < A >

Visual Words Codebook Learning

< D >

Training Trademark Images

< TA > < TB >

Visual Hierarchies Codebook

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 6 / 20

slide-7
SLIDE 7

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Trademark Image Binarization and Shape Extraction

. . . . . .

< B > < C > < A > < D > < C > < C > < B > < D >

. . .

< C > < TA > < TB >

  • Digital trademark images converted into binary images
  • Convert the color images to grayscales
  • Apply median filter to reduce impulsive noise
  • Apply bilateral filter to remove texture without losing overall

shapes

  • Apply Otsu’s method on the textureless grayscale image to
  • btain the final binary image
  • Shape extraction regarding objects and holes

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 7 / 20

slide-8
SLIDE 8

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Shape’s Feature Extraction and Visual Words Codebook Learning

. . . . . .

< B > < C > < A > < D > < C > < C > < B > < D >

. . .

< C > < TA > < TB >

  • Feature extraction descriptors
  • Zernike moments
  • Circularity
  • Average bending energy
  • Eccentricity
  • Convexity
  • Visual codebook learning
  • k-means clustering

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 8 / 20

slide-9
SLIDE 9

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Hierarchical Relationship Encoding

. . . . . .

< B > < C > < A > < D > < C > < C > < B > < D >

. . .

< C > < TA > < TB >

  • Relationship encoding
  • Shape inclusion
  • Shape exclusion
  • Visual hierarchy

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 9 / 20

slide-10
SLIDE 10

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Visual Hierarchies Codebook Learning

. . . . . .

< B > < C > < A > < D > < C > < C > < B > < D >

. . .

< C > < TA > < TB >

  • Hierarchies dissimilarity matrix
  • Rename cost

δr(na, nb) = distE(λa, λb)

  • Insert and remove costs

δx(n) = α 2 m (m − 1)

m

  • i=1

m

  • j=i+1

distE(λi, λj), α = min{log−1

2

L, log−1

2

D}

  • Mean-shift clustering with RBF kernel
  • Dissimilarity matrix

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 10 / 20

slide-11
SLIDE 11

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Hierarchy-of-Visual-Words: Evaluation Stage

Query Trademark Image Trademark Image Binarization Shape Extraction Shape’s Feature Extraction

< B > < A >

Hierarchical Relationship Encoding Learned Visual Words Codebook Similar Images Retrieval Learned Visual Words Codebook

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 11 / 20

slide-12
SLIDE 12

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Similar Images Retrieval

  • is a mean

shift cluster

  • is a mean

shift cluster

  • similar

non-similar

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 12 / 20

slide-13
SLIDE 13

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Experiments Protocol: Datasets and Metrics

  • MPEG-7 Core Experiment CE-Shape-1
  • 1, 400 binary images
  • 70 classes
  • 20 similar images per class
  • MPEG-7 Region Shape Dataset CE-2
  • 871 binary images
  • 51 classes
  • 11 to 21 similar images per class
  • 2, 750 images that do not belong to any category *
  • Used Metrics
  • Precision-Recall Curve
  • F1 Score

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 13 / 20

slide-14
SLIDE 14

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Experiments Protocol: Implementation, Training and Evaluation

  • Liu et al. (2017), Anuar et al. (2013), and ZM approaches
  • Used results were reported by the authors in their original

papers

  • CNN-based approach
  • VGG16 network pretrained on ImageNet
  • Last fully connected layer was replaced by a 70 neurons layer

and a 51 neurons layer, each corresponding to the number of classes in each dataset

  • Hierarchy-of-Visual-Words
  • Median filter of 5 × 5 window size
  • Visual words codebook k parameter: k = 800 for MPEG-7

CE-1 and to k = 600 for MPEG-7 CE-2

  • Visual hierarchies codebook bandwidth parameter: h = 0.7 for

MPEG-7 CE-1 and to h = 1.7 for MPEG-7 CE-2

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 14 / 20

slide-15
SLIDE 15

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Results: Precision-Recall Analysis

  • MPEG-7 Core Experiment CE-Shape-1

0.0 0.2 0.4 0.6 0.8 1.0 Recall 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Precision MEPG-7 CE-1 HoVW Liu et al. Anuar et al. ZM

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 15 / 20

slide-16
SLIDE 16

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Results: Precision-Recall Analysis

  • MPEG-7 Region Shape Dataset CE-2

0.0 0.2 0.4 0.6 0.8 1.0 Recall 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Precision MEPG-7 CE-2 HoVW Liu et al. Anuar et al. ZM

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 16 / 20

slide-17
SLIDE 17

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Results: F1 Score Analysis

Approach MPEG-7 CE-1 MPEG-7 CE-2 HoVW 0.85 0.89 Liu et al. (2017) 0.87 0.87 Anuar et al. (2013) 0.78 0.82 ZM 0.77 0.79 CNN-based 0.88 0.81

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 17 / 20

slide-18
SLIDE 18

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Conclusion

  • The main contributions of our work are:
  • A new learning-based framework for the hierarchical

representation of elements in binary images

  • Its application on trademark image description and retrieval

from image databases

  • The proposed Hierarchy-of-Visual-Words uses two codebooks
  • The first codebook encodes basic shapes expected in the

images combining region-based and contour-based descriptors

  • The second codebook encodes both local and global

information of trademark images through hierarchical arrangements of their component shapes

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 18 / 20

slide-19
SLIDE 19

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Conclusion and Future Work

  • The hierarchy is defined as a tree where each node is related

to a component shape while tree levels describe the topological relationship of the components

  • Experimental results on well-known image databases show that
  • ur approach outperforms state-of-the-art techniques
  • Future Work: Incorporate optical character recognition and

principles from Gestalt psychology

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 19 / 20

slide-20
SLIDE 20

Introduction and Motivation Related Work Contributions The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work

Universidade Federal Fluminense

Hierarchy-of-Visual-Words: a Learning-based Approach for Trademark Image Retrieval

Vítor Lourenço, Gabriela Silva, Leandro A. F. Fernandes

vitorlourenco@id.uff.br

Instituto de Computação Universidade Federal Fluminense

October 31st, 2019

Vítor Lourenço October 31st, 2019 Hierarchy-of-Visual-Words 20 / 20