Hierarchy-of-Visual-Words: a Learning-based Approach for Trademark - PowerPoint PPT Presentation

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work Hierarchy-of-Visual-Words: a Learning-based Approach for Trademark Image Retrieval Vítor Lourenço , Gabriela Silva, Leandro A. F. Fernandes vitorlourenco@id.uff.br Instituto de Computação Universidade Federal Fluminense October 31st, 2019 Vítor Lourenço October 31st, 2019 1 / 20 Hierarchy-of-Visual-Words

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work Introduction and Motivation • Trademarks carry the identification, the reputation, and the quality meanings of the associated product or service • In the year of 2018, about 200000 trademarks were deposited in the Brazilian Intellectual Property Agency • Law offices hires hundreds of people to manually analyze each new trademark image • Not scalable • Error-prune • Costly Vítor Lourenço October 31st, 2019 2 / 20 Hierarchy-of-Visual-Words

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work Related Work • Content-based trademark image retrieval systems Wei et al. (2009), Qi et al. (2010), Anuar et al. (2013), Liu et al. (2017) • Pros: Geometrically partitions the trademark image and uses global and local descriptors separately • Cons: Does not consider topological relationship between trademark image’s objects • Hierarchical representation of multiobjects images Alajlan et al. (2006 e 2008), Chen e Weng (2017) • Pros: Encodes topological relationship between image’s objects within a hierarchical structure • Cons: Measures similarity between hierarchies on inference time • Bag-based models Silva et al. (2013, 2014, 2017) • Pros: Provides bags of graph-like data structures • Cons: Uses texture-based representation and does not encode topological relationship Vítor Lourenço October 31st, 2019 3 / 20 Hierarchy-of-Visual-Words

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work Contributions • A new learning-based framework for the hierarchical representation of elements in binary images • Its application on trademark image description and retrieval from image databases Vítor Lourenço October 31st, 2019 4 / 20 Hierarchy-of-Visual-Words

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work The Hierarchy-of-Visual-Words Framework < D > T < B > R A < TA > < B > < D > < A > I N . . . < C > < C > < C > . . . I . . . < C > < TB > N Visual Words Codebook Training Trademark Image Shape’s Feature Hierarchical Relationship Visual Hierarchies Codebook G Shape Extraction Learning Trademark Images Binarization Extraction Encoding Learning (c) (e) (a) (b) (d) (f) (g) E V A L < B > U A < A > T I Query Trademark Trademark Image Shape’s Feature Hierarchical Relationship O Image Binarization Shape Extraction Extraction Encoding Similar Images Retrieval N (h) (i) (j) (k) (l) (m) Vítor Lourenço October 31st, 2019 5 / 20 Hierarchy-of-Visual-Words

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work Hierarchy-of-Visual-Words: Training Stage Trademark Image Shape's Feature Training Shape Extraction Binarization Extraction Trademark Images . . . . . . < D > < B > < TA > < B > < D > < A > < C > < C > < C > < TB > < C > . . . Visual Words Visual Hierarchies Hierarchical Codebook Learning Codebook Relationship Encoding Vítor Lourenço October 31st, 2019 6 / 20 Hierarchy-of-Visual-Words

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work Trademark Image Binarization and Shape Extraction < D > < B > < TA > < B > < D > < A > . . . < C > < C > < C > . . . . . . < C > < TB > • Digital trademark images converted into binary images • Convert the color images to grayscales • Apply median filter to reduce impulsive noise • Apply bilateral filter to remove texture without losing overall shapes • Apply Otsu’s method on the textureless grayscale image to obtain the final binary image • Shape extraction regarding objects and holes Vítor Lourenço October 31st, 2019 7 / 20 Hierarchy-of-Visual-Words

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work Shape’s Feature Extraction and Visual Words Codebook Learning < D > < B > < TA > < B > < D > < A > . . . < C > < C > < C > . . . . . . < C > < TB > • Feature extraction descriptors • Zernike moments • Circularity • Average bending energy • Eccentricity • Convexity • Visual codebook learning • k -means clustering Vítor Lourenço October 31st, 2019 8 / 20 Hierarchy-of-Visual-Words

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work Hierarchical Relationship Encoding < D > < B > < TA > < B > < D > < A > . . . < C > < C > < C > . . . . . . < C > < TB > • Relationship encoding • Shape inclusion • Shape exclusion • Visual hierarchy Vítor Lourenço October 31st, 2019 9 / 20 Hierarchy-of-Visual-Words

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work Visual Hierarchies Codebook Learning < D > < B > < TA > < B > < D > < A > . . . < C > < C > < C > . . . . . . < C > < TB > • Hierarchies dissimilarity matrix • Rename cost δ r ( n a , n b ) = dist E ( λ a , λ b ) • Insert and remove costs m m 2 � � δ x ( n ) = α dist E ( λ i , λ j ) , m ( m − 1 ) i = 1 j = i + 1 α = min { log − 1 L , log − 1 D} 2 2 • Mean-shift clustering with RBF kernel Vítor Lourenço October 31st, 2019 10 / 20 • Dissimilarity matrix Hierarchy-of-Visual-Words

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work Hierarchy-of-Visual-Words: Evaluation Stage Query Trademark Trademark Image Shape Extraction Shape’s Feature Image Binarization Extraction Learned Learned < B > Visual Visual Words Words < A > Codebook Codebook Similar Images Hierarchical Retrieval Relationship Encoding Vítor Lourenço October 31st, 2019 11 / 20 Hierarchy-of-Visual-Words

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work Similar Images Retrieval � � is a mean � shift cluster similar non-similar is a mean � shift cluster Vítor Lourenço October 31st, 2019 12 / 20 Hierarchy-of-Visual-Words

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work Experiments Protocol: Datasets and Metrics • MPEG-7 Core Experiment CE-Shape-1 • 1 , 400 binary images • 70 classes • 20 similar images per class • MPEG-7 Region Shape Dataset CE-2 • 871 binary images • 51 classes • 11 to 21 similar images per class • 2 , 750 images that do not belong to any category * • Used Metrics • Precision-Recall Curve • F 1 Score Vítor Lourenço October 31st, 2019 13 / 20 Hierarchy-of-Visual-Words

Introduction and Motivation Related Work Contributions Universidade Federal Fluminense The Hierarchy-of-Visual-Words Framework Experiments and Results Conclusion and Future Work Experiments Protocol: Implementation, Training and Evaluation • Liu et al. (2017), Anuar et al. (2013), and ZM approaches • Used results were reported by the authors in their original papers • CNN-based approach • VGG16 network pretrained on ImageNet • Last fully connected layer was replaced by a 70 neurons layer and a 51 neurons layer, each corresponding to the number of classes in each dataset • Hierarchy-of-Visual-Words • Median filter of 5 × 5 window size • Visual words codebook k parameter: k = 800 for MPEG-7 CE-1 and to k = 600 for MPEG-7 CE-2 • Visual hierarchies codebook bandwidth parameter: h = 0 . 7 for MPEG-7 CE-1 and to h = 1 . 7 for MPEG-7 CE-2 Vítor Lourenço October 31st, 2019 14 / 20 Hierarchy-of-Visual-Words