extended bag of words formalism for image classification
play

Extended Bag-of-Words Formalism for Image Classification Sandra - PowerPoint PPT Presentation

Jun 01, 2023 •476 likes •1.3k views

Extended Bag-of-Words Formalism for Image Classification Sandra Avila 1 , 2 (Cotutelle PhD Candidate), ujo 1 (Advisor), Matthieu Cord 2 (Advisor), Arnaldo de A. Ara Nicolas Thome 2 (Co-Advisor), Eduardo Valle 3 (Collaborator) 1 Federal

  1. Extended Bag-of-Words Formalism for Image Classification Sandra Avila 1 , 2 (Cotutelle PhD Candidate), ujo 1 (Advisor), Matthieu Cord 2 (Advisor), Arnaldo de A. Ara´ Nicolas Thome 2 (Co-Advisor), Eduardo Valle 3 (Collaborator) 1 Federal University of Minas Gerais, NPDI Lab – UFMG, Belo Horizonte, Brazil 2 Pierre and Marie Curie University, UPMC-Sorbonne Universities, LIP6, Paris, France 3 State University of Campinas, RECOD Lab, FEEC – UNICAMP, Campinas, Brazil Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 1 / 56

  2. Image Classification: Why do we care? Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 2 / 56

  3. Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 3 / 56

  4. Huge amount of image is available Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 4 / 56

  5. Why image classification is a hard problem? Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 5 / 56

  6. Many classes and concepts Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 6 / 56

  7. Viewpoint changes Illumination variations Occlusion Background clutter Inter-class similarity Intra-class diversity Much diversity in the data Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 6 / 56

  8. How do we classify images? Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 7 / 56

  9. Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 8 / 56

  10. Problem Statement Given an image dataset, how to represent their visual content information for a classification task? Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 9 / 56

  11. Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 10 / 56

  12. night scenes sunset scenes young people old people Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 11 / 56

  13. Bag-of-Visual-Words ( BoW ) [Sivic and Zisserman, 2003; Csurka et al., 2004] Slide credit: Ken Chatfield Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 12 / 56

  14. Low-level Visual Feature Extraction patch 1  l 1 , 1 . . . l 1 ,N  l 2 , 1 . . . l 2 ,N    . .  . .   . .   l M, 1 . . . l M,N patch M Local feature extraction Patch detection : interest points, dense sampling, . . . Feature extraction : SIFT [Lowe, 2004], SURF [Bay et al., 2008], . . . Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 13 / 56

  15. Visual Codebook Coding step Visual codebook learning : random, unsupervised (e.g., k -means, GMM), supervised [Perronnin et al., 2006; Goh et al., 2012], . . . Coding : hard-assignment, soft-assignment [van Gemert et al., 2008, 2010], sparse coding [Yang et al., 2009; Boureau et al., 2010], . . . Feature coding based on the vector difference : VLAD [J´ egou et al., 2010], SVC [Zhou et al., 2010], VLAT [Picard et al., 2011], . . . Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 14 / 56

  16. Pooling step Pooling : sum/average-pooling, max-pooling [Yang et al., 2009], . . . Spatial pooling : spatial pyramid matching [Lazebnik et al., 2006], [Jia et al., 2012], . . . Spatial Pyramid Matching Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 15 / 56

  17. Other Approaches Biologically-inspired Models Deep Learning Models [Fukushima and Miyake, 1982; LeCun et al., [Hinton and Salakhutdinov, 2006; 1990; Riesenhuber and Poggio, 1999; Serre Ranzato et al., 2007; Bengio, 2009] et al., 2007; Th´ eriault et al., 2012] Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 16 / 56

  18. BossaNova Representation Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 17 / 56

  19. Coding & Pooling Matrix Representation ... ... x 1 x j x N   α 1 , 1 . . . α 1 ,j . . . α 1 ,N c 1 . . . . . . . . . . . .     H = α m, 1 . . . α m,j . . . α m,N   c m   . . . . . . . .   . . . .   α M, 1 . . . α M,j . . . α M,N c M Notations : X = { x j } , j ∈ { 1 , . . . , N } : set of local descriptors (e.g., SIFT) C = { c m } , m ∈ { 1 , . . . , M } : visual codebook Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 18 / 56

  20. Coding & Pooling Matrix Representation ... ... x 1 x j x N   α 1 , 1 . . . α 1 ,j . . . α 1 ,N c 1 . . . . . . . . . . . .     H = α m, 1 . . . α m,j . . . α m,N   c m   . . . . . . . .   . . . .   α M, 1 . . . α M,j . . . α M,N c M ⇓ f : Coding � x j − c k � 2 Coding : x j → f ( x j ) = { α m,j } , α m,j = 1 iff m = arg min 2 k ∈{ 1 ,...,M } Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 18 / 56

  21. Coding & Pooling Matrix Representation ... ... x 1 x j x N   α 1 , 1 . . . α 1 ,j . . . α 1 ,N c 1 . . . . . . . . . . . .     H = α m, 1 . . . α m,j . . . α m,N ⇒ g : Pooling   c m   . . . . . . . .   . . . .   α M, 1 . . . α M,j . . . α M,N c M � x j − c k � 2 Coding : x j → f ( x j ) = { α m,j } , α m,j = 1 iff m = arg min 2 k ∈{ 1 ,...,M } N � Pooling : g ( { α j } ) = z : ∀ m, z m = α m,j j =1 Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 18 / 56

  22. Coding & Pooling Matrix Representation ... ... x 1 x j x N     z 1 α 1 , 1 . . . α 1 ,j . . . α 1 ,N c 1 . . . . . . . . . . . . . . .         z = H = α m, 1 . . . α m,j . . . α m,N z m     c m     . . . . . . . . . .     . . . . .     z M α M, 1 . . . α M,j . . . α M,N c M � x j − c k � 2 Coding : x j → f ( x j ) = { α m,j } , α m,j = 1 iff m = arg min 2 k ∈{ 1 ,...,M } N � Pooling : g ( { α j } ) = z : ∀ m, z m = α m,j j =1 BoW representation : z = [ z 1 , z 2 , · · · , z M ] T Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 18 / 56

  23. Early Ideas We pointed out the weakness in the standard pooling operation used in the BoW signature generation. Instead of averaging all the values from one row in the H matrix, we proposed to describe their distribution. BOSSA representation ( B ag O f S tatistical S ampling A nalysis) introduces our density function-based pooling strategy . Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 19 / 56

  24. Early Ideas We pointed out the weakness in the standard pooling operation used in the BoW signature generation. Instead of averaging all the values from one row in the H matrix, we proposed to describe their distribution. BOSSA representation ( B ag O f S tatistical S ampling A nalysis) introduces our density function-based pooling strategy . Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 19 / 56

  25. Early Ideas We pointed out the weakness in the standard pooling operation used in the BoW signature generation. Instead of averaging all the values from one row in the H matrix, we proposed to describe their distribution. BOSSA representation ( B ag O f S tatistical S ampling A nalysis) introduces our density function-based pooling strategy . Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 19 / 56

  26. Our Pooling Illustration Our Pooling BoW Pooling Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 20 / 56

  27. Our Pooling Formalism g : ❘ N ❘ B − → α m − → g ( α m ) = z m � b B ; b + 1 � �� z m,b = card x j | α m,j ∈ B b and b + 1 B ≥ α min ≤ α max m m B B denotes the number of bins of each histogram z m , and [ α min m ; α max ] limits the range of distances m Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 21 / 56

  28. BossaNova Representation Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 22 / 56

  29. BossaNova Representation ... ... x 1 x j x N   α 1 , 1 . . . α 1 ,j . . . α 1 ,N c 1 . . . . . . . . . . . .   exp − β m d 2 ( x j , c m )   α m,j = α m, 1 . . . α m,j . . . α m,N   c m � K   m ′ =1 exp − β m d 2 ( x j , c m ′ ) . . . . . . . .   . . . .   α M, 1 . . . α M,j . . . α M,N c M Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 22 / 56

  30. BossaNova Representation Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 22 / 56

  31. BossaNova Representation Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 22 / 56

  32. BossaNova Representation   z 1 , st 1 . . .     z m , st m     . .   .   z M , st M Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 22 / 56

  33. BossaNova Scheme Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 23 / 56

  34. BossaNova Scheme • SIFT descriptors on a dense spatial grid at multiple scales • Dimensionality reduction by applying PCA (128 → 64) Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 23 / 56

  35. BossaNova Scheme • k -means algorithm Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 23 / 56

  36. BossaNova Scheme Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 23 / 56

  37. BossaNova Scheme • SVM classifiers are applied by using a nonlinear Gauss– ℓ 2 kernel Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 23 / 56

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Bag of Words Model Overview of todays lecture Bag-of-words. K-means clustering.

Bag of Words Model Overview of todays lecture Bag-of-words. K-means clustering.

Bag of Words Model Overview of todays lecture Bag-of-words. K-means clustering. Classification. K nearest neighbors. Support vector machine. Image Classification Image Classification: Problem Data-driven approach

1.52k views • 104 slides
Classification Image Classification Set of predefined categories [eg: table, apple, dog, giraffe]

Classification Image Classification Set of predefined categories [eg: table, apple, dog, giraffe]

Day 1 Lecture 2 Classification Image Classification Set of predefined categories [eg: table, apple, dog, giraffe] Binary classification [1, 0] DOG 2 Image Classification 3 Image Classification pipeline Dog 4 Slide credit: Jose M lvarez

339 views • 23 slides
From image classification to object detection Image classification Object detection Image source

From image classification to object detection Image classification Object detection Image source

From image classification to object detection Image classification Object detection Image source Slides from L. Lazebnik What are the challenges of object detection? Images may contain more than one class, multiple instances from the same

1.11k views • 57 slides
Image Analysis System Example: Image Classification System pre feature feature segmentation

Image Analysis System Example: Image Classification System pre feature feature segmentation

Image Analysis System Example: Image Classification System pre feature feature segmentation processing description selection post feature classification classification extraction 1 IIP Image Segmentation Grey-level thresholding

422 views • 15 slides
for Image Classification Qilong Wang ( ) Dalian University of Technology

for Image Classification Qilong Wang ( ) Dalian University of Technology

Codebook-free Single Gaussian for Image Classification Qilong Wang ( ) Dalian University of Technology http://ice.dlut.edu.cn/PeihuaLi/ Image Model for Classification Scene Object Image Model Fine-grained Texture Face

539 views • 43 slides
Object Detection JunYoung Gwak 1 Motivation Image classification Input: Image

Object Detection JunYoung Gwak 1 Motivation Image classification Input: Image

Object Detection JunYoung Gwak 1 Motivation Image classification Input: Image Output: object class 2 Motivation Limitation of classification Multiple classes Location i.e. Object classification assumes Single

1.07k views • 50 slides
NO WORDS.... please use this image (turn horizontal?):

NO WORDS.... please use this image (turn horizontal?):

NO WORDS.... please use this image (turn horizontal?): https://unsplash.com/photos/4rDCa5hBlCs NO WORDS.... Please use this image https://unsplash.com/photos/ePvBOHsAA54 NO WORDS.... PLease use this image

472 views • 26 slides
Tuning the Performance of Convolutional Neural Network for Image Classification on GPU Agenda

Tuning the Performance of Convolutional Neural Network for Image Classification on GPU Agenda

Tuning the Performance of Convolutional Neural Network for Image Classification on GPU Agenda Adoptions of Image classification or image recognition at Alibaba Easy ways to improve performance of Caffe Further performance

182 views • 15 slides
Natural Language Processing Classification I Dan Klein UC Berkeley 1 2 Classification

Natural Language Processing Classification I Dan Klein UC Berkeley 1 2 Classification

Natural Language Processing Classification I Dan Klein UC Berkeley 1 2 Classification Classification Automatically make a decision about inputs Example: document category Example: image of digit digit Example: image

757 views • 50 slides
1 Image Classification BVM 2018 Tutorial: Advanced Deep Learning Methods Jakob Wasserthal,

1 Image Classification BVM 2018 Tutorial: Advanced Deep Learning Methods Jakob Wasserthal,

1 Image Classification BVM 2018 Tutorial: Advanced Deep Learning Methods Jakob Wasserthal, Division of Medical Image Computing Author Division Classification of skin cancer 02.11.16 | vs Esteva et al., Dermatologist-level classification of

1.08k views • 48 slides
for Large-Scale Image Classification Karn Simonyan, Andrea Vedaldi, Andrew Zisserman Visual

for Large-Scale Image Classification Karn Simonyan, Andrea Vedaldi, Andrew Zisserman Visual

Deep Fisher Networks for Large-Scale Image Classification Karn Simonyan, Andrea Vedaldi, Andrew Zisserman Visual Geometry Group, University of Oxford Deep learning achieves excellent performance in image classification. Do hand-crafted image

159 views • 4 slides
Gaze Embeddings for Zero-Shot Image Classification Nour Karessli Zeynep Akata Bernt

Gaze Embeddings for Zero-Shot Image Classification Nour Karessli Zeynep Akata Bernt

Gaze Embeddings for Zero-Shot Image Classification Nour Karessli Zeynep Akata Bernt Schiele Andreas Bulling Presentation by Hsin-Ping Huang and Shubham Sharma Introduction Attributes Standard image classification models fail

758 views • 28 slides
Prospects with Extended RPA Theories P. Papakonstantinou Institut f ur Kernphysik,

Prospects with Extended RPA Theories P. Papakonstantinou Institut f ur Kernphysik,

Prospects with Extended RPA Theories P. Papakonstantinou Institut f ur Kernphysik, T.U.Darmstadt 1 Overview Introduction From RPA to Second RPA - Formalism and technicalities - Results on Giant Resonances - Issues to be considered

412 views • 40 slides
Metric Learning Applied for Automatic Large Image Classification Supervisors TOON CALDERS (PhD) /

Metric Learning Applied for Automatic Large Image Classification Supervisors TOON CALDERS (PhD) /

September, 2014 UPC Metric Learning Applied for Automatic Large Image Classification Supervisors TOON CALDERS (PhD) / ULB SAHILU WENDESON / IT4BI SALIM JOUILI (PhD) /EuraNova Image Database Classification How? Using K-Nearest Neighbor (kNN)

415 views • 40 slides
Hybrid Deep Learning Topology for Image Classification Petru Radu petru.radu@ness.com 27 th

Hybrid Deep Learning Topology for Image Classification Petru Radu petru.radu@ness.com 27 th

15.07.2019 Hybrid Deep Learning Topology for Image Classification Petru Radu petru.radu@ness.com 27 th Summer School on Image Processing, Timisoara 2019 SSIP 2019 Introduction Classical Neural Networks employed in image classification have a

513 views • 33 slides
Statistical NLP Spring 2011 Lecture 11: Classification Dan Klein UC Berkeley Classification

Statistical NLP Spring 2011 Lecture 11: Classification Dan Klein UC Berkeley Classification

Statistical NLP Spring 2011 Lecture 11: Classification Dan Klein UC Berkeley Classification Automatically make a decision about inputs Example: document category Example: image of digit digit Example: image of object

609 views • 48 slides
Consorzio COMETA FESR Visualization Element: towards the definition of a new Grid service

Consorzio COMETA FESR Visualization Element: towards the definition of a new Grid service

Consorzio COMETA FESR Visualization Element: towards the definition of a new Grid service Giuseppe ANDRONICO (1) , Roberto BARBERA (1)(2) , Andrea FORNAIA (1) , Marcello IACONO MANNO (3) and

614 views • 18 slides
New developments on rational RBF Stefano De Marchi Department of Mathematics Tullio

New developments on rational RBF Stefano De Marchi Department of Mathematics Tullio

New developments on rational RBF Stefano De Marchi Department of Mathematics Tullio Levi-Civita University of Padova Montecatini, 14th February 2018 Outline From RBF to Rational RBF (RRBF) 1 2 Eigen-rational interpolant 3 Numerical

739 views • 45 slides
SMOOTH SOLUTIONS IN VASILIEV THEORY Andrea Campoleoni Universit Libre de Bruxelles &

SMOOTH SOLUTIONS IN VASILIEV THEORY Andrea Campoleoni Universit Libre de Bruxelles &

SMOOTH SOLUTIONS IN VASILIEV THEORY Andrea Campoleoni Universit Libre de Bruxelles & International Solvay Institutes A.C., T. Prochzka, J. Raeymaekers, 1303.0880 Workshop on Higher Spins, Strings and Duality, Galileo

458 views • 35 slides
TRANSITIONING TO A CULTURE OF EVIDENCE: SUSTAINABLE ASSESSMENT PRACTICES Roundt dtable le D

TRANSITIONING TO A CULTURE OF EVIDENCE: SUSTAINABLE ASSESSMENT PRACTICES Roundt dtable le D

TRANSITIONING TO A CULTURE OF EVIDENCE: SUSTAINABLE ASSESSMENT PRACTICES Roundt dtable le D Discussion O Orga ganized b d by Asse ssess ssment S Subc bcommittee 20 2018 8 UW UWSP T Teaching Conference Janua nuary 1 y 18, 2

615 views • 30 slides
Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8.

622 views • 28 slides
Registrant List Page 1 of 5 UCSF OCME Name City, State Ahmad Borhaan MD Redlands, CA 1 2

Registrant List Page 1 of 5 UCSF OCME Name City, State Ahmad Borhaan MD Redlands, CA 1 2

University of California San Francisco 2/25/2014 MOC14001: Developmental Disabilities Registrant List Page 1 of 5 UCSF OCME Name City, State Ahmad Borhaan MD Redlands, CA 1 2 Allen Heather BSN Aptos, CA Amosslee Laurel Oakland,

563 views • 5 slides
Lecture 1: Introduction to Pattern Recognition Dr. Chengjiang Long Computer Vision Researcher at

Lecture 1: Introduction to Pattern Recognition Dr. Chengjiang Long Computer Vision Researcher at

Lecture 1: Introduction to Pattern Recognition Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu Self-introduction 2 C. Long Lecture 1 May 6, 2018 Outline Course Information

967 views • 52 slides
Traffic Predictions Supporting General Aviation Carlo Lancia, D.

Traffic Predictions Supporting General Aviation Carlo Lancia, D.

Traffic Predictions Supporting General Aviation Carlo Lancia, D. Taurino, G. Frau, J. Verstraeten, and C. Le Tallec Motivation Help GA pilots

394 views • 22 slides

More recommend