Bag-of-Visual-Ngrams for Histopathology Image Classification opez-Monroy, M.Sc. 1 A. Pastor L´ omez, Ph.D. 1 H. J. Escalante, Ph.D. 1 M. Montes-y-G´ A. Cruz-Roa, M.Sc. 2 alez, Ph.D. 2 F. A. Gonz´ November-2013 M´ exico LabTL, Computer Science Department, onica 1 ısica, ´ Instituto Nacional de Astrof´ Optica y Electr´ MindLab, Computing Systems and Industrial Engineering Department, National University of Colombia 2 9th International Seminar on Medical Information Processing and Analysis 1 / 24 �
ısica, ´ Instituto Nacional de Astrof´ Optica y Electr´ onica. Contents Introduction Representing images through visual n -grams Evaluating visual words, visual n -grams and language models Conclusions 9th International Seminar on Medical Information Processing and Analysis 2 / 24 �
1.- Introduction Introduction The amount of digital images available is constantly growing. Image Classification (IC) is important for the organization and analysis of visual information (e.g., for automated medical diagnosis from visual imagery). In this work, we focus in the challenging problem of automatic classification of the histophatology images. 9th International Seminar on Medical Information Processing and Analysis 3 / 24 �
1.- Introduction Introduction Histopathological images have several interesting particularities: heterogeneous rich visual content, high intra-class variability and complex mixtures of non-localized patterns. Automatic classification of histopathology images is according to tissue structures (healthy or pathological) that can be recognized by visual inspection of an expert pathologist. The classification is related to pathological lesions, morphological and architectural features, which encompass a complex mixture of visual patterns that allow to decide about the illness presence. [Cruz-Roa et al., 2011]. 9th International Seminar on Medical Information Processing and Analysis 4 / 24 �
1.- Introduction Introduction Example of histopathology images from skin biopsies with healthy (ephitelium) and pathological tissues (morpheaform basal-cell carcinoma), left and right respectively, used for basal-cell carcinoma diagnosis. 9th International Seminar on Medical Information Processing and Analysis 5 / 24 �
1.- Introduction Image representation One of the most used approaches is the Bag-of-Visual-Words (BoVW) formulation [Sivic and Zisserman., 2003]. BoVW representation is inspired in the bag-of-words (BoW) representation used in text classification and information retrieval. The idea of BoW is to represent a document by a numerical vector that indicates the presence/absence of words in a document. In computer vision tasks, a vocabulary of visual words is first generated and then images are represented by histograms that account for the occurrence of visual words in images. 9th International Seminar on Medical Information Processing and Analysis 6 / 24 �
1.- Introduction General view of BoVW 9th International Seminar on Medical Information Processing and Analysis 7 / 24 �
1.- Introduction Introduction BoVW has an important shortcoming: it ignores spatial relationships among visual words. Spatial context has proven to be helpful for boosting the performance in diverse computer vision tasks.[Galleguillos and Belongie, 2010]. We propose a natural extension to the BoVW formulation: the Bag-of-Visual- n grams (BoVN) In Text mining, n − grams are sequences of n words that allows to capture compound word-patterns, e.g., united-states , very-good , visual-words , etc.). We propose building codebooks of visual n − grams to capture visual patterns. 9th International Seminar on Medical Information Processing and Analysis 8 / 24 �
1.- Introduction Main contributions We introduce a different way to use of n -grams under the BoVW 1 formulation for IC; where n -grams are used as attributes for a classification model. We show that the BoVN can outperform the performance of the 2 BoVW approach for the classification of histopathology images. 9th International Seminar on Medical Information Processing and Analysis 9 / 24 �
2.- Representing images through visual n -grams General view of the proposed visual n -grams framework 9th International Seminar on Medical Information Processing and Analysis 10 / 24 �
2.- Representing images through visual n -grams Step 1: Detailed process to build the Visual Word codebook 9th International Seminar on Medical Information Processing and Analysis 11 / 24 �
2.- Representing images through visual n -grams Step 2: Representing images using the Visual Word codebook (a) Original image. (b) Visual Words representation. 9th International Seminar on Medical Information Processing and Analysis 12 / 24 �
2.- Representing images through visual n -grams Step 3: Building the Visual n -grams using a sliding window For the dark path (65) the extracted n -grams are: 65-12, 65-213, 65-546, 65-645, 65-654, 65-565, 65-444, 65-33. 9th International Seminar on Medical Information Processing and Analysis 13 / 24 �
3.- Evaluating visual words, visual n -grams and language models Description of the image dataset Histopathology image distribution for each category. Histopathology image positives negatives 1. basal-cell carcinoma 518 899 2. collagen 1238 179 3. epidermis 147 1270 4. hair follicle 118 1299 5. eccrine glands 126 1291 6. sebaceous glands 136 1281 7. inflammatory infiltrate 99 1318 9th International Seminar on Medical Information Processing and Analysis 14 / 24 �
3.- Evaluating visual words, visual n -grams and language models Evaluation of the standard Visual Words (Unigrams) Experiments using Visual Words (Unigrams) through two kinds of term weighting (TF and BIN) and two different size patches (8 and 16). Visual Words Config FM Bin-8 48.27 Bin-16 47.63 TF-8 58.59 TF-16 52.33 9th International Seminar on Medical Information Processing and Analysis 15 / 24 �
3.- Evaluating visual words, visual n -grams and language models Finding a suitable number of Visual Bi- grams Experiments using Bigrams to analyze the impact of dimensionality (F-measure). Frequency threshold Config 1.5K 2.5K 5K 7.5K 10K 1+2grams 63.73 64.31 64.03 62.24 61.63 9th International Seminar on Medical Information Processing and Analysis 16 / 24 �
3.- Evaluating visual words, visual n -grams and language models66 Visual Bigrams Interesting bigrams for basal-cell carcinoma 9th International Seminar on Medical Information Processing and Analysis 17 / 24 �
3.- Evaluating visual words, visual n -grams and language models Visual Words vs Visual Bigrams Experiments using sequences of Visual Words (Uni-Bi-grams) through two kinds of term weighting (TF and BIN) and two different size patches (8 and 16). Unigrams vs Uni+Bigrams Config F-Measure 1grams 1+2grams Bin-8 48.27 59.50 Bin-16 47.63 56.67 TF-8 58.9 64.31 TF-16 52.33 56.09 9th International Seminar on Medical Information Processing and Analysis 18 / 24 �
3.- Evaluating visual words, visual n -grams and language models The impact of sequence length for n -grams Experiments using sequences of Visual Words (from Unigrams to Tetragrams). Experiments with n-grams Config FM 1grams 58.59 1+2grams 64.31 1+2+3grams 62.69 1+2+3+4grams 61.34 9th International Seminar on Medical Information Processing and Analysis 19 / 24 �
3.- Evaluating visual words, visual n -grams and language models Detailed F-Measure by class Detailed experiments per class using Visual Words (Unigrams) versus sequences of Visual Words (Uni-Bi-grams). Detailed F-Measure by class (a) (b) (b-a) Class 1grams 1+2grams gain/loss 1 86.10 90.70 4.6 2 94.80 95.50 0.7 3 74.40 83.40 9.0 4 36.80 50.80 14.0 5 35.80 52.50 16.7 6 48.00 43.60 -4.4 7 34.20 33.70 -0.5 9th International Seminar on Medical Information Processing and Analysis 20 / 24 �
3.- Evaluating visual words, visual n -grams and language models Comparison with other typical approaches Experiments using sequences of Visual Words (Uni-Bi-grams) compared with a LMC. LMC vs Uni+Bigrams Config F-Measure LMC 1+2grams TF-8 53.0 64.31 TF-16 48.31 56.09 9th International Seminar on Medical Information Processing and Analysis 21 / 24 �
4.- Conclusions Conclusions We proposed an extension to the standard BoVW to extract sequential visual patterns and use them as attributes for a classification model. An histopathology image collection was used to extract n -grams using inspired ideas from NLP, such visual n -grams have proven good performance for this task. Visual n -grams as attributes have proven to be useful versus the traditional BoVW approach and LMC. Future work includes applying the visual n -grams based-representation to other computer vision tasks that use BoVW as representation. 9th International Seminar on Medical Information Processing and Analysis 22 / 24 �
Thank you, ... Questions? 9th International Seminar on Medical Information Processing and Analysis 23 / 24 �
Recommend
More recommend
