lecture 21 unsupervised learning and clustering algorithms
play

Lecture 21: Unsupervised Learning and Clustering Algorithms Dr. - PowerPoint PPT Presentation

Sep 19, 2023 •436 likes •1.08k views

Lecture 21: Unsupervised Learning and Clustering Algorithms Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu Recap Previous Lecture 2 C. Long Lecture 21 April 20, 2018 Outline

  1. Lecture 21: Unsupervised Learning and Clustering Algorithms Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu

  2. Recap Previous Lecture 2 C. Long Lecture 21 April 20, 2018

  3. Outline Introduce Unsupervised Learning and Clustering • K-means Algorithm • Hierarchy Clustering • Applications • 3 C. Long Lecture 21 April 20, 2018

  4. Outline Introduce Unsupervised Learning and Clustering • K-means Algorithm • Hierarchy Clustering • Applications • 4 C. Long Lecture 21 April 20, 2018

  5. Unsupervised learning and clustering All data is unlabeled and All data is labeled and the the algorithms learn the algorithms learn to predict inherent structure from the output from the input the input data. data. 5 C. Long Lecture 21 April 20, 2018

  6. Unsupervised learning and clustering G o a l : t o m o d e l t h e underlying structure or distribution in the input data. 6 C. Long Lecture 21 April 20, 2018

  7. What is clustering? 7 C. Long Lecture 21 April 20, 2018

  8. What is clustering for? E.g.1 : group people of similiar size together to make S,M,L T-shirts. E.g.2 : segment customers to do targeted marketing. E.g.3 : orgnize documents to produce a topic hierarchy. 8 C. Long Lecture 21 April 20, 2018

  9. What is clustering for? 9 C. Long Lecture 21 April 20, 2018

  10. Clustering evaluation Clustering is hard to evaluate. In most applications, expert judgements are still the key. 10 C. Long Lecture 21 April 20, 2018

  11. Data Clustering - Formal Definition Given a set of N unlabeled examples D = x 1 , x 2 , ..., x N in • a d-dimensional feature space, D is partitioned into a number of disjoint subsets Dj’s: A partition is denoted by: • and the problem of data clustering is thus formulated as where f(·) is formulated according to a given criterion. 11 C. Long Lecture 21 April 20, 2018

  12. Outline Introduce Unsupervised Learning and Clustering • K-means Algorithm • Hierarchy Clustering • Applications • 12 C. Long Lecture 21 April 20, 2018

  13. K-means 13 C. Long Lecture 21 April 20, 2018

  14. K-means: an example 14 C. Long Lecture 21 April 20, 2018

  15. K-means: an example 15 C. Long Lecture 21 April 20, 2018

  16. K-means: an example 1-st iteration 16 C. Long Lecture 21 April 20, 2018

  17. K-means: an example 1-st iteration 17 C. Long Lecture 21 April 20, 2018

  18. K-means: an example 2-nd iteration 18 C. Long Lecture 21 April 20, 2018

  19. K-means: an example 2-nd iteration 19 C. Long Lecture 21 April 20, 2018

  20. K-means: an example 3-rd iteration 20 C. Long Lecture 21 April 20, 2018

  21. K-means: an example 3-rd iteration 21 C. Long Lecture 21 April 20, 2018

  22. K-means: an example No changes : Done 22 C. Long Lecture 21 April 20, 2018

  23. K-means Iterate :  Assign / cluster each example to closest center iterate over each point : - get distance to each cluster center - assign to closest center ( hard cluster )  Recalculate centers as the mean of the points in a cluster How do we do this ? 23 C. Long Lecture 21 April 20, 2018

  24. K-means Iterate :  Assign / cluster each example to closest center iterate over each point : - get distance to each cluster center - assign to closest center ( hard cluster )  Recalculate centers as the mean of the points in a cluster What distance measure should we use ? 24 C. Long Lecture 21 April 20, 2018

  25. K-means Iterate :  Assign / cluster each example to closest center  Recalculate centers as the mean of the points in a cluster good for spatial data What distance measure should we use ? 25 C. Long Lecture 21 April 20, 2018

  26. Euclidean Distance Euclidean Distance • n 2 = - dist ( p q ) å k k = k 1 Where n is the number of dimensions (attributes) and p k and q k are, respectively, the k th attributes (components) or data objects p and q . Standardization is necessary , if scales differ . • 26 C. Long Lecture 21 April 20, 2018

  27. Minkowski Distance Minkowski Distance is a generalization of Euclidean • Distance Where r is a parameter, n is the number of dimensions (attributes) and p k and q k are, respectively, the k-th attributes (components) or data objects p and q. 27 C. Long Lecture 21 April 20, 2018

  28. Euclidean Distance 28 C. Long Lecture 21 April 20, 2018

  29. More about Euclidean distance 29 C. Long Lecture 21 April 20, 2018

  30. Manhattan Distance Manhattan distance represents distance that is • measured along directions that are parallel to the x and y axes Manhattan distance between two n -dimensional • vectors x=(x 1, x 2, … , x n ) and y=(y 1, y 2, … , y n ) is: = - + - + + - d ( x , y ) x y x y  x y M 1 1 2 2 n n n å = - x y i i = i 1 Where represents the absolute value of the difference betweeen x i and y i 30 C. Long Lecture 21 April 20, 2018

  31. Minkowski Distance: Examples 31 C. Long Lecture 21 April 20, 2018

  32. Minkowski Distance 32 C. Long Lecture 21 April 20, 2018

  33. K-means Iterate : • Assign - cluster each example to closest center • Recalculate centers as the mean of the points in a cluster Where are the cluster centers ? 33 C. Long Lecture 21 April 20, 2018

  34. K-means Iterate : • Assign - cluster each example to closest center • Recalculate centers as the mean of the points in a cluster How do we calculate these ? 34 C. Long Lecture 21 April 20, 2018

  35. K-means Iterate : • Assign - cluster each example to closest center • Recalculate centers as the mean of the points in a cluster 35 C. Long Lecture 21 April 20, 2018

  36. Pros and cons of K-means Weakneses: The user needs to specify the value of K.  Applicable only when mean is defined.  The algorithm is sensitive to the initial seeds.  The algorithm is sensitive to outliers.   Outliers are data points that are very far away from other data points.  Outliers could be errors in the data recording or some special data points with very different values. 36 C. Long Lecture 21 April 20, 2018

  37. Failure case 37 C. Long Lecture 21 April 20, 2018

  38. Sensitive to initial seeds 38 C. Long Lecture 21 April 20, 2018

  39. Sensitive to outliers outlier outlier 39 C. Long Lecture 21 April 20, 2018

  40. Application to visual object recognition: Bag of Words 40 C. Long Lecture 21 April 20, 2018

  41. Application to visual object recognition: Bag of Words Vector quantize descriptors from a set of training images using k - means Image representation: a normalized histogram of visual words. 41 C. Long Lecture 21 April 20, 2018

  42. Application to visual object recognition: Bag of Words The same visual word 42 C. Long Lecture 21 April 20, 2018

  43. Application to visual object recognition: Bag of Words 43 C. Long Lecture 21 April 20, 2018

  44. Summary 44 C. Long Lecture 21 April 20, 2018

  45. Outline Introduce Unsupervised Learning and Clustering • K-means Algorithm • Hierarchy Clustering • Applications • 45 C. Long Lecture 21 April 20, 2018

  46. Hierarchical Clustering Up to now , considered “flat” clustering • For some data, hierarchical clustering is more • appropriate than “flat” clustering Hierarchical clustering • 46 C. Long Lecture 21 April 20, 2018

  47. Hierarchical Clustering: Biological Taxonomy 47 C. Long Lecture 21 April 20, 2018

  48. Hierarchical Clustering: Dendogram Prefered way to represent a hierarchical clustering is a • dendogram _x0001_ Binary tree _x0001_ Level k corresponds to partitioning with n-k+1 clusters _x0001_ if need k clusters, take clustering from level n-k+1 _x0001_ If samples are in the same cluster at level k, they stay in the same cluster at higher levels _x0001_ dendogram typically shows the similarity of grouped clusters 48 C. Long Lecture 21 April 20, 2018

  49. Hierarchical Clustering: Venn Diagram Can also use Venn diagram to show hierarchical • clustering , but similarity is not represented quantitatively 49 C. Long Lecture 21 April 20, 2018

  50. Hierarchical Clustering Algorithms for hierarchical clustering can be divided • into two types: 1. Agglomerative ( bottom up ) procedures • _x0001_ Start with n singleton clusters _x0001_ Form hierarchy by merging most similar clusters 2. Divisive ( top bottom ) procedures • _x0001_ Start with all samples in one cluster _x0001_ Form hierarchy by splitting the “worst” clusters 50 C. Long Lecture 21 April 20, 2018

  51. Divisive Hierarchical Clustering Any “flat” algorithm which produces a fixed number • of clusters can be used set c = 2 51 C. Long Lecture 21 April 20, 2018

  52. Agglomerative Hierarchical Clustering Initialize with each example in singleton cluster while • there is more than 1 cluster 1. find 2 nearest clusters • 2. merge them • Four common ways to measure cluster distance • 52 C. Long Lecture 21 April 20, 2018

  53. Single Linkage or Nearest Neighbor Agglomerative clustering with minimum distance • generates minimum spanning tree • encourages growth of elongated clusters • disadvantage : very sensitive to noise • 53 C. Long Lecture 21 April 20, 2018

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Lecture 10 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels

Lecture 10 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels

Unsupervised Machine Learning and Data Mining DS 5230 / DS 4420 - Fall 2018 Lecture 10 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels for training) Group data into similar classes that Maximize

1.29k views • 69 slides
Lecture 11 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels

Lecture 11 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels

Unsupervised Machine Learning and Data Mining DS 5230 / DS 4420 - Fall 2018 Lecture 11 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels for training) Group data into similar classes that Maximize

833 views • 26 slides
UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised learning: X - y pairs, f(x) function approximation Unsupervised learning: only X, no y Exploring the space of X measurements,

440 views • 14 slides
3D Deep Clustering a clustering framework for unsupervised learning of 3D object feature

3D Deep Clustering a clustering framework for unsupervised learning of 3D object feature

3D Deep Clustering a clustering framework for unsupervised learning of 3D object feature descriptors LAMBDA Workshop on Retrieval and Shape analysis Alexandros Georgiou M.Sc. in Algorithms, Logic and Discrete Mathematics 1/12 Problem

570 views • 53 slides
Machine Learning Lecture 6 Unsupervised Learning with a bit of Supervised Learning Clustering

Machine Learning Lecture 6 Unsupervised Learning with a bit of Supervised Learning Clustering

Machine Learning Lecture 6 Unsupervised Learning with a bit of Supervised Learning Clustering and nearest neighbour classifiers. Justin Pearson 1 2020 1 http://user.it.uu.se/~justin/Teaching/MachineLearning/index.html 1 / 27 Supervised

813 views • 27 slides
Unsupervised Learning and Clustering l In unsupervised learning you are given a data set with no

Unsupervised Learning and Clustering l In unsupervised learning you are given a data set with no

Unsupervised Learning and Clustering l In unsupervised learning you are given a data set with no output classifications (labels) l Clustering is an important type of unsupervised learning PCA was another type of unsupervised learning l The

858 views • 38 slides
Learning From Data Lecture 19 A Peek At Unsupervised Learning k -Means Clustering Probability

Learning From Data Lecture 19 A Peek At Unsupervised Learning k -Means Clustering Probability

Learning From Data Lecture 19 A Peek At Unsupervised Learning k -Means Clustering Probability Density Estimation Gaussian Mixture Models M. Magdon-Ismail CSCI 4100/6100 recap: Radial Basis Functions Nonparametric RBF Parametric k -RBF-Network

253 views • 23 slides
Hierarchical Clustering 4-4-16 Hierarchical clustering: the setting Unsupervised learning

Hierarchical Clustering 4-4-16 Hierarchical clustering: the setting Unsupervised learning

Hierarchical Clustering 4-4-16 Hierarchical clustering: the setting Unsupervised learning no labels/output, only x/input Clustering Group similar points together Machine learning taxonomy Supervised Semi-Supervised Unsupervised

366 views • 15 slides
Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of

Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of

UNSUPERVISED LEARNING IN R Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of clustering finding groups of homogeneous items Next up, dimensionality reduction Find structure in features

904 views • 29 slides
Clustering Lecture 8 David Sontag New York University

Clustering Lecture 8 David Sontag New York University

Clustering Lecture 8 David Sontag New York University Slides adapted from Luke Zettlemoyer, Vibhav Gogate, Carlos Guestrin, Andrew Moore, Dan Klein Clustering Clustering: Unsupervised learning

1.22k views • 56 slides
Clustering Lecture 14 David Sontag New York University

Clustering Lecture 14 David Sontag New York University

Clustering Lecture 14 David Sontag New York University Slides adapted from Luke Zettlemoyer, Vibhav Gogate, Carlos Guestrin, Andrew Moore, Dan Klein Clustering Clustering: Unsupervised learning

559 views • 33 slides
Unsupervised Learning and Clustering Owen Roberts, Zach Busser, Ganesh Sugunan Hierarchical

Unsupervised Learning and Clustering Owen Roberts, Zach Busser, Ganesh Sugunan Hierarchical

Unsupervised Learning and Clustering Owen Roberts, Zach Busser, Ganesh Sugunan Hierarchical Clustering Hierarchical Clustering Hierarchical Representations Hierarchical Algorithms Agglomeration "Nearest" Clusters Complexity

769 views • 49 slides
Clustering ! Unsupervised learning: learning without a teacher ! Only features and no labels

Clustering ! Unsupervised learning: learning without a teacher ! Only features and no labels

So far in the course Supervised learning: learning with a teacher ! You had training data which was (feature, label) pairs and the goal was to learn a mapping from features to labels Clustering ! Unsupervised learning: learning without a

164 views • 12 slides
9.54 Class 13 Unsupervised learning Clustering Shimon Ullman + Tomaso Poggio Danny Harari +

9.54 Class 13 Unsupervised learning Clustering Shimon Ullman + Tomaso Poggio Danny Harari +

9.54 Class 13 Unsupervised learning Clustering Shimon Ullman + Tomaso Poggio Danny Harari + Daneil Zysman + Darren Seibert Outline Introduction to clustering K-means Bag of words (dictionary learning) Hierarchical clustering

895 views • 54 slides
UNSUPERVISED LEARNING AND CLUSTERING Jeff Robble, Brian Renzenbrink, Doug Roberts Unsupervised

UNSUPERVISED LEARNING AND CLUSTERING Jeff Robble, Brian Renzenbrink, Doug Roberts Unsupervised

UNSUPERVISED LEARNING AND CLUSTERING Jeff Robble, Brian Renzenbrink, Doug Roberts Unsupervised Procedures A procedure that uses unlabeled data in its classification process. Why would we use these? Collecting and labeling large data sets can

544 views • 28 slides
Chapter 7: Clustering (Unsupervised Data Organization) 7.1 Hierarchical Clustering 7.2 Flat

Chapter 7: Clustering (Unsupervised Data Organization) 7.1 Hierarchical Clustering 7.2 Flat

Chapter 7: Clustering (Unsupervised Data Organization) 7.1 Hierarchical Clustering 7.2 Flat Clustering 7.3 Embedding into Vector Space for Visualization 7.4 Applications Clustering: unsupervised grouping (partitioning) of objects into

763 views • 49 slides
Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on

Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on

Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on supervised learning methods such as regression and classification. In that setting we observe both a set of features X 1 , X 2 , . . . , X p for each

721 views • 60 slides
Hierarchical and Ensemble Clustering Ke Chen Reading: [7.8-7.10, EA], [25.5, KPM], [Fred &

Hierarchical and Ensemble Clustering Ke Chen Reading: [7.8-7.10, EA], [25.5, KPM], [Fred &

Hierarchical and Ensemble Clustering Ke Chen Reading: [7.8-7.10, EA], [25.5, KPM], [Fred & Jain, 2005] COMP24111 Machine Learning Outline Introduction Cluster Distance Measures Agglomerative Algorithm Example and

268 views • 26 slides
Sys System tem Eva Evaluat luation ion Darren Urada, Ph.D. and Howard Padwa, Ph.D., and

Sys System tem Eva Evaluat luation ion Darren Urada, Ph.D. and Howard Padwa, Ph.D., and

Ca Cali liforni fornia a Hu Hub and b and Sp Spok oke e Sys System tem Eva Evaluat luation ion Darren Urada, Ph.D. and Howard Padwa, Ph.D., and Vandana UCLA Integrated Substance Abuse Programs October 25, 2017 Statewide Integrated

561 views • 24 slides
Cancer Incidence in Nigeria: a report from population-based Cancer Registries Jedy-Agba E,

Cancer Incidence in Nigeria: a report from population-based Cancer Registries Jedy-Agba E,

Institute of Human Virology, Nigeria Cancer Incidence in Nigeria: a report from population-based Cancer Registries Jedy-Agba E, Curado MP, Ogunbiyi O, Oga E, Igbinoba F, Koechlin A, Dakum P, Blattner W, and Adebamowo CA EP-746 Systems in

364 views • 4 slides
Lecture 22: Clustering Distance measures K-Means Aykut Erdem May 2016 Hacettepe

Lecture 22: Clustering Distance measures K-Means Aykut Erdem May 2016 Hacettepe

Lecture 22: Clustering Distance measures K-Means Aykut Erdem May 2016 Hacettepe University Last time Boosting Idea: given a weak learner, run it multiple times on (reweighted) training data, then let the learned classifiers vote

796 views • 45 slides
Data Mining and Machine Learning: Fundamental Concepts and Algorithms dataminingbook.info

Data Mining and Machine Learning: Fundamental Concepts and Algorithms dataminingbook.info

Data Mining and Machine Learning: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA 2 Department of Computer Science

631 views • 14 slides
RECSM Summer School: Machine Learning for Social Sciences Session 3.4: Hierarchical Clustering

RECSM Summer School: Machine Learning for Social Sciences Session 3.4: Hierarchical Clustering

RECSM Summer School: Machine Learning for Social Sciences Session 3.4: Hierarchical Clustering Reto West Department of Political Science and International Relations University of Geneva 1 Clustering Clustering Hierarchical Clustering

237 views • 22 slides
Clustering algorithms Machine Learning Hamid Beigy Sharif University of Technology Fall 1393

Clustering algorithms Machine Learning Hamid Beigy Sharif University of Technology Fall 1393

Clustering algorithms Machine Learning Hamid Beigy Sharif University of Technology Fall 1393 Hamid Beigy (Sharif University of Technology) Clustering algorithms Fall 1393 1 / 22 Table of contents Supervised & unsupervised learning 1

326 views • 22 slides

More recommend