practical implementation of k means clustering
play

Practical implementation of k-means clustering Karolis Urbonas - PowerPoint PPT Presentation

Sep 11, 2023 •475 likes •862 views

DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Practical implementation of k-means clustering Karolis Urbonas Head of Data Science, Amazon DataCamp Customer Segmentation in Python Key steps Data pre-processing

  1. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Practical implementation of k-means clustering Karolis Urbonas Head of Data Science, Amazon

  2. DataCamp Customer Segmentation in Python Key steps Data pre-processing Choosing a number of clusters Running k-means clustering on pre-processed data Analyzing average RFM values of each cluster

  3. DataCamp Customer Segmentation in Python Data pre-processing We've completed the pre-processing steps and have these two objects: datamart_rfm datamart_normalized Code from previous lesson: import numpy as np datamart_log = np.log(datamart_rfm) from sklearn.preprocessing import StandardScaler scaler = StandardScaler() scaler.fit(datamart_log) datamart_normalized = scaler.transform(datamart_log)

  4. DataCamp Customer Segmentation in Python Methods to define the number of clusters Visual methods - elbow criterion Mathematical methods - silhouette coefficient Experimentation and interpretation

  5. DataCamp Customer Segmentation in Python Running k-means Import KMeans from sklearn library and initialize it as kmeans from sklearn.cluster import KMeans kmeans = KMeans(n_clusters=2, random_state=1) Compute k-means clustering on pre-processed data kmeans.fit(datamart_normalized) Extract cluster labels from labels_ attribute cluster_labels = kmeans.labels_

  6. DataCamp Customer Segmentation in Python Analyzing average RFM values of each cluster Create a cluster label column in the original DataFrame: datamart_rfm_k2 = datamart_rfm.assign(Cluster = cluster_labels) Calculate average RFM values and size for each cluster: datamart_rfm_k2.groupby(['Cluster']).agg({ 'Recency': 'mean', 'Frequency': 'mean', 'MonetaryValue': ['mean', 'count'], }).round(0)

  7. DataCamp Customer Segmentation in Python Analyzing average RFM values of each cluster The result of a simple 2-cluster solution:

  8. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Let's practice running k- means clustering!

  9. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Choosing number of clusters Karolis Urbonas Head of Data Science, Amazon

  10. DataCamp Customer Segmentation in Python Methods Visual methods - elbow criterion Mathematical methods - silhouette coefficient Experimentation and interpretation

  11. DataCamp Customer Segmentation in Python Elbow criterion method Plot the number of clusters against within-cluster sum-of-squared-errors (SSE) - sum of squared distances from every data point to their cluster center Identify an "elbow" in the plot Elbow - a point representing an "optimal" number of clusters

  12. DataCamp Customer Segmentation in Python Elbow criterion method # Import key libraries from sklearn.cluster import KMeans import seaborn as sns from matplotlib import pyplot as plt # Fit KMeans and calculate SSE for each *k* sse = {} for k in range(1, 11): kmeans = KMeans(n_clusters=k, random_state=1) kmeans.fit(data_normalized) sse[k] = kmeans.inertia_ # sum of squared distances to closest cluster cente # Plot SSE for each *k* plt.title('The Elbow Method') plt.xlabel('k'); plt.ylabel('SSE') sns.pointplot(x=list(sse.keys()), y=list(sse.values())) plt.show()

  13. DataCamp Customer Segmentation in Python Elbow criterion method The elbow criterion chart:

  14. DataCamp Customer Segmentation in Python Elbow criterion method The elbow criterion chart:

  15. DataCamp Customer Segmentation in Python Using elbow criterion method Best to choose the point on elbow, or the next point Use as a guide but test multiple solutions Elbow plot built on datamart_rfm

  16. DataCamp Customer Segmentation in Python Experimental approach - analyze segments Build clustering at and around elbow solution Analyze their properties - average RFM values Compare against each other and choose one which makes most business sense

  17. DataCamp Customer Segmentation in Python Experimental approach - analyze segments Previous 2-cluster solution 3-cluster solution on the same normalized RFM dataset

  18. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Let's practice finding the optimal number of clusters!

  19. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Profile and interpret segments Karolis Urbonas Head of Data Science, Amazon

  20. DataCamp Customer Segmentation in Python Approaches to build customer personas Summary statistics for each cluster e.g. average RFM values Snake plots (from market research Relative importance of cluster attributes compared to population

  21. DataCamp Customer Segmentation in Python Summary statistics of each cluster Run k-means segmentation for several k values around the recommended value. Create a cluster label column in the original DataFrame: datamart_rfm_k2 = datamart_rfm.assign(Cluster = cluster_labels) Calculate average RFM values and sizes for each cluster: datamart_rfm_k2.groupby(['Cluster']).agg({ 'Recency': 'mean', 'Frequency': 'mean', 'MonetaryValue': ['mean', 'count'], }).round(0) Repeat the same for k=3

  22. DataCamp Customer Segmentation in Python Summary statistics of each cluster Compare average RFM values of each clustering solution

  23. DataCamp Customer Segmentation in Python Snake plots to understand and compare segments Market research technique to compare different segments Visual representation of each segment's attributes Need to first normalize data (center & scale) Plot each cluster's average normalized values of each attribute

  24. DataCamp Customer Segmentation in Python Prepare data for a snake plot Transform datamart_normalized as DataFrame and add a Cluster column datamart_normalized = pd.DataFrame(datamart_normalized, index=datamart_rfm.index, columns=datamart_rfm.columns) datamart_normalized['Cluster'] = datamart_rfm_k3['Cluster'] Melt the data into a long format so RFM values and metric names are stored in 1 column each datamart_melt = pd.melt(datamart_normalized.reset_index(), id_vars=['CustomerID', 'Cluster'], value_vars=['Recency', 'Frequency', 'MonetaryValue'], var_name='Attribute', value_name='Value')

  25. DataCamp Customer Segmentation in Python Visualize a snake plot plt.title('Snake plot of standardized variables') sns.lineplot(x="Attribute", y="Value", hue='Cluster', data=datamart_melt)

  26. DataCamp Customer Segmentation in Python Relative importance of segment attributes Useful technique to identify relative importance of each segment's attribute Calculate average values of each cluster Calculate average values of population Calculate importance score by dividing them and subtracting 1 ( ensures 0 is returned when cluster average equals population average ) cluster_avg = datamart_rfm_k3.groupby(['Cluster']).mean() population_avg = datamart_rfm.mean() relative_imp = cluster_avg / population_avg - 1

  27. DataCamp Customer Segmentation in Python Analyze and plot relative importance The further a ratio is from 0, the more important that attribute is for a segment relative to the total population. relative_imp.round(2) Recency Frequency MonetaryValue Cluster 0 -0.82 1.68 1.83 1 0.84 -0.84 -0.86 2 -0.15 -0.34 -0.42 Plot a heatmap for easier interpretation: plt.figure(figsize=(8, 2)) plt.title('Relative importance of attributes') sns.heatmap(data=relative_imp, annot=True, fmt='.2f', cmap='RdYlGn') plt.show()

  28. DataCamp Customer Segmentation in Python Relative importance heatmap Heatmap plot: vs. printed output: Recency Frequency MonetaryValue Cluster 0 -0.82 1.68 1.83 1 0.84 -0.84 -0.86 2 -0.15 -0.34 -0.42

  29. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Your time to experiment with different customer profiling techniques!

  30. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Implement end-to-end segmentation solution Karolis Urbonas Head of Data Science, Amazon

  31. DataCamp Customer Segmentation in Python Key steps of the segmentation project Gather data - updated data with an additional variable Pre-process the data Explore the data and decide on the number of clusters Run k-means clustering Analyze and visualize results

  32. DataCamp Customer Segmentation in Python Updated RFM data Same RFM values plus additional Tenure variable Tenure - time since the first transaction Defines how long the customer has been with the company

  33. DataCamp Customer Segmentation in Python Goals for this project Remember key pre-processing rules Apply data exploration techniques Practice running several k-means iterations Analyze results quantitatively and visually

  34. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Let's dig in!

  35. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Final thoughts Karolis Urbonas Head of Data Science, Amazon

  36. DataCamp Customer Segmentation in Python What you have learned Cohort analysis and visualization RFM segmentation Data pre-processing for k-means Customer segmentation with k-means Evaluating number of clusters Reviewing and visualizing segmentation solutions

  37. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Congratulations!

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

1 K-means clustering The K-means clustering algorithm can be seen as applying the EM algorithm to

1 K-means clustering The K-means clustering algorithm can be seen as applying the EM algorithm to

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 18 notes: K-means and Factor Analysis Tues, 4.17 1 K-means clustering The K-means clustering algorithm can be seen as

547 views • 3 slides
Clustering: Hierarchical Clustering and K- Means Clustering Machine

Clustering: Hierarchical Clustering and K- Means Clustering Machine

Clustering: Hierarchical Clustering and K- Means Clustering Machine Learning 10-601B Seyoung Kim Many of these slides are derived from William Cohen,

660 views • 46 slides
Clustering CS294 Practical Machine Learning Junming Yin 10/09/06 Outline Introduction

Clustering CS294 Practical Machine Learning Junming Yin 10/09/06 Outline Introduction

Clustering CS294 Practical Machine Learning Junming Yin 10/09/06 Outline Introduction Unsupervised learning What is clustering? Application Dissimilarity (similarity) of objects Clustering algorithm K-means, VQ,

649 views • 62 slides
Multi-variable Optimization K-means clustering K-means clustering on points is finding K

Multi-variable Optimization K-means clustering K-means clustering on points is finding K

Multi-variable Optimization K-means clustering K-means clustering on points is finding K central locations that reduce the distance of each point to the nearest central location (summed over all points) K=3 K-means clustering For

383 views • 10 slides
Clustering and Dimensionality Reduction Preview Clustering K -means clustering

Clustering and Dimensionality Reduction Preview Clustering K -means clustering

Clustering and Dimensionality Reduction Preview Clustering K -means clustering Mixture models Hierarchical clustering Dimensionality reduction Principal component analysis Multidimensional scaling Isomap

1.61k views • 17 slides
k -means clustering Method to automatically separate data sets into distinct groups. Clustering

k -means clustering Method to automatically separate data sets into distinct groups. Clustering

k -means clustering Method to automatically separate data sets into distinct groups. Clustering example Clustering example k -means clustering algorithm 1. Start with k randomly chosen means 2. Color data points by the shortest distance to any

1.56k views • 19 slides
Data Clustering: Data Clustering: 50 Years Beyond K means 50 Years Beyond K means 50 Years

Data Clustering: Data Clustering: 50 Years Beyond K means 50 Years Beyond K means 50 Years

Data Clustering: Data Clustering: 50 Years Beyond K means 50 Years Beyond K means 50 Years Beyond K-means 50 Years Beyond K-means Anil K. Jain Department of Computer Science Michigan State University King-Sun Fu King-Sun Fu King-Sun

1.03k views • 39 slides
Basics of k-means clustering CLUS TERIN G METH ODS W ITH S CIP Y Shaumik Daityari Business

Basics of k-means clustering CLUS TERIN G METH ODS W ITH S CIP Y Shaumik Daityari Business

Basics of k-means clustering CLUS TERIN G METH ODS W ITH S CIP Y Shaumik Daityari Business Analyst Why k-means clustering? A critical drawback of hierarchical clustering: runtime K means runs signicantly faster on large datasets

691 views • 25 slides
Contents Clustering K-means Mixture of Gaussians Expectation Maximization

Contents Clustering K-means Mixture of Gaussians Expectation Maximization

Contents Clustering K-means Mixture of Gaussians Expectation Maximization Variational Methods 1 Introduction to Machine Learning CMU-10701 Clustering and EM Barnabs Pczos & Aarti Singh Clustering 3 K- means

1.04k views • 67 slides
Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering

Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering

Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering K-means Mixture of Gaussians Expectation Maximization Variational Methods 2 Clustering 3 K- means clustering What is clustering?

1.32k views • 66 slides
LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm

LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm

DATA MINING LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm Clustering Evaluation What is a Clustering? In general a grouping of objects such that the objects in a group (cluster) are similar (or

1.39k views • 110 slides
Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means

Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means

Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means clustering Hierarchical clustering Incremental clustering Probability-based clustering Self-Organising Maps Classification vs. Clustering

1k views • 67 slides
Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering? Aykut Erdem May 2016 Hacettepe University Last time K-Means An iterative clustering algorithm - Initialize: Pick K random points as cluster

932 views • 64 slides
Clustering with k-means and Gaussian mixture distributions Machine Learning and Object Recognition

Clustering with k-means and Gaussian mixture distributions Machine Learning and Object Recognition

Clustering with k-means and Gaussian mixture distributions Machine Learning and Object Recognition 2016-2017 Jakob Verbeek Practical matters Online course information Schedule, slides, papers

439 views • 39 slides
K-MEANS++ OPTIMAL INITIALIZATION ALGORITHM An Improved K-means Clustering Method OVERVIEW

K-MEANS++ OPTIMAL INITIALIZATION ALGORITHM An Improved K-means Clustering Method OVERVIEW

K-MEANS++ OPTIMAL INITIALIZATION ALGORITHM An Improved K-means Clustering Method OVERVIEW K-means Clustering Algorithm K-means++ Initialization Algorithm Experiment Datasets Conclusion K-MEANS CLUSTERING ALGORITHM A

313 views • 14 slides
Implementation of K-means on the Cell Broadband Engine Filip imek (simekf1@fel.cvut.cz) About

Implementation of K-means on the Cell Broadband Engine Filip imek (simekf1@fel.cvut.cz) About

IBM - CVUT Student Research Projects Implementation of K-means on the Cell Broadband Engine Filip imek (simekf1@fel.cvut.cz) About K-means a data clustering algorithm finds clusters of points in space through an iterative process

294 views • 6 slides
Dynamic Programming 11.1 Overview Dynamic Programming is a powerful technique that allows one to

Dynamic Programming 11.1 Overview Dynamic Programming is a powerful technique that allows one to

Lecture 11 Dynamic Programming 11.1 Overview Dynamic Programming is a powerful technique that allows one to solve many different types of problems in time O ( n 2 ) or O ( n 3 ) for which a naive approach would take exponential time. In this

259 views • 7 slides
R/exams: A One-for-All Exams Generator Written Exams, Online Tests, and Live Quizzes with R Achim

R/exams: A One-for-All Exams Generator Written Exams, Online Tests, and Live Quizzes with R Achim

R/exams: A One-for-All Exams Generator Written Exams, Online Tests, and Live Quizzes with R Achim Zeileis http://www.R-exams.org/ R/exams: A One-for-All Exams Generator Written Exams, Online Tests, and Live Quizzes with R Achim Zeileis

1.08k views • 74 slides
Emulation Outline Emulation Interpretation basic, threaded, directed threaded

Emulation Outline Emulation Interpretation basic, threaded, directed threaded

Emulation Outline Emulation Interpretation basic, threaded, directed threaded other issues Binary translation code discovery, code location other issues Control Transfer Optimizations 1 EECS 768 Virtual Machines

660 views • 63 slides
Towards Automated Dynamic Analysis for Linux-based Embedded Firmware Dominic Chen 1 , Manuel Egele

Towards Automated Dynamic Analysis for Linux-based Embedded Firmware Dominic Chen 1 , Manuel Egele

Towards Automated Dynamic Analysis for Linux-based Embedded Firmware Dominic Chen 1 , Manuel Egele 2 , Maverick Woo 1 , David Brumley 1 1 Carnegie Mellon University, 2 Boston University {ddchen, pooh, dbrumley}@cmu.edu, megele@bu.edu 2 FIRMADYNE

404 views • 22 slides
Regulations.gov Overview of the Latest Features and Functionality The Status of Social Media in

Regulations.gov Overview of the Latest Features and Functionality The Status of Social Media in

Regulations.gov Overview of the Latest Features and Functionality The Status of Social Media in Government September 17, 2013 1 Agenda Overview of eRulemaking Program Current Regulations.gov Features and Functionality Use of

297 views • 12 slides
MOL2NET, 2017 , 3, doi:10.3390/mol2net-03-xxxx 2 25923, Pseudomonas aeruginosa ATCC 27853, Proteus

MOL2NET, 2017 , 3, doi:10.3390/mol2net-03-xxxx 2 25923, Pseudomonas aeruginosa ATCC 27853, Proteus

MOL2NET, 2017 , 3, doi:10.3390/mol2net-03-xxxx 1 MOL2NET, International Conference Series on Multidisciplinary Sciences MDPI http://sciforum.net/conference/mol2net-03 Standardization of the Safety Level of the Use of DMSO in Viability Assays in

176 views • 6 slides
Towards More Adequate Natural Idea: Using . . . Linear Dependence . . . Value-Added How to

Towards More Adequate Natural Idea: Using . . . Linear Dependence . . . Value-Added How to

Assessment is Important Need for Value-Added . . . Current Approaches to . . . Towards More Adequate Natural Idea: Using . . . Linear Dependence . . . Value-Added How to Determine the . . . Case of Interval . . . Teacher Assessments:

595 views • 16 slides
The View from AI2 Oren Etzioni, CEO Allen Institute for AI (AI2) Mission: contribute to the world

The View from AI2 Oren Etzioni, CEO Allen Institute for AI (AI2) Mission: contribute to the world

The View from AI2 Oren Etzioni, CEO Allen Institute for AI (AI2) Mission: contribute to the world through high-impact AI research and engineering, with emphasis on reasoning, learning, and reading capabilities. Outline: 1. Overview of AI2

784 views • 37 slides

More recommend