data pre processing for k means clustering
play

Data pre-processing for k- means clustering Karolis Urbonas Head - PowerPoint PPT Presentation

Mar 14, 2024 •217 likes •481 views

DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Data pre-processing for k- means clustering Karolis Urbonas Head of Data Science, Amazon DataCamp Customer Segmentation in Python Advantages of k-means clustering One

  1. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Data pre-processing for k- means clustering Karolis Urbonas Head of Data Science, Amazon

  2. DataCamp Customer Segmentation in Python Advantages of k-means clustering One of the most popular unsupervised learning method Simple and fast Works well* * with certain assumptions about the data

  3. DataCamp Customer Segmentation in Python Key k-means assumptions Symmetric distribution of variables (not skewed) Variables with same average values Variables with same variance

  4. DataCamp Customer Segmentation in Python Skewed variables Left-skewed Right-skewed

  5. DataCamp Customer Segmentation in Python Skewed variables Skew removed with logarithmic transformation

  6. DataCamp Customer Segmentation in Python Variables on the same scale datamart_rfm.describe() K-means assumes equal mean And equal variance It's not the case with RFM data

  7. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Let's review the concepts

  8. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Managing skewed variables Karolis Urbonas Head of Data Science, Amazon

  9. DataCamp Customer Segmentation in Python Identifying skewness Visual analysis of the distribution If it has a tail - it's skewed

  10. DataCamp Customer Segmentation in Python Exploring distribution of Recency import seaborn as sns from matplotlib import pyplot as plt sns.distplot(datamart['Recency']) plt.show()

  11. DataCamp Customer Segmentation in Python Exploring distribution of Frequency sns.distplot(datamart['Frequency']) plt.show()

  12. DataCamp Customer Segmentation in Python Data transformations to manage skewness Logarithmic transformation (positive values only) import numpy as np frequency_log= np.log(datamart['Frequency']) sns.distplot(frequency_log) plt.show()

  13. DataCamp Customer Segmentation in Python Dealing with negative values Adding a constant before log transformation Cube root transformation

  14. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Let's practice how to identify and manage skewed variables!

  15. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Centering and scaling variables Karolis Urbonas Head of Data Science, Amazon

  16. DataCamp Customer Segmentation in Python Identifying an issue datamart_rfm.describe() Analyze key statistics of the dataset Compare mean and standard deviation

  17. DataCamp Customer Segmentation in Python Centering variables with different means K-means works well on variables with the same mean Centering variables is done by subtracting average value from each observation datamart_centered = datamart_rfm - datamart_rfm.mean() datamart_centered.describe().round(2)

  18. DataCamp Customer Segmentation in Python Scaling variables with different variance K-means works better on variables with the same variance / standard deviation Scaling variables is done by dividing them by standard deviation of each datamart_scaled = datamart_rfm / datamart_rfm.std() datamart_scaled.describe().round(2)

  19. DataCamp Customer Segmentation in Python Combining centering and scaling Subtract mean and divide by standard deviation manually Or use a scaler from scikit-learn library (returns numpy.ndarray object) from sklearn.preprocessing import StandardScaler scaler = StandardScaler() scaler.fit(datamart_rfm) datamart_normalized = scaler.transform(datamart_rfm) print('mean: ', datamart_normalized.mean(axis=0).round(2)) print('std: ', datamart_normalized.std(axis=0).round(2)) mean: [-0. -0. 0.] std: [1. 1. 1.]

  20. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Test different approaches by yourself!

  21. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Sequence of structuring pre-processing steps Karolis Urbonas Head of Data Science, Amazon

  22. DataCamp Customer Segmentation in Python Why the sequence matters? Log transformation only works with positive data Normalization forces data to have negative values and log will not work

  23. DataCamp Customer Segmentation in Python Sequence 1. Unskew the data - log transformation 2. Standardize to the same average values 3. Scale to the same standard deviation 4. Store as a separate array to be used for clustering

  24. DataCamp Customer Segmentation in Python Coding the sequence Unskew the data with log transformation import numpy as np datamart_log = np.log(datamart_rfm) Normalize the variables with StandardScaler from sklearn.preprocessing import StandardScaler scaler = StandardScaler() scaler.fit(datamart_log) Store it separately for clustering datamart_normalized = scaler.transform(datamart_log)

  25. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Practice on RFM data!

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

1 K-means clustering The K-means clustering algorithm can be seen as applying the EM algorithm to

1 K-means clustering The K-means clustering algorithm can be seen as applying the EM algorithm to

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 18 notes: K-means and Factor Analysis Tues, 4.17 1 K-means clustering The K-means clustering algorithm can be seen as

547 views • 3 slides
k -means clustering Method to automatically separate data sets into distinct groups. Clustering

k -means clustering Method to automatically separate data sets into distinct groups. Clustering

k -means clustering Method to automatically separate data sets into distinct groups. Clustering example Clustering example k -means clustering algorithm 1. Start with k randomly chosen means 2. Color data points by the shortest distance to any

1.56k views • 19 slides
Data Clustering: Data Clustering: 50 Years Beyond K means 50 Years Beyond K means 50 Years

Data Clustering: Data Clustering: 50 Years Beyond K means 50 Years Beyond K means 50 Years

Data Clustering: Data Clustering: 50 Years Beyond K means 50 Years Beyond K means 50 Years Beyond K-means 50 Years Beyond K-means Anil K. Jain Department of Computer Science Michigan State University King-Sun Fu King-Sun Fu King-Sun

1.03k views • 39 slides
Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns in data, or grouping similar groups of data-points together into clusters . Clustering algorithms for numeric data: Lloyds K-means, EM

688 views • 19 slides
Clustering: Hierarchical Clustering and K- Means Clustering Machine

Clustering: Hierarchical Clustering and K- Means Clustering Machine

Clustering: Hierarchical Clustering and K- Means Clustering Machine Learning 10-601B Seyoung Kim Many of these slides are derived from William Cohen,

660 views • 46 slides
LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm

LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm

DATA MINING LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm Clustering Evaluation What is a Clustering? In general a grouping of objects such that the objects in a group (cluster) are similar (or

1.39k views • 110 slides
Multi-variable Optimization K-means clustering K-means clustering on points is finding K

Multi-variable Optimization K-means clustering K-means clustering on points is finding K

Multi-variable Optimization K-means clustering K-means clustering on points is finding K central locations that reduce the distance of each point to the nearest central location (summed over all points) K=3 K-means clustering For

383 views • 10 slides
Clustering and Dimensionality Reduction Preview Clustering K -means clustering

Clustering and Dimensionality Reduction Preview Clustering K -means clustering

Clustering and Dimensionality Reduction Preview Clustering K -means clustering Mixture models Hierarchical clustering Dimensionality reduction Principal component analysis Multidimensional scaling Isomap

1.61k views • 17 slides
K-Means Clustering 3/3/17 Unsupervised Learning We have a collection of unlabeled data

K-Means Clustering 3/3/17 Unsupervised Learning We have a collection of unlabeled data

K-Means Clustering 3/3/17 Unsupervised Learning We have a collection of unlabeled data points. We want to find underlying structure in the data. Examples: Identify groups of similar data points. Clustering Find a better basis

831 views • 14 slides
Basics of k-means clustering CLUS TERIN G METH ODS W ITH S CIP Y Shaumik Daityari Business

Basics of k-means clustering CLUS TERIN G METH ODS W ITH S CIP Y Shaumik Daityari Business

Basics of k-means clustering CLUS TERIN G METH ODS W ITH S CIP Y Shaumik Daityari Business Analyst Why k-means clustering? A critical drawback of hierarchical clustering: runtime K means runs signicantly faster on large datasets

691 views • 25 slides
Contents Clustering K-means Mixture of Gaussians Expectation Maximization

Contents Clustering K-means Mixture of Gaussians Expectation Maximization

Contents Clustering K-means Mixture of Gaussians Expectation Maximization Variational Methods 1 Introduction to Machine Learning CMU-10701 Clustering and EM Barnabs Pczos & Aarti Singh Clustering 3 K- means

1.04k views • 67 slides
Asynchronous K-Means Clustering of Multiple Data Sets Marek Fiser, Illia Ziamtsov, Ariful Azad,

Asynchronous K-Means Clustering of Multiple Data Sets Marek Fiser, Illia Ziamtsov, Ariful Azad,

Asynchronous K-Means Clustering of Multiple Data Sets Marek Fiser, Illia Ziamtsov, Ariful Azad, Bedrich Benes, Alex Pothen Motivation Clustering bottleneck in Flow Cytometry research 3,000 data sets 25,000 points in 7D per data set 19 separate

406 views • 19 slides
Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering

Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering

Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering K-means Mixture of Gaussians Expectation Maximization Variational Methods 2 Clustering 3 K- means clustering What is clustering?

1.32k views • 66 slides
DS504/CS586: Big Data Analytics Big Data Clustering II Prof. Yanhua Li Time: 6pm 8:50pm Thu

DS504/CS586: Big Data Analytics Big Data Clustering II Prof. Yanhua Li Time: 6pm 8:50pm Thu

Welcome to DS504/CS586: Big Data Analytics Big Data Clustering II Prof. Yanhua Li Time: 6pm 8:50pm Thu Location: AK 232 Fall 2016 More Discussions, Limitations v Center based clustering K-means BFR algorithm v Hierarchical clustering

873 views • 52 slides
Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means

Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means

Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means clustering Hierarchical clustering Incremental clustering Probability-based clustering Self-Organising Maps Classification vs. Clustering

1k views • 67 slides
Evolutionary Clustering Presenter: Lei Tang Evolutionary Clustering Evolutionary Clustering

Evolutionary Clustering Presenter: Lei Tang Evolutionary Clustering Evolutionary Clustering

Evolutionary Clustering Presenter: Lei Tang Evolutionary Clustering Evolutionary Clustering Processing time stamped data to produce Processing time stamped data to produce a sequence of clustering. Each clustering should be similar to

431 views • 32 slides
Peak-End Rule: Main Result A Utility-Based Discussion First Open Problem Explanation Second

Peak-End Rule: Main Result A Utility-Based Discussion First Open Problem Explanation Second

Peak-End Rule: . . . Towards an Explanation Need for a Utility- . . . Natural Properties of . . . Peak-End Rule: Main Result A Utility-Based Discussion First Open Problem Explanation Second Open Problem Proof Olga Kosheleva, Martine

582 views • 16 slides
Recent Trends in Algorithms National Institute of Science Education and Research The standard

Recent Trends in Algorithms National Institute of Science Education and Research The standard

Chamberlin-Courant on Restricted Domains Neeldhara Misra Recent Trends in Algorithms National Institute of Science Education and Research The standard Voting Setup The standard Voting Setup and typical computational problems. The standard

2.17k views • 161 slides
Bypassing Combinatorial Protections Polynomial-Time Algorithms for Single-Peaked Electorates

Bypassing Combinatorial Protections Polynomial-Time Algorithms for Single-Peaked Electorates

Bypassing Combinatorial Protections Polynomial-Time Algorithms for Single-Peaked Electorates Markus Brill COMSOC 2010 Dsseldorf Joint work with Felix Brandt, Edith Hemaspaandra, and Lane Hemaspaandra Voting Rules C = {a,b,c,...} is a

660 views • 11 slides
I NF ORMAT I ON T E CHNOL OGY ADVI SORY COMMI T T E E DAT A E XCHANGE WORK ST RE

I NF ORMAT I ON T E CHNOL OGY ADVI SORY COMMI T T E E DAT A E XCHANGE WORK ST RE

I NF ORMAT I ON T E CHNOL OGY ADVI SORY COMMI T T E E DAT A E XCHANGE WORK ST RE AM RE PORT June 10, 2016 DX WORK ST RE AM SUMMARY OF E VE NT S E nte ring 18 Mo nth E sta b lishe d F a c e -to -F a c e

218 views • 5 slides
What is a monoid? How I learnt to stop worrying and love skewness Paul Blain Levy University of

What is a monoid? How I learnt to stop worrying and love skewness Paul Blain Levy University of

What is a monoid? How I learnt to stop worrying and love skewness Paul Blain Levy University of Birmingham July 12, 2019 Paul Blain Levy (University of Birmingham) What is a monoid? July 12, 2019 1 / 23

868 views • 38 slides
Skew structures in 2-category theory and homotopy theory John Bourke Department of Mathematics

Skew structures in 2-category theory and homotopy theory John Bourke Department of Mathematics

Introduction Background on skew structures Homotopical version of the Eilenberg-Kelly theorem Examples and applications of homotopical EK theorem Skew structures in 2-category theory and homotopy theory John Bourke Department of Mathematics

786 views • 64 slides
Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family

Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family

Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations Wu Lin (UBC) June 11, 2019 Joint work with Mohammad Emtiyaz Khan (AIP, RIKEN) and Mark Schmidt (UBC) 1 / 7 Variational Inference (VI) VI

119 views • 7 slides
STAT 113 Variability Colin Reimer Dawson Oberlin College September 14, 2017 1 / 48 Last Time:

STAT 113 Variability Colin Reimer Dawson Oberlin College September 14, 2017 1 / 48 Last Time:

Last Time: Shape and Center Variability Variance and Standard Deviaton Transformations STAT 113 Variability Colin Reimer Dawson Oberlin College September 14, 2017 1 / 48 Last Time: Shape and Center Variability Variance and Standard

743 views • 48 slides

More recommend