diagnose data for cleaning
play

Diagnose data for cleaning CLEAN IN G DATA IN P YTH ON Daniel - PowerPoint PPT Presentation

Jan 31, 2024 •386 likes •780 views

Diagnose data for cleaning CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Cleaning data Prepare data for analysis Data almost never comes in clean Diagnose your data for problems CLEANING DATA IN PYTHON Common data problems

  1. Diagnose data for cleaning CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor

  2. Cleaning data Prepare data for analysis Data almost never comes in clean Diagnose your data for problems CLEANING DATA IN PYTHON

  3. Common data problems Inconsistent column names Missing data Outliers Duplicate rows Untidy Need to process columns Column types can signal unexpected data values CLEANING DATA IN PYTHON

  4. 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  5. 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  6. 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  7. Column name inconsistencies 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  8. Column name inconsistencies 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  9. Column name inconsistencies 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  10. Column name inconsistencies 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  11. Column name inconsistencies Missing data 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  12. Column name inconsistencies Missing data Country names are in French 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  13. Load your data import pandas as pd df = pd.read_csv('literary_birth_rate.csv') CLEANING DATA IN PYTHON

  14. Visually inspect df.head() Continent Country female literacy fertility population 0 ASI Chine 90.5 1.769 1.324655e+09 1 ASI Inde 50.8 2.682 1.139965e+09 2 NAM USA 99.0 2.077 3.040600e+08 3 ASI Indonésie 88.8 2.132 2.273451e+08 4 LAT Brésil 90.2 1.827 NaN df.tail() Continent Country female literacy fertility population 0 AF Sao Tomé-et-Principe 90.5 1.769 1.324655e+09 1 LAT Aruba 50.8 2.682 1.139965e+09 2 ASI Tonga 99.0 2.077 3.040600e+08 3 OCE Australia 88.8 2.132 2.273451e+08 4 OCE Sweden 90.2 1.827 NaN CLEANING DATA IN PYTHON

  15. Visually inspect df.columns Index(['Continent', 'Country ', 'female literacy', 'fertility', 'population'], dtype='object') df.shape (164, 5) df.info() <class 'pandas.core.frame.DataFrame'> RangeIndex: 164 entries, 0 to 163 Data columns (total 5 columns): Continent 164 non-null object Country 164 non-null object female literacy 164 non-null float64 fertility 164 non-null object population 122 non-null float64 dtypes float64(2), object(3) memory usage: 6.5+ KB CLEANING DATA IN PYTHON

  16. Let's practice! CLEAN IN G DATA IN P YTH ON

  17. Exploratory data analysis CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor

  18. Frequency counts Count the number of unique values in our data CLEANING DATA IN PYTHON

  19. Data type of each column df.info() <class 'pandas.core.frame.DataFrame'> RangeIndex: 164 entries, 0 to 163 Data columns (total 5 columns): continent 164 non-null object country 164 non-null object female literacy 164 non-null float64 fertility 164 non-null object population 122 non-null float64 dtypes float64(2), object(3) memory usage: 6.5+ KB CLEANING DATA IN PYTHON

  20. Frequency counts: continent df.continent.value_counts(dropna=False) AF 49 ASI 47 EUR 36 LAT 24 OCE 6 NAM 2 Name: continent, dtype: int64 CLEANING DATA IN PYTHON

  21. Frequency counts: continent df['continent'].value_counts(dropna=False) AF 49 ASI 47 EUR 36 LAT 24 OCE 6 NAM 2 Name: continent, dtype: int64 CLEANING DATA IN PYTHON

  22. Frequency counts: country df.country.value_counts(dropna=False).head() Sweden 2 Algerie 1 Germany 1 Angola 1 Indonésie 1 Name: country, dtype: int64 CLEANING DATA IN PYTHON

  23. Frequency counts: fertility df.fertility.value_counts(dropna=False).head() missing 5 1.854 2 1.93 2 1.841 2 1.393 2 Name: fertility, dtype: int64 CLEANING DATA IN PYTHON

  24. Frequency counts: population df.population.value_counts(dropna=False).head() NaN 42 5.667325e+06 1 3.773100e+06 1 1.333388e+06 1 1.661115e+08 1 Name: population, dtype: int64 CLEANING DATA IN PYTHON

  25. Summary statistics Numeric columns Outliers Considerably higher or lower Require further investigation CLEANING DATA IN PYTHON

  26. Summary statistics: numeric data df.describe() female_literacy population count 164.000000 1.220000e+02 mean 80.301220 6.345768e+07 std 22.977265 2.605977e+08 min 12.600000 1.035660e+05 25% 66.675000 3.778175e+06 50% 90.200000 9.995450e+06 75% 98.500000 2.642217e+07 max 100.000000 2.313000e+09 CLEANING DATA IN PYTHON

  27. Let's practice! CLEAN IN G DATA IN P YTH ON

  28. Visual exploratory data analysis CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor

  29. Data visualization Great way to spot outliers and obvious errors More than just looking for patterns Plan data cleaning steps CLEANING DATA IN PYTHON

  30. Summary statistics df.describe() female_literacy fertility population count 164.000000 163.000000 1.220000e+02 mean 80.301220 2.872853 6.345768e+07 std 22.977265 1.425122 2.605977e+08 min 12.600000 0.966000 1.035660e+05 25% 66.675000 1.824500 3.778175e+06 50% 90.200000 2.362000 9.995450e+06 75% 98.500000 3.877500 2.642217e+07 max 100.000000 7.069000 2.313000e+09 CLEANING DATA IN PYTHON

  31. Bar plots and histograms Bar plots for discrete data counts Histograms for continuous data counts Look at frequencies CLEANING DATA IN PYTHON

  32. Histogram df.population.plot('hist') <matplotlib.axes._subplots.AxesSubplot at 0x7f78e4abafd0> import matplotlib.pyplot as plt plt.show() CLEANING DATA IN PYTHON

  33. Identifying the error df[df.population > 1000000000] continent country female literacy fertility population 0 ASI Chine 90.5 1.769 1.324655e+09 1 ASI Inde 50.8 2.682 1.139965e+09 162 OCE Australia 96.0 1.930 2.313000e+09 Not all outliers are bad data points Some can be an error, but others are valid values CLEANING DATA IN PYTHON

  34. Box plots Visualize basic summary statistics Outliers Min/max 25th, 50th, 75th percentiles CLEANING DATA IN PYTHON

  35. Box plot df.boxplot(column='population', by='continent') <matplotlib.axes._subplots.AxesSubplot at 0x7ff5581bb630> plt.show() CLEANING DATA IN PYTHON

  36. Box plot df.boxplot(column='population', by='continent') <matplotlib.axes._subplots.AxesSubplot at 0x7ff5581bb630> plt.show() CLEANING DATA IN PYTHON

  37. Scatter plots Relationship between 2 numeric variables Flag potentially bad data Errors not found by looking at 1 variable CLEANING DATA IN PYTHON

  38. Let's practice! CLEAN IN G DATA IN P YTH ON

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Diagnose data for cleaning Cleaning Data in Python Cleaning data Prepare data for analysis

Diagnose data for cleaning Cleaning Data in Python Cleaning data Prepare data for analysis

CLEANING DATA IN PYTHON Diagnose data for cleaning Cleaning Data in Python Cleaning data Prepare data for analysis Data almost never comes in clean Diagnose your data for problems Cleaning Data in Python Common data problems

1.08k views • 29 slides
What do we know? What do we know? (How do we diagnose?) (How do we diagnose?) Behavior

What do we know? What do we know? (How do we diagnose?) (How do we diagnose?) Behavior

Diagnosis and Misdiagnosis of Gifted Youth Presented at the 2012 California Psychological Association, Monterey, CA Daniel Peters, Ph.D. Stephen H. Chou, Psy.D. What do we know? What do we know? (How do we diagnose?) (How do we diagnose?)

172 views • 13 slides
Data cleaning and standardisation (1) Data cleaning and standardisation (2) Real world data is

Data cleaning and standardisation (1) Data cleaning and standardisation (2) Real world data is

Data cleaning and standardisation Febrl A parallel open source record linkage and geocoding system Record linkage and data integration Febrl overview Peter Christen Probabilistic data cleaning and standardisation Data Mining Group,

663 views • 5 slides
Workflow basics, RMarkdown, git/Github Cleaning up Cleaning up Cleaning up Cleaning up

Workflow basics, RMarkdown, git/Github Cleaning up Cleaning up Cleaning up Cleaning up

Workflow basics, RMarkdown, git/Github Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Don't worry, your history is preserved Settings Cleaning up Cleaning up Lots more options! RMarkdown Structure of an Rmd

656 views • 50 slides
Data Mining: A Powerful Data Mining: A Powerful Tool for Data Cleaning Tool for Data Cleaning

Data Mining: A Powerful Data Mining: A Powerful Tool for Data Cleaning Tool for Data Cleaning

Data Mining: A Powerful Data Mining: A Powerful Tool for Data Cleaning Tool for Data Cleaning Jiawei Han Department of Computer Science University of Illinois at Urbana-Champaign Nov. 4, 2003 1 Data mining for data quality assurance

721 views • 40 slides
Concatenating data Cleaning Data in Python Combining data Data may not always come in 1

Concatenating data Cleaning Data in Python Combining data Data may not always come in 1

CLEANING DATA IN PYTHON Concatenating data Cleaning Data in Python Combining data Data may not always come in 1 huge file 5 million row dataset may be broken into 5 separate datasets Easier to store and share May have

425 views • 26 slides
MORPH-II Dataset 1. Introduction to the Data 2. Inconsistencies in the Data 3. Cleaning the Data

MORPH-II Dataset 1. Introduction to the Data 2. Inconsistencies in the Data 3. Cleaning the Data

Garrett Bingham &amp; Ben Yip Summary and Cleaning June 16, 2017 University of North Carolina Wilmington MORPH-II Dataset 1. Introduction to the Data 2. Inconsistencies in the Data 3. Cleaning the Data 4. New Datasets 5. Dirty Data 6.

652 views • 26 slides
Intro to data cleaning with Apache Spark CLEAN IN G DATA W ITH P YS PARK Mike Metzger Data

Intro to data cleaning with Apache Spark CLEAN IN G DATA W ITH P YS PARK Mike Metzger Data

Intro to data cleaning with Apache Spark CLEAN IN G DATA W ITH P YS PARK Mike Metzger Data Engineering Consultant What is Data Cleaning? Data Cleaning : Preparing raw data for use in data processing pipelines. Possible tasks in data

1.03k views • 20 slides
Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang,

Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang,

Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Data Cleaning? 2 Data is Dirty 2 incomplete inconsistent inaccurate Data is Dirty 2 incomplete 25% companies: flawed data

1.07k views • 106 slides
Data types Cleaning Data in Python Prepare and clean data Cleaning Data in Python Data types

Data types Cleaning Data in Python Prepare and clean data Cleaning Data in Python Data types

CLEANING DATA IN PYTHON Data types Cleaning Data in Python Prepare and clean data Cleaning Data in Python Data types In [1]: print(df.dtypes) name object sex object treatment a object treatment b int64 dtype:

714 views • 44 slides
Data Preparation Data cleaning Data integration and transformation (Data

Data Preparation Data cleaning Data integration and transformation (Data

Data Preprocessing Why preprocess the data? Data Preparation Data cleaning Data integration and transformation (Data preprocessing) Data reduction, Feature selection Discretization and concept hierarchy generation 2

891 views • 21 slides
ASR Data Cleaning Guidelines Presenter: Asima Hameed Data Cleaning Process The speech files will

ASR Data Cleaning Guidelines Presenter: Asima Hameed Data Cleaning Process The speech files will

ASR Data Cleaning Guidelines Presenter: Asima Hameed Data Cleaning Process The speech files will be cleaned on the basis of: Silence Noise Mispronunciation Silence 1. A silence period of at least 200 ms 2. For voiced and voiceless

600 views • 26 slides
Outline Data cleaning and standardisation Febrl A parallel open source data linkage and

Outline Data cleaning and standardisation Febrl A parallel open source data linkage and

Outline Data cleaning and standardisation Febrl A parallel open source data linkage and geocoding system Data linkage and data integration Febrl overview Peter Christen, Tim Churches and others Probabilistic data cleaning and

833 views • 8 slides
DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING &quot;Data cleaning and

DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING &quot;Data cleaning and

DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING &quot;Data cleaning and repairing account for about 60% of the work of data scientists.&quot; Christian Kaestner Required reading: Schelter, S., Lange, D., Schmidt, P.,

1.1k views • 86 slides
Data Handling: Import, Cleaning and Visualisation Lecture 7: Data Sources, Data Gathering, Data

Data Handling: Import, Cleaning and Visualisation Lecture 7: Data Sources, Data Gathering, Data

9/12/2019 Data Handling: Import, Cleaning and Visualisation Data Handling: Import, Cleaning and Visualisation Lecture 7: Data Sources, Data Gathering, Data Import Prof. Dr. Ulrich Matter 24/10/2019 file:///home/umatter/Dropbox/T

1.06k views • 54 slides
Data Preparation Data cleaning Discretization (Data preprocessing) Data

Data Preparation Data cleaning Discretization (Data preprocessing) Data

Data Preprocessing Why preprocess the data? Data Preparation Data cleaning Discretization (Data preprocessing) Data integration and transformation Data reduction, Feature selection 2 Why Prepare Data? Why Prepare

462 views • 15 slides
Fundamentals of Machine Learning for Neural Machine Translation Dr. John D. Kelleher ADAPT

Fundamentals of Machine Learning for Neural Machine Translation Dr. John D. Kelleher ADAPT

Fundamentals of Machine Learning for Neural Machine Translation Dr. John D. Kelleher ADAPT Centre for Digital Content Technology Dublin Institute of Technology, Ireland 1 Introduction This paper 1 presents a short introduction to neural

321 views • 15 slides
III.1.3 Climate change Geoscience: The Earth and its Resources

III.1.3 Climate change Geoscience: The Earth and its Resources

III.1.3 Climate change Geoscience: The Earth and its Resources Prof. Dr. G. Berto, Climate Temperatures Energy from the sun Wind circula=on on

494 views • 14 slides
Damage due to the Great East Japan Fukushima Daiichi Nuclear Power Station Earthquake (NPS)

Damage due to the Great East Japan Fukushima Daiichi Nuclear Power Station Earthquake (NPS)

Damage due to the Great East Japan Fukushima Daiichi Nuclear Power Station Earthquake (NPS) Accident A 9.0magnitude earthquake occurred off the coast of Sanriku at 14:46 p.m. on Friday, March 11, 2011. The Earthquake and subsequent

270 views • 4 slides
8.7 PLANAR GRAPHS def: A graph is planar if it can be drawn with- out edge-crossings in the

8.7 PLANAR GRAPHS def: A graph is planar if it can be drawn with- out edge-crossings in the

8.7.1 Section 8.7 Planar Graphs 8.7 PLANAR GRAPHS def: A graph is planar if it can be drawn with- out edge-crossings in the plane. Given a graph G and a Imbedding Problem: surface S , is it possible to draw G on S without any

372 views • 18 slides
EUROPA NO SCULO XIX http://historiaonline.com.br A INGLATERRA NO SCULO XIX: Era

EUROPA NO SCULO XIX http://historiaonline.com.br A INGLATERRA NO SCULO XIX: Era

EUROPA NO SCULO XIX http://historiaonline.com.br A INGLATERRA NO SCULO XIX: Era Vitoriana (1837-1901): Hegemonia martima inglesa. Fortalecimento do poder poltico da burguesia. Expanso da economia industrial 2

754 views • 57 slides
1 Unioncamere Europa- 22/03/2019 Focus areas Members 2 14 Calls, 63 Projects More than

1 Unioncamere Europa- 22/03/2019 Focus areas Members 2 14 Calls, 63 Projects More than

1 Unioncamere Europa- 22/03/2019 Focus areas Members 2 14 Calls, 63 Projects More than 2100 participations 3.4 bn RDI costs 1.6 bn EU and National funding 1.8 bn industries own financing 3 Lighthouse

241 views • 12 slides
HUC Sustainability Workshop Community: Earth Science Grid 1

HUC Sustainability Workshop Community: Earth Science Grid 1

EGI-InSPIRE Lyon - September 2011 HUC Sustainability Workshop Community: Earth Science Grid 1 19.09.11 www.egi.eu www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE RI-261323 Sustainability of

201 views • 6 slides
Ontario Food and Nutrition Strategy Possibility Conversation Meeting #2 Ontario Collaborative

Ontario Food and Nutrition Strategy Possibility Conversation Meeting #2 Ontario Collaborative

Ontario Food and Nutrition Strategy Possibility Conversation Meeting #2 Ontario Collaborative Group on Healthy Eating and Physical Activity (OCGHEPA) November 28, 2011 Agenda for this session Welcome Update on the work of the OFNS

178 views • 16 slides

More recommend