Data Cleaning Duen Horng (Polo) Chau Assistant Professor Associate - PowerPoint PPT Presentation

Nov 12, 2022 •162 likes •316 views

http://poloclub.gatech.edu/cse6242 CSE6242 / CX4242: Data & Visual Analytics Data Cleaning Duen Horng (Polo) Chau Assistant Professor Associate Director, MS Analytics Georgia Tech Partly based on materials by

http://poloclub.gatech.edu/cse6242   CSE6242 / CX4242: Data & Visual Analytics   Data Cleaning Duen Horng (Polo) Chau   Assistant Professor   Associate Director, MS Analytics   Georgia Tech Partly based on materials by   Professors Guy Lebanon, Jeffrey Heer, John Stasko, Christos Faloutsos
Data Cleaning   How dirty is real data?
  How dirty is real data? Examples • Jan 19, 2016 • January 19, 16 • 1/19/16 • 2006-01-19 • 19/1/16 3 http://blogs.verdantis.com/wp-content/uploads/2015/02/Data-cleansing.jpg
How dirty is real data? Discuss with you neighbors (group of 2-3) 60 seconds Comes up with 5+ kinds of “data dirtiness” 4
How dirty is real data? • spelling errors • missing data • different units/measurements • leading zeros… • wrong data types • cases lower/upper • inconsistent (last name/first name order exchange) • duplication • language writing order • different “null” • white spaces • big/little endian (maybe) 5
Importance of Data Cleaning
“80%” Time Spent on Data Preparation Cleaning Big Data: Most Time-Consuming, Least Enjoyable Data Science Task, Survey Says [Forbes]   http://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time- consuming-least-enjoyable-data-science-task-survey-says/#73bf5b137f75 7
Data Janitor
Writing “Clean Code” • Be careful with trailing whitespaces • Indent code ( spaces vs tabs ) following coding practices in your team/company   https://google.github.io/styleguide/javaguide.html#s4.2-block-indentation …there’s no way I'm going to be with someone who uses spaces over tabs… http://www.businessinsider.com/tabs-vs-spaces-from-silicon-valley-2016-5 Trailing whitespace is evil. Don't commit evil into your repo. http://codeimpossible.com/2012/04/02/trailing-whitespace-is-evil-don-t-commit-evil-into-your-repo/ 9
10
Data Cleaners Watch videos • Data Wrangler (research at Stanford) • Open Refine (previously Google Refine ) Write down • Examples of data dirtiness • Tool’s features demo-ed (or that you like) Will collectively summarize similarities and differences afterwards Open Refine : http://openrefine.org Data Wrangler : http://vis.stanford.edu/wrangler/ 11
What can Open Refine and Wrangler do? O = Open Refine W = Data wrangler 14
! The videos only show some of the tools’ features. Try them out. Open Refine : http://openrefine.org Data Wrangler : http://vis.stanford.edu/wrangler/ 15

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Data cleaning and standardisation (1) Data cleaning and standardisation (2) Real world data is

Data cleaning and standardisation Febrl A parallel open source record linkage and geocoding system Record linkage and data integration Febrl overview Peter Christen Probabilistic data cleaning and standardisation Data Mining Group,

663 views • 5 slides

Workflow basics, RMarkdown, git/Github Cleaning up Cleaning up Cleaning up Cleaning up

Workflow basics, RMarkdown, git/Github Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Don't worry, your history is preserved Settings Cleaning up Cleaning up Lots more options! RMarkdown Structure of an Rmd

656 views • 50 slides

Data Mining: A Powerful Data Mining: A Powerful Tool for Data Cleaning Tool for Data Cleaning

Data Mining: A Powerful Data Mining: A Powerful Tool for Data Cleaning Tool for Data Cleaning Jiawei Han Department of Computer Science University of Illinois at Urbana-Champaign Nov. 4, 2003 1 Data mining for data quality assurance

721 views • 40 slides

Concatenating data Cleaning Data in Python Combining data Data may not always come in 1

CLEANING DATA IN PYTHON Concatenating data Cleaning Data in Python Combining data Data may not always come in 1 huge file 5 million row dataset may be broken into 5 separate datasets Easier to store and share May have

425 views • 26 slides

MORPH-II Dataset 1. Introduction to the Data 2. Inconsistencies in the Data 3. Cleaning the Data

Garrett Bingham & Ben Yip Summary and Cleaning June 16, 2017 University of North Carolina Wilmington MORPH-II Dataset 1. Introduction to the Data 2. Inconsistencies in the Data 3. Cleaning the Data 4. New Datasets 5. Dirty Data 6.

652 views • 26 slides

Intro to data cleaning with Apache Spark CLEAN IN G DATA W ITH P YS PARK Mike Metzger Data

Intro to data cleaning with Apache Spark CLEAN IN G DATA W ITH P YS PARK Mike Metzger Data Engineering Consultant What is Data Cleaning? Data Cleaning : Preparing raw data for use in data processing pipelines. Possible tasks in data

1.03k views • 20 slides

Diagnose data for cleaning Cleaning Data in Python Cleaning data Prepare data for analysis

CLEANING DATA IN PYTHON Diagnose data for cleaning Cleaning Data in Python Cleaning data Prepare data for analysis Data almost never comes in clean Diagnose your data for problems Cleaning Data in Python Common data problems

1.08k views • 29 slides

Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang,

Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Data Cleaning? 2 Data is Dirty 2 incomplete inconsistent inaccurate Data is Dirty 2 incomplete 25% companies: flawed data

1.07k views • 106 slides

Diagnose data for cleaning CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Cleaning data

Diagnose data for cleaning CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Cleaning data Prepare data for analysis Data almost never comes in clean Diagnose your data for problems CLEANING DATA IN PYTHON Common data problems

780 views • 38 slides

Data types Cleaning Data in Python Prepare and clean data Cleaning Data in Python Data types

CLEANING DATA IN PYTHON Data types Cleaning Data in Python Prepare and clean data Cleaning Data in Python Data types In [1]: print(df.dtypes) name object sex object treatment a object treatment b int64 dtype:

714 views • 44 slides

Data Preparation Data cleaning Data integration and transformation (Data

Data Preprocessing Why preprocess the data? Data Preparation Data cleaning Data integration and transformation (Data preprocessing) Data reduction, Feature selection Discretization and concept hierarchy generation 2

891 views • 21 slides

ASR Data Cleaning Guidelines Presenter: Asima Hameed Data Cleaning Process The speech files will

ASR Data Cleaning Guidelines Presenter: Asima Hameed Data Cleaning Process The speech files will be cleaned on the basis of: Silence Noise Mispronunciation Silence 1. A silence period of at least 200 ms 2. For voiced and voiceless

600 views • 26 slides

Outline Data cleaning and standardisation Febrl A parallel open source data linkage and

Outline Data cleaning and standardisation Febrl A parallel open source data linkage and geocoding system Data linkage and data integration Febrl overview Peter Christen, Tim Churches and others Probabilistic data cleaning and

833 views • 8 slides

DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING "Data cleaning and

DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING "Data cleaning and repairing account for about 60% of the work of data scientists." Christian Kaestner Required reading: Schelter, S., Lange, D., Schmidt, P.,

1.1k views • 86 slides

Data Handling: Import, Cleaning and Visualisation Lecture 7: Data Sources, Data Gathering, Data

9/12/2019 Data Handling: Import, Cleaning and Visualisation Data Handling: Import, Cleaning and Visualisation Lecture 7: Data Sources, Data Gathering, Data Import Prof. Dr. Ulrich Matter 24/10/2019 file:///home/umatter/Dropbox/T

1.06k views • 54 slides

Data Preparation Data cleaning Discretization (Data preprocessing) Data

Data Preprocessing Why preprocess the data? Data Preparation Data cleaning Discretization (Data preprocessing) Data integration and transformation Data reduction, Feature selection 2 Why Prepare Data? Why Prepare

462 views • 15 slides

110 Announcements Announcements - Houses How-to use Zoom for Office-hours Video Posted on

110 Announcements Announcements - Houses How-to use Zoom for Office-hours Video Posted on Course Site We will be doing house zooms today! Join yours via Sakai > Postem Password: comp110! Additionally, houses have setup GroupMes

318 views • 8 slides

Data Cleaning Mahdi Roozbahani Lecturer, Computational Science and Engineering, Georgia Tech

CX4242: Data Cleaning Mahdi Roozbahani Lecturer, Computational Science and Engineering, Georgia Tech Data Cleaning How dirty is real data? How dirty is real data? Examples Jan 19, 2016 January 19, 16 1/19/16 2006-01-19

763 views • 19 slides

New developments in Singular H. Sch onemann University of Kaiserslautern, Germany 2016/03/31

New developments in Singular H. Sch onemann University of Kaiserslautern, Germany 2016/03/31 1 / 43 What is Singular ? A computer algebra system for polynomial computations, with special emphasis on algebraic geometry, commutative and

557 views • 43 slides

CERN OHL version 2 Tristan Gingold Document reference 2 13/03/2019 Foreword... I am not a

CERN OHL version 2 Tristan Gingold Document reference 2 13/03/2019 Foreword... I am not a lawyer, I just want to explain CERN OHLv2 US: to license, a license UK: to license, a licence Document reference 3 07/03/2019 Why a

300 views • 18 slides

CS1150 Principles of Computer Science Introduction Yanyan Zhuang Department of Computer Science

CS1150 Principles of Computer Science Introduction Yanyan Zhuang Department of Computer Science http://www.cs.uccs.edu/~yzhuang CS1150 UC. Colorado Springs Intro of Intro Yanyan Zhuang o PhD 2012 o yzhuang@uccs.edu o Office ENGR 184

513 views • 29 slides

Python basics for NLP Type : which python3 in the command prompt

Programming environment Is Python installed Python basics for NLP Type : which python3 in the command prompt (applications->utilitaires->Terminal) Vincent Claveau It should answer: /usr/bin/python3 IRISA-CNRS Text file

540 views • 4 slides

Coding with dunetpc and LArSoft: tips, tricks, gotchas, and debugging advice Tom Junk DUNE

Coding with dunetpc and LArSoft: tips, tricks, gotchas, and debugging advice Tom Junk DUNE Computing Tutorial November 14, 2017 The Tutorial Example Instructions are available at the bottom of the homework link:

683 views • 35 slides

Lecture #13: More Sequences and Strings Last modified: Tue Mar 18 16:17:54 2014 CS61A: Lecture

Lecture #13: More Sequences and Strings Last modified: Tue Mar 18 16:17:54 2014 CS61A: Lecture #13 1 Odds and Ends: Multi-Argument Map Pythons built-in map function actually applies a function to one or more sequences: >>> from

461 views • 21 slides