CSE217 INTRODUCTION TO DATA SCIENCE LECTURE 2: EXPLORATORY DATA - - PowerPoint PPT Presentation

▶
cse217 introduction to data science
SMART_READER_LITE
LIVE PREVIEW

CSE217 INTRODUCTION TO DATA SCIENCE LECTURE 2: EXPLORATORY DATA - - PowerPoint PPT Presentation

Sep 15, 2023 171 likes •432 views

CSE217 INTRODUCTION TO DATA SCIENCE LECTURE 2: EXPLORATORY DATA ANALYSIS Spring 2019 Marion Neumann RECAP: WHAT IS DATA SCIENCE? solving problems with data scientific, collect & clean & use data social, or data understand

slide-1
SLIDE 1

CSE217 INTRODUCTION TO DATA SCIENCE

Spring 2019 Marion Neumann

LECTURE 2: EXPLORATORY DATA ANALYSIS

slide-2
SLIDE 2

RECAP: WHAT IS DATA SCIENCE?

2

…solving problems with data…

collect & understand data clean & format data data problem use data to create solution scientific, social, or business problem f

data analysis and/or machine learning

slide-3
SLIDE 3

WHERE DOES DATA COME FROM?

  • Internal Sources
  • business-centric data in organizational data bases recording day to day operations
  • scientific or experimental data
  • Existing External Sources à data is available for free or a fee
  • public government databases, stock market data, Yelp reviews
  • usually (somewhat) pre-processed
  • Collect your own data à beyond the scope of this course
  • Online Data à typically raw data
  • from APIs (e.g. Google Map API, Facebook API, Twitter API)
  • web scraping: using software, scripts or by-hand extracting data from what is

displayed on a page or what is contained in the HTML file

3

Caution: not all data that is accessible is good to be used!

  • Are you violating their terms of service?
  • Privacy concerns for website and their clients?
  • Do they have an API or fee that you are bypassing?
  • Are they willing to share this data?
slide-4
SLIDE 4
  • Types of Variables
  • Data Types

VARIABLES AND DATA TYPES

4

Example:

https://www.zillow.com

numeric

2

Order

continuous or

discrete

categorical

no

  • rder

binary

categorical w 2 categories

t.ES No integer

discrete categorical binary

Boolean

binary

we prefernumeric

floatingpoint

continuous

1 datatypes

arrays

string formatted text

categorical free form

text

compound datatypes lists dictionaries arrays

slide-5
SLIDE 5

DATA(SET) REPRESENTATION

  • Tables (csv, xlsx etc.)
  • two-dimensional representation
  • rows represent data records
  • columns represents one type of measurement
  • Structured Data (json, xml etc.)
  • complex and multi-tiered dictionary
  • Semi-structured Data (.txt)
  • flat text representation with known structure
  • data can be easily parsed
  • Unstructured Data (.txt)
  • prose text

5

slide-6
SLIDE 6

DATA IS (ALWAYS) MESSY

  • Common issues with data:
  • missing values: how do we fill in?
  • wrong values: how can we detect and correct?
  • messy format/representation
  • Example: number of produce deliveries over a weekend

6

Common causes of messiness:

  • variables/features are stored in both rows and columns
  • multiple features are stored in one column
  • multiple types of experimental units stored in same table
slide-7
SLIDE 7

DATA (PRE-)PROCESSING

Goal: bring data in a format we can use for analysis (and/or machine learning)

à use a format that is good for Python J (e.g. 2d arrays) à recall from last lecture: data points vs features/variables

  • Data Parsing and Formatting
  • Data Profiling à asses data amount and quality
  • Data Cleaning
  • Data Engineering (more later in this course…)
  • detect outliers
  • feature engineering
  • data augmentation

7

data wrangling

slide-8
SLIDE 8

DATA ≠ DATA

  • Two kinds of data: population vs. sample
  • What are problems with sample data?

8

A population is the entire set

  • f objects or events under
  • study. Population can be

hypothetical “all students” or all students in this class. A sample is a (representative) subset of the objects or events under study. à needed because it’s impossible or intractable to

  • btain or use population data.
slide-9
SLIDE 9

EXPLORATORY DATA ANALYSIS (EDA)

Different ways of exploring data:

  • explore each individual variable in the dataset
  • summary statistics
  • spread
  • distribution
  • assess interactions between variables (or between

individual variables and the target)

  • correlation, analysis of variance (ANOVA)
  • explore data across many dimensions (more later in this course…)
  • clustering
  • dimensionality reduction (e.g. principal component analysis

(PCA), etc.)

9

slide-10
SLIDE 10

SUMMARY STATISTICS

  • (sample) mean
  • (sample) median
  • Example: Ages: 17, 19, 21, 22, 23, 23, 23, 38

What is the median age? What is the mean/average age?

  • mean vs median
  • which one is easier/more efficient to compute?

10

Caution: the mean is sensitive to outliers! Caution: consider practicality (efficiency) of implementation!

slide-11
SLIDE 11

SUMMARY STATISTICS

  • mode = variable that occurs most often
  • useful for categorical variables

à visualize with a bar plot

11

DSFS Ch3

slide-12
SLIDE 12

MEASURES OF SPREAD

  • range = max value – min value
  • variance
  • Caution: does not have the same unit as xi
  • standard deviation

Why is measuring the spread important?

12

slide-13
SLIDE 13

DATA VISUALIZATION

  • Can summary statistics and measures of spread

tell us everything?

13

slide-14
SLIDE 14

DATA VISUALIZATION

  • Can summary statistics and measures of spread

tell us everything?

14

slide-15
SLIDE 15

TYPES OF VISUALIZATION

  • distribution

à how does a variable distribute over a range of possible values

  • relationship

à how do the values of multiple variables in the dataset relate

  • comparison

à how do trends in multiple variable or datasets compare

  • composition

à how does the dataset break down into subgroups

15

slide-16
SLIDE 16

VISUALIZE DISTRIBUTION

  • histogram

16

Caution: Trends in histograms are sensitive to the number of bins.

PDSH p245

slide-17
SLIDE 17

VISUALIZE RELATIONSHIP

  • scatter plot
  • distribution of two variables
  • relationship between two variables

17

DSFS Ch3 PDSH p233

slide-18
SLIDE 18

VISUALIZE COMPARISONS

  • multiple histograms
  • visualize how different variables compare (or how a

variable differs over specific groups) à we can also use box plots to compare different variables

18

slide-19
SLIDE 19

VISUALIZE COMPOSITION/COMPARISON

  • box plots
  • compare different variables à cf. Lab1
  • compare a quantitative variable across groups

à highlights the range, quartiles, median and outliers

19

This plot illustrates composition, since it looks at classes/categories

  • f one variable.

Lab1

slide-20
SLIDE 20
  • pie chart
  • stacked area graph

VISUALIZE COMPOSITION

20

Visualize trend

  • ver time!
slide-21
SLIDE 21

ACTIVITY 2

  • TASK 1: What do the following plots produced in

Lab1 visualize?

  • TASK 2: Which of the following visualizations are

good/proper visualization and which do you think are problematic (and why)?

21

Caution: Not all visualizations are good visualizations.

slide-22
SLIDE 22

MORE DIMENSIONS

  • How about relationship between 3 variables?

à 3D is not always better

22

slide-23
SLIDE 23

CATEGORICAL VARIABLES

  • use color coding for categorical variables

23

Data visualization can help figure out what we need to predict class labels! pedal_length sepal_length

slide-24
SLIDE 24

24

  • DSFS
  • Ch3: Visualizing Data (matplotlib, bar/line charts, scatter plots)
  • PDSH
  • Ch4: Visualization with Matplotlib
  • plotting with matplotlib (p217-221)
  • scatter plots (p233-237)
  • histograms (p245-247)

SUMMARY & READING

  • EDA process
  • (pre-)process data
  • summarize data
  • present/visualize distribution and relationships
  • EDA goals
  • develop/find hypothesis/question(s) to be investigated
  • use data to answer the question(s)