visual exploratory data analysis
play

Visual exploratory data analysis pandas Foundations The iris data - PowerPoint PPT Presentation

Aug 06, 2023 •129 likes •636 views

PANDAS FOUNDATIONS Visual exploratory data analysis pandas Foundations The iris data set Famous data set in pa ern recognition 150 observations, 4 features each Sepal length Sepal width Petal length Petal

  1. PANDAS FOUNDATIONS Visual exploratory data analysis

  2. pandas Foundations The iris data set ● Famous data set in pa � ern recognition ● 150 observations, 4 features each ● Sepal length ● Sepal width ● Petal length ● Petal width ● 3 species: setosa, versicolor, virginica Source: R.A. Fisher, Annual Eugenics, 7, Part II, 179-188 (1936), h � p://archive.ics.uci.edu/ml/datasets/Iris

  3. pandas Foundations Data import In [1]: import pandas as pd In [2]: import matplotlib.pyplot as plt In [3]: iris = pd.read_csv('iris.csv', index_col=0) In [4]: print(iris.shape) (150, 5)

  4. pandas Foundations Line plot In [5]: iris.head() Out[5]: sepal_length sepal_width petal_length petal_width species 0 5.1 3.5 1.4 0.2 setosa 1 4.9 3.0 1.4 0.2 setosa 2 4.7 3.2 1.3 0.2 setosa 3 4.6 3.1 1.5 0.2 setosa 4 5.0 3.6 1.4 0.2 setosa In [6]: iris.plot(x='sepal_length', y='sepal_width') In [7]: plt.show()

  5. pandas Foundations Line plot

  6. pandas Foundations Sca � er plot In [8]: iris.plot(x='sepal_length', y='sepal_width', ...: kind='scatter') In [9]: plt.xlabel('sepal length (cm)') In [10]: plt.ylabel('sepal width (cm)') In [11]: plt.show()

  7. pandas Foundations Sca � er plot

  8. pandas Foundations Box plot In [12]: iris.plot(y='sepal_length’, kind='box') In [13]: plt.ylabel('sepal width (cm)') In [14]: plt.show()

  9. pandas Foundations Box plot

  10. pandas Foundations Histogram In [15]: iris.plot(y='sepal_length', kind='hist') In [16]: plt.xlabel('sepal length (cm)') In [17]: plt.show()

  11. pandas Foundations Histogram

  12. pandas Foundations Histogram options ● bins (integer): number of intervals or bins ● range (tuple): extrema of bins (minimum, maximum) ● normed (boolean): whether to normalize to one ● cumulative (boolean): compute Cumulative Distribution Function (CDF) ● … more Matplotlib customizations

  13. pandas Foundations Customizing histogram In [18]: iris.plot(y='sepal_length', kind='hist', ...: bins=30, range=(4,8), normed=True) In [19]: plt.xlabel('sepal length (cm)') In [20]: plt.show()

  14. pandas Foundations Customizing histogram

  15. pandas Foundations Cumulative distribution In [21]: iris.plot(y='sepal_length', kind='hist', bins=30, ...: range=(4,8), cumulative=True, normed=True) In [22]: plt.xlabel('sepal length (cm)') In [23]: plt.title('Cumulative distribution function (CDF)') In [24]: plt.show()

  16. pandas Foundations Cumulative distribution

  17. pandas Foundations Word of warning ● Three di ff erent DataFrame plot idioms ● iris.plot(kind=‘hist’) ● iris.plt.hist() ● iris.hist() ● Syntax/results di ff er! ● Pandas API still evolving: check documentation!

  18. PANDAS FOUNDATIONS Let’s practice!

  19. PANDAS FOUNDATIONS Statistical exploratory data analysis

  20. pandas Foundations Summarizing with describe() In [1]: iris.describe() # summary statistics Out[1]: sepal_length sepal_width petal_length petal_width count 150.000000 150.000000 150.000000 150.000000 mean 5.843333 3.057333 3.758000 1.199333 std 0.828066 0.435866 1.765298 0.762238 min 4.300000 2.000000 1.000000 0.100000 25% 5.100000 2.800000 1.600000 0.300000 50% 5.800000 3.000000 4.350000 1.300000 75% 6.400000 3.300000 5.100000 1.800000 max 7.900000 4.400000 6.900000 2.500000

  21. pandas Foundations Describe ● count : number of entries ● mean : average of entries ● std : standard deviation ● min: minimum entry ● 25% : first quartile ● 50% : median or second quartile ● 75% : third quartile ● max : maximum entry

  22. pandas Foundations Counts In [2]: iris['sepal_length'].count() # Applied to Series Out[2]: 150 In [3]: iris['sepal_width'].count() # Applied to Series Out[3]: 150 In [4]: iris[['petal_length', 'petal_width']].count() # Applied ...: to DataFrame Out[4]: petal_length 150 petal_width 150 dtype: int64 In [5]: type(iris[['petal_length', 'petal_width']].count()) # ...: returns Series Out[5]: pandas.core.series.Series

  23. pandas Foundations Averages In [6]: iris['sepal_length'].mean() # Applied to Series Out[6]: 5.843333333333335 In [7]: iris.mean() # Applied to entire DataFrame Out[7]: sepal_length 5.843333 sepal_width 3.057333 petal_length 3.758000 petal_width 1.199333 dtype: float64

  24. pandas Foundations Standard deviations In [8]: iris.std() Out[8]: sepal_length 0.828066 sepal_width 0.435866 petal_length 1.765298 petal_width 0.762238 dtype: float64

  25. pandas Foundations Mean and standard deviation on a bell curve

  26. pandas Foundations Medians In [9]: iris.median() Out[9]: sepal_length 5.80 sepal_width 3.00 petal_length 4.35 petal_width 1.30 dtype: float64

  27. pandas Foundations Medians & 0.5 quantiles In [10]: iris.median() Out[10]: sepal_length 5.80 sepal_width 3.00 petal_length 4.35 petal_width 1.30 dtype: float64 In [11]: q = 0.5 In [12]: iris.quantile(q) Out[12]: sepal_length 5.80 sepal_width 3.00 petal_length 4.35 petal_width 1.30 dtype: float64

  28. pandas Foundations Inter-quartile range (IQR) In [13]: q = [0.25, 0.75] In [14]: iris.quantile(q) Out[14]: sepal_length sepal_width petal_length petal_width 0.25 5.1 2.8 1.6 0.3 0.75 6.4 3.3 5.1 1.8

  29. pandas Foundations Ranges In [15]: iris.min() Out[15]: sepal_length 4.3 sepal_width 2 petal_length 1 petal_width 0.1 species setosa dtype: object In [16]: iris.max() Out[16]: sepal_length 7.9 sepal_width 4.4 petal_length 6.9 petal_width 2.5 species virginica dtype: object

  30. pandas Foundations Box plots In [17]: iris.plot(kind= 'box') Out[17]: <matplotlib.axes._subplots.AxesSubplot at 0x118a3d5f8> In [18]: plt.ylabel('[cm]') Out[18]: <matplotlib.text.Text at 0x118a524e0> In [19]: plt.show()

  31. pandas Foundations Box plots

  32. pandas Foundations Percentiles as quantiles In [20]: iris.describe() # summary statistics Out[20]: sepal_length sepal_width petal_length petal_width count 150.000000 150.000000 150.000000 150.000000 mean 5.843333 3.057333 3.758000 1.199333 std 0.828066 0.435866 1.765298 0.762238 min 4.300000 2.000000 1.000000 0.100000 25% 5.100000 2.800000 1.600000 0.300000 50% 5.800000 3.000000 4.350000 1.300000 75% 6.400000 3.300000 5.100000 1.800000 max 7.900000 4.400000 6.900000 2.500000

  33. PANDAS FOUNDATIONS Let’s practice!

  34. PANDAS FOUNDATIONS Separating populations

  35. pandas Foundations In [1]: iris.head() Out[1]: sepal_length sepal_width petal_length petal_width species 0 5.1 3.5 1.4 0.2 setosa 1 4.9 3.0 1.4 0.2 setosa 2 4.7 3.2 1.3 0.2 setosa 3 4.6 3.1 1.5 0.2 setosa 4 5.0 3.6 1.4 0.2 setosa

  36. pandas Foundations Describe species column In [2]: iris['species'].describe() Out[2]: count: # non-null entries count 150 unique: # distinct values unique 3 top: most frequent category top setosa freq: # occurrences of top freq 50 Name: species, dtype: object

  37. pandas Foundations Unique & factors In [3]: iris['species'].unique() Out[3]: array(['setosa', 'versicolor', 'virginica'], dtype=object)

  38. pandas Foundations Filtering by species In [4]: indices = iris['species'] == 'setosa' In [5]: setosa = iris.loc[indices,:] # extract new DataFrame In [6]: indices = iris['species'] == 'versicolor' In [7]: versicolor = iris.loc[indices,:] # extract new DataFrame In [8]: indices = iris['species'] == 'virginica' In [9]: virginica = iris.loc[indices,:] # extract new DataFrame

  39. pandas Foundations Checking species In [10]: setosa['species'].unique() Out[10]: array(['setosa'], dtype=object) In [11]: versicolor['species'].unique() Out[11]: array(['versicolor'], dtype=object) In [12]: virginica['species'].unique() Out[12]: array(['virginica'], dtype=object) In [13]: del setosa['species'], versicolor['species'], ...: virginica['species']

  40. pandas Foundations Checking indexes In [14]: setosa.head(2) Out[14]: sepal_length sepal_width petal_length petal_width 0 5.1 3.5 1.4 0.2 1 4.9 3.0 1.4 0.2 In [15]: versicolor.head(2) Out[15]: sepal_length sepal_width petal_length petal_width 50 7.0 3.2 4.7 1.4 51 6.4 3.2 4.5 1.5 In [16]: virginica.head(2) Out[16]: sepal_length sepal_width petal_length petal_width 100 6.3 3.3 6.0 2.5 101 5.8 2.7 5.1 1.9

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Subgroup Discovery Exploratory Data Analysis Exploratory Data Analysis Classification:

Subgroup Discovery Exploratory Data Analysis Exploratory Data Analysis Classification:

Subgroup Discovery Exploratory Data Analysis Exploratory Data Analysis Classification: model the dependence of the target on the remaining attributes. problem: sometimes uses only some of the available dependencies, or classifier is

1.64k views • 77 slides
VISUALIZATION Jeff Goldsmith, PhD Department of Biostatistics 1 Exploratory data analysis

VISUALIZATION Jeff Goldsmith, PhD Department of Biostatistics 1 Exploratory data analysis

VISUALIZATION Jeff Goldsmith, PhD Department of Biostatistics 1 Exploratory data analysis Exploratory analysis is a loosely-defined process Roughly, the stuff between loading data and formal analysis is exploratory This

407 views • 15 slides
Exploratory Data Analysis Exploratory Data Analysis for Ecological Modelling and for Ecological

Exploratory Data Analysis Exploratory Data Analysis for Ecological Modelling and for Ecological

Exploratory Data Analysis Exploratory Data Analysis for Ecological Modelling and for Ecological Modelling and Decision Support Decision Support Gennady Andrienko &amp; Natalia Andrienko Fraunhofer Institute AIS Sankt Augustin Germany

937 views • 35 slides
Introduction to Data Science: x (1) x 1 x 2 x ( n ) x i n 1 1 Size: size

Introduction to Data Science: x (1) x 1 x 2 x ( n ) x i n 1 1 Size: size

Exploratory Data Analysis Exploratory Data Analysis: Summary Statistics Exploratory Data Analysis: Summary Statistics EDA with the grammar of graphics Exploratory Data Analysis: Summary Statistics Exploratory Data Analysis: Summary Statistics

1.27k views • 90 slides
Sifting Through Lines, Events, and Trees: Stories of EHR Visual Exploratory Analysis

Sifting Through Lines, Events, and Trees: Stories of EHR Visual Exploratory Analysis

5/28/2014 Sifting Through Lines, Events, and Trees: Stories of EHR Visual Exploratory Analysis Christopher Herrick, Vivian Gainer, Alyssa Goodson, Shawn Murphy Victor Castro, Kay Mahamaneerat, Yanbing Wang, and Taowei David Wang

258 views • 8 slides
Exploratory Data Analysis Maneesh Agrawala CS 448B: Visualization Fall 2018 1 A2: Exploratory

Exploratory Data Analysis Maneesh Agrawala CS 448B: Visualization Fall 2018 1 A2: Exploratory

Exploratory Data Analysis Maneesh Agrawala CS 448B: Visualization Fall 2018 1 A2: Exploratory Data Analysis Use Tableau to formulate &amp; answer questions First steps Step 1: Pick a domain Step 2: Pose questions Step 3: Find data

297 views • 26 slides
What Is the Output of What Is the Output of Visual Data Analysis? Visual Data Analysis? Gennady

What Is the Output of What Is the Output of Visual Data Analysis? Visual Data Analysis? Gennady

What Is the Output of What Is the Output of Visual Data Analysis? Visual Data Analysis? Gennady &amp; Natalia Andrienko Fraunhofer Institute IAIS, Sankt Augustin, Germany http://www.ais.fraunhofer.de/and Visual Analytics Panel @ InfoVis

315 views • 9 slides
Exploratory Data Analysis Nam Wook Kim Mini-Courses January @ GSAS 2018 Goal Learn the

Exploratory Data Analysis Nam Wook Kim Mini-Courses January @ GSAS 2018 Goal Learn the

Download Tableau &amp; H-1B petition data Exploratory Data Analysis Nam Wook Kim Mini-Courses January @ GSAS 2018 Goal Learn the Philosophy of Exploratory Data Analysis Exposure, the e ff ective laying open of the data to display the

797 views • 50 slides
CME/STATS 195 CME/STATS 195 Lecture 5: Exploratory Data Analysis Lecture 5: Exploratory Data

CME/STATS 195 CME/STATS 195 Lecture 5: Exploratory Data Analysis Lecture 5: Exploratory Data

CME/STATS 195 CME/STATS 195 Lecture 5: Exploratory Data Analysis Lecture 5: Exploratory Data Analysis Evan Rosenman Evan Rosenman April 16, 2019 April 16, 2019 6.5 Contents Contents Missing values Exploratory Data Analysis Variation

746 views • 55 slides
2. Exploratory Data Analysis (Chapter 1.6) 1/22/2020 Quiz 1 - Data and where it comes from A

2. Exploratory Data Analysis (Chapter 1.6) 1/22/2020 Quiz 1 - Data and where it comes from A

Unit 1: Introduction to Data 2. Exploratory Data Analysis (Chapter 1.6) 1/22/2020 Quiz 1 - Data and where it comes from A sampling metaphor When you taste a spoonful of soup and decide the spoonful you tasted isnt salty enough, thats

198 views • 15 slides
Exploratory Data Analysis Ma Maneesh Agrawala CS 448B: Visualization Winter 2020 1 A2:

Exploratory Data Analysis Ma Maneesh Agrawala CS 448B: Visualization Winter 2020 1 A2:

Exploratory Data Analysis Ma Maneesh Agrawala CS 448B: Visualization Winter 2020 1 A2: Exploratory Data Analysis Use Tableau to formulate &amp; answer questions First steps Step 1: Pick domain &amp; data Step 2: Pose questions Step 3:

522 views • 24 slides
Exploratory Data Analysis Ma Maneesh Agrawala CS 448B: Visualization Fall 2020 1 A2:

Exploratory Data Analysis Ma Maneesh Agrawala CS 448B: Visualization Fall 2020 1 A2:

Exploratory Data Analysis Ma Maneesh Agrawala CS 448B: Visualization Fall 2020 1 A2: Exploratory Data Analysis Use Tableau to formulate &amp; answer questions First steps Step 1: Pick domain &amp; data Step 2: Pose questions Step 3:

239 views • 23 slides
Exploratory Data Analysis Paul Cohen ISTA 370 Spring, 2012 Paul Cohen ISTA 370 () Exploratory

Exploratory Data Analysis Paul Cohen ISTA 370 Spring, 2012 Paul Cohen ISTA 370 () Exploratory

Exploratory Data Analysis Paul Cohen ISTA 370 Spring, 2012 Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, 2012 1 / 46 Outline Data, revisited The purpose of exploratory data analysis Learning to see Paul Cohen ISTA 370 ()

1.05k views • 46 slides
DATA MINING LECTURE 2 Data Preprocessing Exploratory Analysis Post-processing What is Data

DATA MINING LECTURE 2 Data Preprocessing Exploratory Analysis Post-processing What is Data

DATA MINING LECTURE 2 Data Preprocessing Exploratory Analysis Post-processing What is Data Mining? Data mining is the use of efficient techniques for the analysis of very large collections of data and the extraction of useful and possibly

1.38k views • 46 slides
Unit 1: Introduction to data 3. More exploratory data analysis STA 104 - Summer 2017 PS 1 is

Unit 1: Introduction to data 3. More exploratory data analysis STA 104 - Summer 2017 PS 1 is

Announcements Unit 1: Introduction to data 3. More exploratory data analysis STA 104 - Summer 2017 PS 1 is posted in Sakai, due this Tuesday at 12.30pm. Duke University, Department of Statistical Science Prof. van den Boom Slides posted at

77 views • 3 slides
CS 109a: Data Science Effective Exploratory Data Analysis and Visualization Pavlos Protopapas,

CS 109a: Data Science Effective Exploratory Data Analysis and Visualization Pavlos Protopapas,

CS 109a: Data Science Effective Exploratory Data Analysis and Visualization Pavlos Protopapas, Kevin Rader, Rahul Dave, Margo Levine Ask an interesting What is the scientific goal ? What would you do if you had all the data ? question. What do

1.31k views • 129 slides
Management of Acute Poisoning : General approaches RPh Adilah Mohamed Ariff National Poison

Management of Acute Poisoning : General approaches RPh Adilah Mohamed Ariff National Poison

Management of Acute Poisoning : General approaches RPh Adilah Mohamed Ariff National Poison Centre Universiti Sains Malaysia Introduction Basic knowledge and practices in managing acute poisoning. Understanding this basic and general

919 views • 45 slides
DEVELOPMENT OF A USER FRIENDLY FRAMEWORK FOR GEOSPATIAL IDENTIFICATION OF POTENTIAL PFAS SOURCE

DEVELOPMENT OF A USER FRIENDLY FRAMEWORK FOR GEOSPATIAL IDENTIFICATION OF POTENTIAL PFAS SOURCE

DEVELOPMENT OF A USER FRIENDLY FRAMEWORK FOR GEOSPATIAL IDENTIFICATION OF POTENTIAL PFAS SOURCE ZONES Jennifer Guelfo, PhD June 12, 2017 1 PRESENTATION OVERVIEW PFAS PFAS in U.S. Source Zone Overview Drinking H 2 O EvaluaJon

246 views • 21 slides
Chemistry 121(01) Winter 2009 Introduction to Organic Chemistry and Biochemistry Introduction to

Chemistry 121(01) Winter 2009 Introduction to Organic Chemistry and Biochemistry Introduction to

Chemistry 121(01) Winter 2009 Introduction to Organic Chemistry and Biochemistry Introduction to Organic Chemistry and Biochemistry Instructor Dr. Upali Siriwardane (Ph.D. Ohio State) E-mail: upali@chem.latech.edu Office: 311 Carson Taylor

628 views • 16 slides
UV Disinfection UV Disinfection Nick Landes Nick Landes March 20, 2019 Agenda What is UV How

UV Disinfection UV Disinfection Nick Landes Nick Landes March 20, 2019 Agenda What is UV How

The Basics of UV Disinfection UV Disinfection Nick Landes Nick Landes March 20, 2019 Agenda What is UV How does UV Disinfect Types of UV Systems Design Considerations What is Ultraviolet Energy 750 nm 390 nm Visible Spectrum UV

453 views • 13 slides
Contents Introduction Have a Look at Data Explore Individual Variables Explore Multiple

Contents Introduction Have a Look at Data Explore Individual Variables Explore Multiple

Data Exploration and Visualisation with R Yanchang Zhao http://www.RDataMining.com R and Data Mining Course Beijing University of Posts and Telecommunications, Beijing, China July 2019 Chapter 3: Data Exploration, in R and Data Mining:

924 views • 45 slides
Anthony Kougkas, Hariharan Devarajan, Xian-He Sun akougkas@hawk.iit.edu Department of Computer

Anthony Kougkas, Hariharan Devarajan, Xian-He Sun akougkas@hawk.iit.edu Department of Computer

IRIS: I/O Redirection via Integrated Storage Anthony Kougkas, Hariharan Devarajan, Xian-He Sun akougkas@hawk.iit.edu Department of Computer Science Special thanks to Dr. Shuibing He Illinois Institute of Technology ICS18, Beijing, China who

338 views • 29 slides
Iris: a framework for higher-order concurrent separation logic in Coq Robbert Krebbers 1 Delft

Iris: a framework for higher-order concurrent separation logic in Coq Robbert Krebbers 1 Delft

Iris: a framework for higher-order concurrent separation logic in Coq Robbert Krebbers 1 Delft University of Technology, The Netherlands January 15, 2017 @ TTT, Paris, France 1 Iris is joint work with: Ralf Jung, Jacques-Hendri Jourdan, Ale s

1.49k views • 124 slides
Demonstration of the Iris separation logic in Coq Robbert Krebbers 1 Delft University of

Demonstration of the Iris separation logic in Coq Robbert Krebbers 1 Delft University of

Demonstration of the Iris separation logic in Coq Robbert Krebbers 1 Delft University of Technology, The Netherlands January 21, 2017 @ Coq PL, Paris, France 1 Iris is joint work with: Ralf Jung, Jacques-Hendri Jourdan, Ale s Bizjak, David

948 views • 51 slides

More recommend