summary statistics
play

Summary statistics DATA MAN IP ULATION W ITH PAN DAS Maggie - PowerPoint PPT Presentation

Feb 28, 2023 •134 likes •472 views

Summary statistics DATA MAN IP ULATION W ITH PAN DAS Maggie Matsui Content Developer at DataCamp Summarizing numerical data .median() , .mode() dogs["height_cm"].mean() .min() , .max() .var() , .std() 49.714285714285715 .sum()

  1. Summary statistics DATA MAN IP ULATION W ITH PAN DAS Maggie Matsui Content Developer at DataCamp

  2. Summarizing numerical data .median() , .mode() dogs["height_cm"].mean() .min() , .max() .var() , .std() 49.714285714285715 .sum() .quantile() DATA MANIPULATION WITH PANDAS

  3. Summarizing dates Oldest dog: dogs["date_of_birth"].min() '2011-12-11' Youngest dog: dogs["date_of_birth"].max() '2018-02-27' DATA MANIPULATION WITH PANDAS

  4. The .agg() method def pct30(column): return column.quantile(0.3) dogs["weight_kg"].agg(pct30) 22.599999999999998 DATA MANIPULATION WITH PANDAS

  5. Summaries on multiple columns dogs[["weight_kg", "height_cm"]].agg(pct30) weight_kg 22.6 height_cm 45.4 dtype: float64 DATA MANIPULATION WITH PANDAS

  6. Multiple summaries def pct40(column): return column.quantile(0.4) dogs["weight_kg"].agg([pct30, pct40]) pct30 22.6 pct40 24.0 Name: weight_kg, dtype: float64 DATA MANIPULATION WITH PANDAS

  7. Cumulative sum dogs["height_cm"] dogs["height_cm"].cumsum() 0 56 0 56 1 43 1 99 2 46 2 145 3 49 3 194 4 59 4 253 5 18 5 271 6 77 6 348 Name: height_cm, dtype: int64 Name: height_cm, dtype: int64 DATA MANIPULATION WITH PANDAS

  8. Cumulative statistics .cummax() .cummin() .cumprod() DATA MANIPULATION WITH PANDAS

  9. Walmart sales.head() store type dept date weekly_sales is_holiday temp_c fuel_price unemp 0 1 A 1 2010-02-05 24924.50 False 5.73 0.679 8.106 1 1 A 2 2010-02-05 50605.27 False 5.73 0.679 8.106 2 1 A 3 2010-02-05 13740.12 False 5.73 0.679 8.106 3 1 A 4 2010-02-05 39954.04 False 5.73 0.679 8.106 4 1 A 5 2010-02-05 32229.38 False 5.73 0.679 8.106 DATA MANIPULATION WITH PANDAS

  10. Let's practice! DATA MAN IP ULATION W ITH PAN DAS

  11. Counting DATA MAN IP ULATION W ITH PAN DAS Maggie Matsui Content Developer at DataCamp

  12. Avoiding double counting DATA MANIPULATION WITH PANDAS

  13. Vet visits print(vet_visits) date name breed weight_kg 0 2018-09-02 Bella Labrador 24.87 1 2019-06-07 Max Labrador 28.35 2 2018-01-17 Stella Chihuahua 1.51 3 2019-10-19 Lucy Chow Chow 24.07 .. ... ... ... ... 71 2018-01-20 Stella Chihuahua 2.83 72 2019-06-07 Max Chow Chow 24.01 73 2018-08-20 Lucy Chow Chow 24.40 74 2019-04-22 Max Labrador 28.54 DATA MANIPULATION WITH PANDAS

  14. Dropping duplicate names vet_visits.drop_duplicates(subset="name") date name breed weight_kg 0 2018-09-02 Bella Labrador 24.87 1 2019-06-07 Max Chow Chow 24.01 2 2019-03-19 Charlie Poodle 24.95 3 2018-01-17 Stella Chihuahua 1.51 4 2019-10-19 Lucy Chow Chow 24.07 7 2019-03-30 Cooper Schnauzer 16.91 10 2019-01-04 Bernie St. Bernard 74.98 (6 2019-06-07 Max Labrador 28.35) DATA MANIPULATION WITH PANDAS

  15. Dropping duplicate pairs unique_dogs = vet_visits.drop_duplicates(subset=["name", "breed"]) print(unique_dogs) date name breed weight_kg 0 2018-09-02 Bella Labrador 24.87 1 2019-03-13 Max Chow Chow 24.13 2 2019-03-19 Charlie Poodle 24.95 3 2018-01-17 Stella Chihuahua 1.51 4 2019-10-19 Lucy Chow Chow 24.07 6 2019-06-07 Max Labrador 28.35 7 2019-03-30 Cooper Schnauzer 16.91 10 2019-01-04 Bernie St. Bernard 74.98 DATA MANIPULATION WITH PANDAS

  16. Easy as 1, 2, 3 unique_dogs["breed"].value_counts() unique_dogs["breed"].value_counts(sort=Tru Labrador 2 Labrador 2 Schnauzer 1 Chow Chow 2 St. Bernard 1 Schnauzer 1 Chow Chow 2 St. Bernard 1 Poodle 1 Poodle 1 Chihuahua 1 Chihuahua 1 Name: breed, dtype: int64 Name: breed, dtype: int64 DATA MANIPULATION WITH PANDAS

  17. Proportions unique_dogs["breed"].value_counts(normalize=True) Labrador 0.250 Chow Chow 0.250 Schnauzer 0.125 St. Bernard 0.125 Poodle 0.125 Chihuahua 0.125 Name: breed, dtype: float64 DATA MANIPULATION WITH PANDAS

  18. Let's practice! DATA MAN IP ULATION W ITH PAN DAS

  19. Grouped summary statistics DATA MAN IP ULATION W ITH PAN DAS Maggie Matsui Content Developer at DataCamp

  20. Summaries by group dogs[dogs["color"] == "Black"]["weight_kg"].mean() dogs[dogs["color"] == "Brown"]["weight_kg"].mean() dogs[dogs["color"] == "White"]["weight_kg"].mean() dogs[dogs["color"] == "Gray"]["weight_kg"].mean() dogs[dogs["color"] == "Tan"]["weight_kg"].mean() 26.0 24.0 74.0 17.0 2.0 DATA MANIPULATION WITH PANDAS

  21. Grouped summaries dogs.groupby("color")["weight_kg"].mean() color Black 26.5 Brown 24.0 Gray 17.0 Tan 2.0 White 74.0 Name: weight_kg, dtype: float64 DATA MANIPULATION WITH PANDAS

  22. Multiple grouped summaries dogs.groupby("color")["weight_kg"].agg([min, max, sum]) min max sum color Black 24 29 53 Brown 24 24 48 Gray 17 17 17 Tan 2 2 2 White 74 74 74 DATA MANIPULATION WITH PANDAS

  23. Grouping by multiple variables dogs.groupby(["color", "breed"])["weight_kg"].mean() color breed Black Chow Chow 25 Labrador 29 Poodle 24 Brown Chow Chow 24 Labrador 24 Gray Schnauzer 17 Tan Chihuahua 2 White St. Bernard 74 Name: weight_kg, dtype: int64 DATA MANIPULATION WITH PANDAS

  24. Many groups, many summaries dogs.groupby(["color", "breed"])[["weight_kg", "height_cm"]].mean() weight_kg height_cm color breed Black Labrador 29 59 Poodle 24 43 Brown Chow Chow 24 46 Labrador 24 56 Gray Schnauzer 17 49 Tan Chihuahua 2 18 White St. Bernard 74 77 DATA MANIPULATION WITH PANDAS

  25. Let's practice! DATA MAN IP ULATION W ITH PAN DAS

  26. Pivot tables DATA MAN IP ULATION W ITH PAN DAS Maggie Matsui Content Developer at DataCamp

  27. Group by to pivot table dogs.groupby("color")["weight_kg"].mean() dogs.pivot_table(values="weight_kg", index="color") color Black 26 weight_kg Brown 24 color Gray 17 Black 26.5 Tan 2 Brown 24.0 White 74 Gray 17.0 Name: weight_kg, dtype: int64 Tan 2.0 White 74.0 DATA MANIPULATION WITH PANDAS

  28. Different statistics import numpy as np dogs.pivot_table(values="weight_kg", index="color", aggfunc=np.median) weight_kg color Black 26.5 Brown 24.0 Gray 17.0 Tan 2.0 White 74.0 DATA MANIPULATION WITH PANDAS

  29. Multiple statistics dogs.pivot_table(values="weight_kg", index="color", aggfunc=[np.mean, np.median]) mean median weight_kg weight_kg color Black 26.5 26.5 Brown 24.0 24.0 Gray 17.0 17.0 Tan 2.0 2.0 White 74.0 74.0 DATA MANIPULATION WITH PANDAS

  30. Pivot on two variables dogs.groupby(["color", "breed"])["weight_kg"].mean() dogs.pivot_table(values="weight_kg", index="color", columns="breed") breed Chihuahua Chow Chow Labrador Poodle Schnauzer St. Bernard color Black NaN NaN 29.0 24.0 NaN NaN Brown NaN 24.0 24.0 NaN NaN NaN Gray NaN NaN NaN NaN 17.0 NaN Tan 2.0 NaN NaN NaN NaN NaN White NaN NaN NaN NaN NaN 74.0 DATA MANIPULATION WITH PANDAS

  31. Filling missing values in pivot tables dogs.pivot_table(values="weight_kg", index="color", columns="breed", fill_value=0) breed Chihuahua Chow Chow Labrador Poodle Schnauzer St. Bernard color Black 0 0 29 24 0 0 Brown 0 24 24 0 0 0 Gray 0 0 0 0 17 0 Tan 2 0 0 0 0 0 White 0 0 0 0 0 74 DATA MANIPULATION WITH PANDAS

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Probability & Statistics: Intro, summary statistics, probability 2 - Efron & Tibshirani,

Probability & Statistics: Intro, summary statistics, probability 2 - Efron & Tibshirani,

1 Mathematical Tools for Neural and Cognitive Science Fall semester, 2018 Probability & Statistics: Intro, summary statistics, probability 2 - Efron & Tibshirani, Introduction to the Bootstrap , 1998 3 Some history 1600s:

536 views • 19 slides
Next Generation Vital Statistics Summary Report Letter to the Secretary Next Steps

Next Generation Vital Statistics Summary Report Letter to the Secretary Next Steps

Next Generation Vital Statistics Summary Report Letter to the Secretary Next Steps Hearing Summary (Table of Contents) Executive Summary I. Introduction and Overview of the Hearing II. The Vital Records and Statistics System

283 views • 10 slides
The Statistics of Summary-Data MR Qingyuan Zhao Department of Statistics, Wharton School,

The Statistics of Summary-Data MR Qingyuan Zhao Department of Statistics, Wharton School,

The Statistics of Summary-Data MR Qingyuan Zhao Department of Statistics, Wharton School, University of Pennsylvania ( From August 1st : Statistical Laboratory, University of Cambridge) July 17, 2019 @ MRC-IEU Mendelian randomization conference,

247 views • 22 slides
Frequency Distribution and Summary Statistics Dongmei Li Department of Public Health Sciences

Frequency Distribution and Summary Statistics Dongmei Li Department of Public Health Sciences

Frequency Distribution and Summary Statistics Dongmei Li Department of Public Health Sciences Office of Public Health Studies University of Hawaii at M noa Outline 1. Stemplot 2. Frequency table 3. Summary statistics 2 1. Stem-and-leaf

924 views • 61 slides
1 Practical Information 2 Introduction to Statistics Per Bruun Brockhoff 3 Descriptive Statistics:

1 Practical Information 2 Introduction to Statistics Per Bruun Brockhoff 3 Descriptive Statistics:

Agenda Course 02402 Introduction to Statistics Lecture 1: Introduction to Statistics 1 Practical Information 2 Introduction to Statistics Per Bruun Brockhoff 3 Descriptive Statistics: Summary Statistics DTU Informatics Building 305 - room 110

194 views • 5 slides
Introduction to Data Science: x (1) x 1 x 2 x ( n ) x i n 1 1 Size: size

Introduction to Data Science: x (1) x 1 x 2 x ( n ) x i n 1 1 Size: size

Exploratory Data Analysis Exploratory Data Analysis: Summary Statistics Exploratory Data Analysis: Summary Statistics EDA with the grammar of graphics Exploratory Data Analysis: Summary Statistics Exploratory Data Analysis: Summary Statistics

1.27k views • 90 slides
Practical methods for handling missing summary statistics in meta-analysis of continuous outcomes

Practical methods for handling missing summary statistics in meta-analysis of continuous outcomes

www.ed.ac.uk/usher @EdinUniUsher Practical methods for handling missing summary statistics in meta-analysis of continuous outcomes Cochrane Webinar, 5 February 2019 Professor Christopher Weir Personal Chair in Medical Statistics & Clinical

682 views • 29 slides
How to choose summary statistics for model selection and model checking. Sarah Filippi Imperial

How to choose summary statistics for model selection and model checking. Sarah Filippi Imperial

How to choose summary statistics for model selection and model checking. Sarah Filippi Imperial College London Theoretical Systems Biology Group 13/02/2012 Choice of summary statistics Sarah Filippi 1 of 22 Model selection vs model checking

631 views • 33 slides
Math 140 The values of a summary statistic (e.g. the Introductory Statistics average age of the

Math 140 The values of a summary statistic (e.g. the Introductory Statistics average age of the

Visualizing Distributions Recall the definition: Math 140 The values of a summary statistic (e.g. the Introductory Statistics average age of the laid-off workers) and how often they occur. Four of the most common basic shapes :

317 views • 21 slides
2.2: Numerical summary Measures of location. Measures of spread. Measures of form.

2.2: Numerical summary Measures of location. Measures of spread. Measures of form.

Applied Statistics 2.2: Numerical summary Measures of location. Measures of spread. Measures of form. Recommended reading: Chapters 3 to 7 of Portilla (2004) Applied Statistics DESCRIPTIVE STATISTICS Why are they useful?

538 views • 34 slides
Review: Types of Summary Statistics Were often interested in describing the following

Review: Types of Summary Statistics Were often interested in describing the following

Review: Types of Summary Statistics Were often interested in describing the following characteristics of the distribution of a data series: Central tendency - where is the middle of the distribution? Dispersion - how spread out is the

448 views • 28 slides
Elements of Statistics for Economists Session 3 Presentation of Data: Numerical Summary

Elements of Statistics for Economists Session 3 Presentation of Data: Numerical Summary

ECON 214 Elements of Statistics for Economists Session 3 Presentation of Data: Numerical Summary Measures Part 2 Lecturer: Dr. Bernardin Senadza , Dept. of Economics Contact Information: bsenadza@ug.edu.gh College of Education School of

661 views • 38 slides
Descriptive and Summary Statistics BIO5312 FALL2017 STEPHANIE J. SPIELMAN, PHD Logistics All

Descriptive and Summary Statistics BIO5312 FALL2017 STEPHANIE J. SPIELMAN, PHD Logistics All

Descriptive and Summary Statistics BIO5312 FALL2017 STEPHANIE J. SPIELMAN, PHD Logistics All course materials will be hosted here: http://sjspielman.org/bio5312_fall2017 Submit assignments via Canvas: https://templeu.instructure.com Please

823 views • 36 slides
Distributed Summary Statistics with Bro Vlad Grigorescu 1 > whoami Member of the Bro

Distributed Summary Statistics with Bro Vlad Grigorescu 1 > whoami Member of the Bro

Distributed Summary Statistics with Bro Vlad Grigorescu 1 > whoami Member of the Bro development team Senior Developer at Broala LLC Senior Information Security Engineer at Carnegie Mellon University

305 views • 26 slides
Choosing the Summary Statistics and the Acceptance Rate in Approximate Bayesian Computation (ABC)

Choosing the Summary Statistics and the Acceptance Rate in Approximate Bayesian Computation (ABC)

Introduction Standard algorithms Potential pitfalls with ABC Local Bayesian linear regression Conclusion Choosing the Summary Statistics and the Acceptance Rate in Approximate Bayesian Computation (ABC) Michael G.B. Blum Laboratoire

241 views • 22 slides
A Linked Data Representation for Summary Statistics and Grouping Criteria RPI IDEA/Tetherless

A Linked Data Representation for Summary Statistics and Grouping Criteria RPI IDEA/Tetherless

A Linked Data Representation for Summary Statistics and Grouping Criteria RPI IDEA/Tetherless World Constellation James P. McCusker, Michel Dumontier, Shruthi Chari, Joanne S. Luciano, and Deborah L. McGuinness Class: G(case:TCGA-BRCA)

292 views • 17 slides
The Chow form of a reciprocal linear space Cynthia Vinzant North Carolina State University joint

The Chow form of a reciprocal linear space Cynthia Vinzant North Carolina State University joint

The Chow form of a reciprocal linear space Cynthia Vinzant North Carolina State University joint work with Mario Kummer, Universit at Konstanz Cynthia Vinzant The Chow form of a reciprocal linear space Hyperbolicity and determinantal

446 views • 32 slides
Logistic Regression Matt Gormley Lecture 9 Feb. 13, 2019 1 Q&A Q: In recitation, we only

Logistic Regression Matt Gormley Lecture 9 Feb. 13, 2019 1 Q&A Q: In recitation, we only

10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Logistic Regression Matt Gormley Lecture 9 Feb. 13, 2019 1 Q&A Q: In recitation, we only covered the Perceptron

559 views • 40 slides
Is the best model good enough? Assessing the absolute fit of phylogenetic models via posterior

Is the best model good enough? Assessing the absolute fit of phylogenetic models via posterior

Is the best model good enough? Assessing the absolute fit of phylogenetic models via posterior predictive sampling Gerhard Jger Tbingen University Workshop Computational and phylogenetic historical linguistics ICHL24, Canberra, July 4, 2019

534 views • 41 slides
Nutritional Information Calories Saturated Trans Fat Cholestero Carbohydrate Dietary Vitamin

Nutritional Information Calories Saturated Trans Fat Cholestero Carbohydrate Dietary Vitamin

Nutritional Information Calories Saturated Trans Fat Cholestero Carbohydrate Dietary Vitamin Recipe Name Calories Total Fat Sodium Sugars Protein Vitamin C Calcium Iron from Fat Fat l s Fiber A Breakfast Burrito - Bacon 1170

225 views • 3 slides
Stanley @ 70 June 23, 2014 4 n Theorem 1 . There are binary sequences of length 4 n + 1 2

Stanley @ 70 June 23, 2014 4 n Theorem 1 . There are binary sequences of length 4 n + 1 2

Timothy Y. Chow Center for Communications Research Stanley @ 70 June 23, 2014 4 n Theorem 1 . There are binary sequences of length 4 n + 1 2 n with the property that for all j , the j th occurrence of 10 appears in positions 4 j + 1 and

310 views • 8 slides
Tropical cycles and Chow polytopes Alex Fink Department of Mathematics University of California,

Tropical cycles and Chow polytopes Alex Fink Department of Mathematics University of California,

Tropical cycles and Chow polytopes Alex Fink Department of Mathematics University of California, Berkeley Tropical Geometry in Combinatorics and Algebra MSRI October 16, 2009 Alex Fink (UC Berkeley) Tropical cycles and Chow polytopes MSRI,

1.01k views • 49 slides
Machine Learning 10-701 Tom M. Mitchell Machine Learning Department Carnegie Mellon University

Machine Learning 10-701 Tom M. Mitchell Machine Learning Department Carnegie Mellon University

Machine Learning 10-701 Tom M. Mitchell Machine Learning Department Carnegie Mellon University February 22, 2011 Today: Readings: Recommended: Clustering Jordan Graphical Models Mixture model clustering Muphy Intro

655 views • 10 slides
Geometric control and applications Ludovic Rifford Universit e Nice Sophia Antipolis Fields

Geometric control and applications Ludovic Rifford Universit e Nice Sophia Antipolis Fields

Geometric control and applications Ludovic Rifford Universit e Nice Sophia Antipolis Fields Institute - November 6-7, 2014 Ludovic Rifford Fields Institute minischool Outline Lecture 1: A controllability result: The Chow-Rashevsky Theorem

825 views • 71 slides

More recommend