mean median mode imputations
play

Mean, median & mode imputations DEALIN G W ITH MIS S IN G DATA - PowerPoint PPT Presentation

Nov 13, 2023 •406 likes •821 views

Mean, median & mode imputations DEALIN G W ITH MIS S IN G DATA IN P YTH ON Suraj Donthi Deep Learning & Computer Vision Consultant Basic imputation techniques constant (e.g. 0) mean median mode or most frequent DEALING WITH

  1. Mean, median & mode imputations DEALIN G W ITH MIS S IN G DATA IN P YTH ON Suraj Donthi Deep Learning & Computer Vision Consultant

  2. Basic imputation techniques constant (e.g. 0) mean median mode or most frequent DEALING WITH MISSING DATA IN PYTHON

  3. Mean Imputation from sklearn.impute import SimpleImputer diabetes_mean = diabetes.copy(deep=True) mean_imputer = SimpleImputer(strategy='mean') DEALING WITH MISSING DATA IN PYTHON

  4. Mean Imputation from sklearn.impute import SimpleImputer diabetes_mean = diabetes.copy(deep=True) mean_imputer = SimpleImputer(strategy='mean') diabetes_mean.iloc[:, :] = mean_imputer.fit_transform(diabetes_mean) DEALING WITH MISSING DATA IN PYTHON

  5. Median imputation diabetes_median = diabetes.copy(deep=True) median_imputer = SimpleImputer(strategy='median') diabetes_median.iloc[:, :] = median_imputer.fit_transform(diabetes_median) DEALING WITH MISSING DATA IN PYTHON

  6. Mode imputation diabetes_mode = diabetes.copy(deep=True) mode_imputer = SimpleImputer(strategy='most_frequent') diabetes_mode.iloc[:, :] = mode_imputer.fit_transform(diabetes_mode) DEALING WITH MISSING DATA IN PYTHON

  7. Imputing a constant diabetes_constant = diabetes.copy(deep=True) constant_imputer = SimpleImputer(strategy='constant', fill_value=0)) diabetes_constant.iloc[:, :] = constant_imputer.fit_transform(diabetes_constant) DEALING WITH MISSING DATA IN PYTHON

  8. Scatterplot of imputation nullity = diabetes['Serum_Insulin'].isnull()+diabetes['Glucose'].isnull() diabetes_mean.plot(x='Serum_Insulin', y='Glucose', kind='scatter', alpha=0.5, c=nullity, cmap='rainbow', title='Mean Imputation') DEALING WITH MISSING DATA IN PYTHON

  9. Visualizing imputations fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(10, 10)) nullity = diabetes['Serum_Insulin'].isnull()+diabetes['Glucose'].isnull() imputations = {'Mean Imputation': diabetes_mean, 'Median Imputation': diabetes_median, 'Most Frequent Imputation': diabetes_mode, 'Constant Imputation': diabetes_constant} for ax, df_key in zip(axes.flatten(), imputations): imputations[df_key].plot(x='Serum_Insulin', y='Glucose', kind='scatter', alpha=0.5, c=nullity, cmap='rainbow', ax=ax, colorbar=False, title=df_key) DEALING WITH MISSING DATA IN PYTHON

  10. DEALING WITH MISSING DATA IN PYTHON

  11. Summary You learned to Impute with statistical parameters like mean, median and mode Graphically compare the imputations Analyze the imputations DEALING WITH MISSING DATA IN PYTHON

  12. Let's practice! DEALIN G W ITH MIS S IN G DATA IN P YTH ON

  13. Imputing time-series data DEALIN G W ITH MIS S IN G DATA IN P YTH ON Suraj Donthi Deep Learning & Computer Vision Consultant

  14. Airquality Dataset import pandas as pd airquality = pd.read_csv('air-quality.csv', parse_dates='Date', index_col='Date') airquality.head() Ozone Solar Wind Temp Date 1976-05-01 41.0 190.0 7.4 67 1976-05-02 36.0 118.0 8.0 72 1976-05-03 12.0 149.0 12.6 74 1976-05-04 18.0 313.0 11.5 62 1976-05-05 NaN NaN 14.3 56 DEALING WITH MISSING DATA IN PYTHON

  15. Airquality Dataset airquality.isnull().sum() airquality.isnull.mean() * 100 Ozone 37 Ozone 24.183007 Solar 7 Solar 4.575163 Wind 0 Wind 0.000000 Temp 0 Temp 0.000000 dtype: int64 dtype: float64 DEALING WITH MISSING DATA IN PYTHON

  16. The .�llna() method The attribute method in .fillna() can be set to 'ffill' or 'pad' 'bfill' or 'backwardfill' DEALING WITH MISSING DATA IN PYTHON

  17. F�ll method Replace NaN s with last observed value pad is the same as 'ffill' airquality.fillna(method='ffill', inplace=True) DEALING WITH MISSING DATA IN PYTHON

  18. airquality.fillna(method='ffill', inplace=True) airquality['Ozone'][30:40] airquality['Ozone'][30:40] Date Ozone Date Ozone 1976-05-31 37.0 1976-05-31 37.0 1976-06-01 NaN 1976-06-01 37.0 1976-06-02 NaN 1976-06-02 37.0 1976-06-03 NaN 1976-06-03 37.0 1976-06-04 NaN 1976-06-04 37.0 1976-06-05 NaN 1976-06-05 37.0 1976-06-06 NaN 1976-06-06 37.0 1976-06-07 29.0 1976-06-07 29.0 1976-06-08 NaN 1976-06-08 29.0 1976-06-09 71.0 1976-06-09 71.0 DEALING WITH MISSING DATA IN PYTHON

  19. B�ll method Replace NaN s with next observed value backfill is the same as 'bfill' df.fillna(method='bfill', inplace=True) DEALING WITH MISSING DATA IN PYTHON

  20. airquality.fillna(method='bfill', inplace=True) airquality['Ozone'][30:40] airquality['Ozone'][30:40] Date Ozone Date Ozone 1976-05-31 37.0 1976-05-31 37.0 1976-06-01 NaN 1976-06-01 29.0 1976-06-02 NaN 1976-06-02 29.0 1976-06-03 NaN 1976-06-03 29.0 1976-06-04 NaN 1976-06-04 29.0 1976-06-05 NaN 1976-06-05 29.0 1976-06-06 NaN 1976-06-06 29.0 1976-06-07 29.0 1976-06-07 29.0 1976-06-08 NaN 1976-06-08 71.0 1976-06-09 71.0 1976-06-09 71.0 DEALING WITH MISSING DATA IN PYTHON

  21. The .interpolate() method The .interpolate() method extends the sequence of values to the missing values The attribute method in .interpolate() can be set to 'linear' 'quadratic' 'nearest' DEALING WITH MISSING DATA IN PYTHON

  22. Linear interpolation Impute linearly or with equidistant values df.interpolate(method='linear', inplace=True) DEALING WITH MISSING DATA IN PYTHON

  23. airquality.interpolate( method='linear', inplace=True) airquality['Ozone'][30:40] airquality['Ozone'][30:40] Date Ozone Date Ozone 1976-05-31 37.0 1976-05-31 37.0 1976-06-01 NaN 1976-06-01 35.9 1976-06-02 NaN 1976-06-02 34.7 1976-06-03 NaN 1976-06-03 33.6 1976-06-04 NaN 1976-06-04 32.4 1976-06-05 NaN 1976-06-05 31.3 1976-06-06 NaN 1976-06-06 30.1 1976-06-07 29.0 1976-06-07 29.0 1976-06-08 NaN 1976-06-08 50.0 1976-06-09 71.0 1976-06-09 71.0 DEALING WITH MISSING DATA IN PYTHON

  24. Quadratic interpolation Impute the values quadratically df.interpolate(method='quadratic', inplace=True) DEALING WITH MISSING DATA IN PYTHON

  25. airquality.interpolate( method='quadratic', inplace=True) airquality['Ozone'][30:39] airquality['Ozone'][30:39] Ozone Ozone Date Date 1976-05-31 37.0 1976-05-31 37.0 1976-06-01 NaN 1976-06-01 -38.4 1976-06-02 NaN 1976-06-02 -79.4 1976-06-03 NaN 1976-06-03 -85.9 1976-06-04 NaN 1976-06-04 -62.4 1976-06-05 NaN 1976-06-06 -2.8 1976-06-06 NaN 1976-06-07 29.0 1976-06-07 29.0 1976-06-08 62.2 1976-06-08 NaN DEALING WITH MISSING DATA IN PYTHON

  26. Nearest value imputation Impute with the nearest observable value df.interpolate(method='nearest', inplace=True) DEALING WITH MISSING DATA IN PYTHON

  27. airquality.interpolate( method='nearest', inplace=True) airquality['Ozone'][30:39] airquality['Ozone'][30:39] Date Ozone Date Ozone 1976-05-31 37.0 1976-05-31 37.0 1976-06-01 NaN 1976-06-01 37.0 1976-06-02 NaN 1976-06-02 37.0 1976-06-03 NaN 1976-06-03 37.0 1976-06-04 NaN 1976-06-04 29.0 1976-06-05 NaN 1976-06-05 29.0 1976-06-06 NaN 1976-06-06 29.0 1976-06-07 29.0 1976-06-07 29.0 1976-06-08 NaN 1976-06-08 29.0 DEALING WITH MISSING DATA IN PYTHON

  28. Let's practice! DEALIN G W ITH MIS S IN G DATA IN P YTH ON

  29. Visualizing time- series imputations DEALIN G W ITH MIS S IN G DATA IN P YTH ON Suraj Donthi Deep Learning & Computer Vision Learning

  30. Air quality time-series plot airquality['Ozone'].plot(title='Ozone', marker='o', figsize=(30, 5)) DEALING WITH MISSING DATA IN PYTHON

  31. F�ll Imputation ffill_imp['Ozone'].plot(color='red', marker='o', linestyle='dotted', figsize=(30, 5)) airquality['Ozone'].plot(title='Ozone', marker='o') DEALING WITH MISSING DATA IN PYTHON

  32. B�ll Imputation bfill_imp['Ozone'].plot(color='red', marker='o', linestyle='dotted', figsize=(30, 5)) airquality['Ozone'].plot(title='Ozone', marker='o') DEALING WITH MISSING DATA IN PYTHON

  33. Linear Interpolation linear_interp['Ozone'].plot(color='red', marker='o', linestyle='dotted', figsize=(30, 5) airquality['Ozone'].plot(title='Ozone', marker='o') DEALING WITH MISSING DATA IN PYTHON

  34. Quadratic Interpolation quadratic_interp['Ozone'].plot(color='red', marker='o', linestyle='dotted', figsize=(30, 5)) airquality['Ozone'].plot(title='Ozone', marker='o') DEALING WITH MISSING DATA IN PYTHON

  35. Nearest Interpolation nearest_interp['Ozone'].plot(color='red', marker='o', linestyle='dotted', figsize=(30, 5 airquality['Ozone'].plot(title='Ozone', marker='o') DEALING WITH MISSING DATA IN PYTHON

  36. A comparison of the interpolations # Create subplots fig, axes = plt.subplots(3, 1, figsize=(30, 20)) # Create interpolations dictionary interpolations = {'Linear Interpolation': linear_interp, 'Quadratic Interpolation': quadratic_interp, 'Nearest Interpolation': nearest_interp} # Visualize each interpolation for ax, df_key in zip(axes, interpolations): interpolations[df_key].Ozone.plot(color='red', marker='o', linestyle='dotted', ax=ax) airquality.Ozone.plot(title=df_key + ' - Ozone', marker='o', ax=ax) DEALING WITH MISSING DATA IN PYTHON

  37. A comparison of the interpolations DEALING WITH MISSING DATA IN PYTHON

  38. A comparison of imputation techniques DEALING WITH MISSING DATA IN PYTHON

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

the nerves sensory radial median ulnar median median sensory median median ulnar radial

the nerves sensory radial median ulnar median median sensory median median ulnar radial

the nerves sensory radial median ulnar median median sensory median median ulnar radial median ulnar http://www.eatonhand.com/jpg/1494401.jpg Image Credit: Donnie Ray Jones on Flickr | CC License |

618 views • 17 slides
Sensitivity of the population size estimates for different imputations of a covariate B.F.M.

Sensitivity of the population size estimates for different imputations of a covariate B.F.M.

C APTURE -R ECAPTURE U SUAL R ESIDENTS S CENARIO S E XPECTATIONS R ESULTS Sensitivity of the population size estimates for different imputations of a covariate B.F.M. Bakker 2 P.G.M. van der Heijden 3 S.C. Gerritse Utrecht University 2

557 views • 22 slides
Numerical summary of data Measures of location: mode , median, mean, Measures of spread:

Numerical summary of data Measures of location: mode , median, mean, Measures of spread:

Introduction to Statistics Numerical summary of data Measures of location: mode , median, mean, Measures of spread: range, interquartile range, standard deviation , Measures of form: skewness, kurtosis , Introduction to

614 views • 30 slides
Using the Superpopulation Model for Imputations and Variance Computation in Survey Sampling Petr

Using the Superpopulation Model for Imputations and Variance Computation in Survey Sampling Petr

Using the Superpopulation Model for Imputations and Variance Computation in Survey Sampling Petr Novk, Vclav Kosina Czech Statistical Office Petr Novk, Vclav Kosina Using the Superpopulation Model for Imputations and Variance

355 views • 19 slides
Assumptions and normal distributions EX P ERIMEN TAL DES IGN IN P YTH ON Luke Hayden

Assumptions and normal distributions EX P ERIMEN TAL DES IGN IN P YTH ON Luke Hayden

Assumptions and normal distributions EX P ERIMEN TAL DES IGN IN P YTH ON Luke Hayden Instructor Summary stats Mean Sum divided by count Median Half of values fall above and below the median Mode Value that occurs most often Standard

745 views • 41 slides
Linear-time Median Def: Median of elements A=a 1 , a 2 , , a n is the (n/2)-th smallest element

Linear-time Median Def: Median of elements A=a 1 , a 2 , , a n is the (n/2)-th smallest element

Linear-time Median Def: Median of elements A=a 1 , a 2 , , a n is the (n/2)-th smallest element in A. How to find median? sort the elements, output the elem. at (n/2)-th position - running time ? Linear-time Median Def: Median of

229 views • 11 slides
African American Strategy Equitable Access to Homeownership Presentation April 16, 2018

African American Strategy Equitable Access to Homeownership Presentation April 16, 2018

CON FI DEN T I AL African American Strategy Equitable Access to Homeownership Presentation April 16, 2018 Individual Demographics Age Income (1000s) FICO Total DTI All Debt Student Debt ($) Median Median Median Median Median Median

299 views • 13 slides
Median Finding Test Cases What's Next 1. Median finding, part 2 2. Why we write test cases 3.

Median Finding Test Cases What's Next 1. Median finding, part 2 2. Why we write test cases 3.

Median Finding Test Cases What's Next 1. Median finding, part 2 2. Why we write test cases 3. What's next? A problem Find the (upper) median of a list of n items. Upper median means "if the list has an even number of items, pick

1.38k views • 52 slides
Median filter Non-linear filtering example Replace each pixel by the median over N pixels (5

Median filter Non-linear filtering example Replace each pixel by the median over N pixels (5

Today Median filter Non-linear filtering example Replace each pixel by the median over N pixels (5 pixels, for these examples). Generalizes to rank order filters. Spike In: Out: noise is removed 5-pixel neighborhood Monotonic

432 views • 16 slides
Events MedIAN Jobs Contact The Network Background About MedIAN UKs national Medical

Events MedIAN Jobs Contact The Network Background About MedIAN UKs national Medical

About Events MedIAN Jobs Contact The Network Background About MedIAN UKs national Medical Image Analysis community A network of academics , clinicians , and industry Steering New clinical research and tech collaborations

728 views • 15 slides
Business Statistics CONTENTS Hypotheses on the median The sign test The Wilcoxon signed ranks

Business Statistics CONTENTS Hypotheses on the median The sign test The Wilcoxon signed ranks

MEDIAN: NON-PARAMETRIC TESTS Business Statistics CONTENTS Hypotheses on the median The sign test The Wilcoxon signed ranks test Old exam question Further study HYPOTHESES ON THE MEDIAN The median is a central value that may be more

327 views • 19 slides
Today Non-linear filtering example Median filter Replace each pixel by the median over N pixels

Today Non-linear filtering example Median filter Replace each pixel by the median over N pixels

Today Non-linear filtering example Median filter Replace each pixel by the median over N pixels (5 pixels, for these examples). Generalizes to rank order filters. Spike In: Out: noise is removed 5-pixel neighborhood Monotonic

1.25k views • 91 slides
City of Atascadero Size: 26 sq miles Incorporated: 1979 Population: 30,305 o Median

City of Atascadero Size: 26 sq miles Incorporated: 1979 Population: 30,305 o Median

City of Atascadero Size: 26 sq miles Incorporated: 1979 Population: 30,305 o Median Age 38.2 yrs (US Median 37.8 yrs ) o Avg. Family Income $104,744 (County $104,489) o Total # of Households 11,431 Median Home Price: $480,000 City

567 views • 14 slides
W4231: Analysis of Algorithms Definition of median 9/14/1999 Let A = a 1 a n be a

W4231: Analysis of Algorithms Definition of median 9/14/1999 Let A = a 1 a n be a

W4231: Analysis of Algorithms Definition of median 9/14/1999 Let A = a 1 a n be a sequence of integers. The median of A is a value v such that Median Selection |{ i : a i < v }| n/ 2 and |{ i : a i > v }| n/ 2 That is,

218 views • 4 slides
Harbor Bay Median Sheet Mulch project HARBOR BAY MEDIAN Located in the City of Alameda 2

Harbor Bay Median Sheet Mulch project HARBOR BAY MEDIAN Located in the City of Alameda 2

Short case study of Harbor Bay Median Sheet Mulch project HARBOR BAY MEDIAN Located in the City of Alameda 2 miles of medians along a busy road Project located in environmentally sensitive area less than 100 yards from shoreline

392 views • 11 slides
Median Filter (5) sparsely populated 5x5 median filters effect on white noise; std of output

Median Filter (5) sparsely populated 5x5 median filters effect on white noise; std of output

Castleman, 1996 Median Filter (5) sparsely populated 5x5 median filters effect on white noise; std of output image vs. # pixels participating 41 IIP Sharpening (1) Used to: highlight fine detail Enhance detail that has been

305 views • 8 slides
On Medians of (Randomized) Pairwise Means Pierre Laforgue 1 , Stephan Cl on 1 , Patrice Bertail

On Medians of (Randomized) Pairwise Means Pierre Laforgue 1 , Stephan Cl on 1 , Patrice Bertail

On Medians of (Randomized) Pairwise Means Pierre Laforgue 1 , Stephan Cl on 1 , Patrice Bertail 2 emenc 1 LTCI, T el ecom Paris, Institut Polytechnique de Paris, France 2 ModalX, UPL, Universit e Paris-Nanterre, France 1 The

144 views • 11 slides
Looking For Truth Or At Least Data Elizabeth D. Zwicky zwicky@otoh.org LISA 2009 Important

Looking For Truth Or At Least Data Elizabeth D. Zwicky zwicky@otoh.org LISA 2009 Important

Looking For Truth Or At Least Data Elizabeth D. Zwicky zwicky@otoh.org LISA 2009 Important Disclaimers All the numbers in this presentation are made up. The stories are true. I am not a statistician. Im done with the funky

1.02k views • 74 slides
CSE101: Algorithm Design and Analysis Russell Impagliazzo Sanjoy Dasgupta Ragesh Jaiswal

CSE101: Algorithm Design and Analysis Russell Impagliazzo Sanjoy Dasgupta Ragesh Jaiswal

CSE101: Algorithm Design and Analysis Russell Impagliazzo Sanjoy Dasgupta Ragesh Jaiswal (Thanks for slides: Miles Jones) Week-06 Lecture 23: Divide and Conquer (Sorting and Selection) Divide and Conquer sort Starting with a list of

727 views • 43 slides
Statistical Data Analysis DS GA 1002 Statistical and Mathematical Models

Statistical Data Analysis DS GA 1002 Statistical and Mathematical Models

Statistical Data Analysis DS GA 1002 Statistical and Mathematical Models http://www.cims.nyu.edu/~cfgranda/pages/DSGA1002_fall16 Carlos Fernandez-Granda Descriptive statistics Statistical estimation Histogram Technique to visualize

1.16k views • 99 slides
Order Statistics Carola Wenk Slides courtesy of Charles Leiserson with additions by Carola Wenk

Order Statistics Carola Wenk Slides courtesy of Charles Leiserson with additions by Carola Wenk

CMPS 6610/4610 Fall 2016 Order Statistics Carola Wenk Slides courtesy of Charles Leiserson with additions by Carola Wenk CMPS 6610/4610 Algorithms 1 Order statistics Select the i th smallest of n elements (the element with rank i ). i

665 views • 23 slides
CS4102 Algorithms Fall 2018 Warm up Compare + with + () When

CS4102 Algorithms Fall 2018 Warm up Compare + with + () When

CS4102 Algorithms Fall 2018 Warm up Compare + with + () When = () When = () 1 = O() () () () () + +

738 views • 44 slides
Fast and efficient estimation of the geometric median in high dimension : a stochastic gradient

Fast and efficient estimation of the geometric median in high dimension : a stochastic gradient

Fast and efficient estimation of the geometric median in high dimension : a stochastic gradient approach Herv e Cardot Institut de Math ematiques de Bourgogne, Universit e de Bourgogne with Peggy C enac (Univ. Bourgogne) and Mohamed

571 views • 17 slides
Saving Behavior Across the Wealth Distribution: The Importance of Capital Gains Andreas Fagereng

Saving Behavior Across the Wealth Distribution: The Importance of Capital Gains Andreas Fagereng

Saving Behavior Across the Wealth Distribution: The Importance of Capital Gains Andreas Fagereng Martin Holm Benjamin Moll Gisle Natvik SF Fed, 31 October 2019 Question Do wealthier households save larger share of income than poorer ones?

650 views • 54 slides

More recommend