data preparation
play

Data Preparation MACH IN E LEARN IN G W ITH P YS PARK Andrew - PowerPoint PPT Presentation

Aug 15, 2023 •226 likes •688 views

Data Preparation MACH IN E LEARN IN G W ITH P YS PARK Andrew Collier Data Scientist, Exegetic Analytics Do you need all of those columns? +-----+-------+-------+------+----+----+------+------+----+-----------+ |maker| model| origin| type|

  1. Data Preparation MACH IN E LEARN IN G W ITH P YS PARK Andrew Collier Data Scientist, Exegetic Analytics

  2. Do you need all of those columns? +-----+-------+-------+------+----+----+------+------+----+-----------+ |maker| model| origin| type| cyl|size|weight|length| rpm|consumption| +-----+-------+-------+------+----+----+------+------+----+-----------+ |Mazda| RX-7|non-USA|Sporty|null| 1.3| 2895| 169.0|6500| 9.41| | Geo| Metro|non-USA| Small| 3| 1.0| 1695| 151.0|5700| 4.7| | Ford|Festiva| USA| Small| 4| 1.3| 1845| 141.0|5000| 7.13| +-----+-------+-------+------+----+----+------+------+----+-----------+ Remove the maker and model �elds. MACHINE LEARNING WITH PYSPARK

  3. Dropping columns # Either drop the columns you don't want... cars = cars.drop('maker', 'model') # ... or select the columns you want to retain. cars = cars.select('origin', 'type', 'cyl', 'size', 'weight', 'length', 'rpm', 'consumption') +-------+------+----+----+------+------+----+-----------+ | origin| type| cyl|size|weight|length| rpm|consumption| +-------+------+----+----+------+------+----+-----------+ |non-USA|Sporty|null| 1.3| 2895| 169.0|6500| 9.41| |non-USA| Small| 3| 1.0| 1695| 151.0|5700| 4.7| | USA| Small| 4| 1.3| 1845| 141.0|5000| 7.13| +-------+------+----+----+------+------+----+-----------+ MACHINE LEARNING WITH PYSPARK

  4. Filtering out missing data # How many missing values? cars.filter('cyl IS NULL').count() 1 Drop records with missing values in the cylinders column. cars = cars.filter('cyl IS NOT NULL') Drop records with missing values in any column. cars = cars.dropna() MACHINE LEARNING WITH PYSPARK

  5. Mutating columns from pyspark.sql.functions import round # Create a new 'mass' column cars = cars.withColumn('mass', round(cars.weight / 2.205, 0)) # Convert length to metres cars = cars.withColumn('length', round(cars.length * 0.0254, 3)) +-------+-----+---+----+------+------+----+-----------+-----+ | origin| type|cyl|size|weight|length| rpm|consumption| mass| +-------+-----+---+----+------+------+----+-----------+-----+ |non-USA|Small| 3| 1.0| 1695| 3.835|5700| 4.7|769.0| | USA|Small| 4| 1.3| 1845| 3.581|5000| 7.13|837.0| |non-USA|Small| 3| 1.3| 1965| 4.089|6000| 5.47|891.0| +-------+-----+---+----+------+------+----+-----------+-----+ MACHINE LEARNING WITH PYSPARK

  6. Indexing categorical data from pyspark.ml.feature import StringIndexer +-------+--------+ | type|type_idx| indexer = StringIndexer(inputCol='type', +-------+--------+ outputCol='type_idx') |Midsize| 0.0| <- most frequent value | Small| 1.0| |Compact| 2.0| # Assign index values to strings | Sporty| 3.0| indexer = indexer.fit(cars) | Large| 4.0| | Van| 5.0| <- least frequent value # Create column with index values +-------+--------+ cars = indexer.transform(cars) Use stringOrderType to change order. MACHINE LEARNING WITH PYSPARK

  7. Indexing country of origin # Index country of origin: +-------+-----+ # | origin|label| # USA -> 0 +-------+-----+ # non-USA -> 1 | USA| 0.0| # |non-USA| 1.0| cars = StringIndexer( +-------+-----+ inputCol="origin", outputCol="label" ).fit(cars).transform(cars) MACHINE LEARNING WITH PYSPARK

  8. Assembling columns Use a vector assembler to transform the data. from pyspark.ml.feature import VectorAssembler assembler = VectorAssembler(inputCols=['cyl', 'size'], outputCol='features') assembler.transform(cars) +---+----+---------+ |cyl|size| features| +---+----+---------+ | 3| 1.0|[3.0,1.0]| | 4| 1.3|[4.0,1.3]| | 3| 1.3|[3.0,1.3]| +---+----+---------+ MACHINE LEARNING WITH PYSPARK

  9. Let's practice! MACH IN E LEARN IN G W ITH P YS PARK

  10. Decision Tree MACH IN E LEARN IN G W ITH P YS PARK Andrew Collier Data Scientist, Exegetic Analytics

  11. Anatomy of a Decision Tree: Root node MACHINE LEARNING WITH PYSPARK

  12. Anatomy of a Decision Tree: First split MACHINE LEARNING WITH PYSPARK

  13. Anatomy of a Decision Tree: Second split MACHINE LEARNING WITH PYSPARK

  14. Anatomy of a Decision Tree: Third split MACHINE LEARNING WITH PYSPARK

  15. Classifying cars Classify cars according to country of manufacture. +---+----+------+------+----+-----------+----------------------------------+-----+ |cyl|size|mass |length|rpm |consumption|features |label| +---+----+------+------+----+-----------+----------------------------------+-----+ |6 |3.0 |1451.0|4.775 |5200|9.05 |[6.0,3.0,1451.0,4.775,5200.0,9.05]|1.0 | |4 |2.2 |1129.0|4.623 |5200|6.53 |[4.0,2.2,1129.0,4.623,5200.0,6.53]|0.0 | |4 |2.2 |1399.0|4.547 |5600|7.84 |[4.0,2.2,1399.0,4.547,5600.0,7.84]|1.0 | |4 |1.8 |1147.0|4.343 |6500|7.84 |[4.0,1.8,1147.0,4.343,6500.0,7.84]|0.0 | |4 |1.6 |1111.0|4.216 |5750|9.05 |[4.0,1.6,1111.0,4.216,5750.0,9.05]|0.0 | +---+----+------+------+----+-----------+----------------------------------+-----+ label = 0 -> manufactured in the USA = 1 -> manufactured elsewhere MACHINE LEARNING WITH PYSPARK

  16. Split train/test Split data into training and testing sets. # Specify a seed for reproducibility cars_train, cars_test = cars.randomSplit([0.8, 0.2], seed=23) Two DataFrames: cars_train and cars_test . [cars_train.count(), cars_test.count()] [79, 13] MACHINE LEARNING WITH PYSPARK

  17. Build a Decision Tree model from pyspark.ml.classification import DecisionTreeClassifier Create a Decision Tree classi�er. tree = DecisionTreeClassifier() Learn from the training data. tree_model = tree.fit(cars_train) MACHINE LEARNING WITH PYSPARK

  18. Evaluating Make predictions on the testing data and compare to known values. prediction = tree_model.transform(cars_test) +-----+----------+---------------------------------------+ |label|prediction|probability | +-----+----------+---------------------------------------+ |1.0 |0.0 |[0.9615384615384616,0.0384615384615385]| |1.0 |1.0 |[0.2222222222222222,0.7777777777777778]| |1.0 |1.0 |[0.2222222222222222,0.7777777777777778]| |0.0 |0.0 |[0.9615384615384616,0.0384615384615385]| |1.0 |1.0 |[0.2222222222222222,0.7777777777777778]| +-----+----------+---------------------------------------+ MACHINE LEARNING WITH PYSPARK

  19. Confusion matrix A confusion matrix is a table which describes performance of a model on testing data. prediction.groupBy("label", "prediction").count().show() +-----+----------+-----+ |label|prediction|count| +-----+----------+-----+ | 1.0| 1.0| 8| <- True positive (TP) | 0.0| 1.0| 2| <- False positive (FP) | 1.0| 0.0| 3| <- False negative (FN) | 0.0| 0.0| 6| <- True negative (TN) +-----+----------+-----+ Accuracy = (TN + TP) / (TN + TP + FN + FP) — proportion of correct predictions. MACHINE LEARNING WITH PYSPARK

  20. Let's build Decision Trees! MACH IN E LEARN IN G W ITH P YS PARK

  21. Logistic Regression MACH IN E LEARN IN G W ITH P YS PARK Andrew Collier Data Scientist, Exegetic Analytics

  22. Logistic Curve MACHINE LEARNING WITH PYSPARK

  23. Logistic Curve MACHINE LEARNING WITH PYSPARK

  24. Logistic Curve MACHINE LEARNING WITH PYSPARK

  25. Logistic Curve MACHINE LEARNING WITH PYSPARK

  26. Logistic Curve MACHINE LEARNING WITH PYSPARK

  27. Logistic Curve MACHINE LEARNING WITH PYSPARK

  28. Logistic Curve MACHINE LEARNING WITH PYSPARK

  29. Cars revisited Prepare for modeling: assemble the predictors into a single column (called features ) and split data into training and testing sets. +---+----+------+------+----+-----------+----------------------------------+-----+ |cyl|size|mass |length|rpm |consumption|features |label| +---+----+------+------+----+-----------+----------------------------------+-----+ |6 |3.0 |1451.0|4.775 |5200|9.05 |[6.0,3.0,1451.0,4.775,5200.0,9.05]|1.0 | |4 |2.2 |1129.0|4.623 |5200|6.53 |[4.0,2.2,1129.0,4.623,5200.0,6.53]|0.0 | |4 |2.2 |1399.0|4.547 |5600|7.84 |[4.0,2.2,1399.0,4.547,5600.0,7.84]|1.0 | |4 |1.8 |1147.0|4.343 |6500|7.84 |[4.0,1.8,1147.0,4.343,6500.0,7.84]|0.0 | |4 |1.6 |1111.0|4.216 |5750|9.05 |[4.0,1.6,1111.0,4.216,5750.0,9.05]|0.0 | +---+----+------+------+----+-----------+----------------------------------+-----+ MACHINE LEARNING WITH PYSPARK

  30. Build a Logistic Regression model from pyspark.ml.classification import LogisticRegression Create a Logistic Regression classi�er. logistic = LogisticRegression() Learn from the training data. logistic = logistic.fit(cars_train) MACHINE LEARNING WITH PYSPARK

  31. Predictions prediction = logistic.transform(cars_test) +-----+----------+---------------------------------------+ |label|prediction|probability | +-----+----------+---------------------------------------+ |0.0 |0.0 |[0.8683802216422138,0.1316197783577862]| |0.0 |1.0 |[0.1343792056399585,0.8656207943600416]| |0.0 |0.0 |[0.9773546766387631,0.0226453233612368]| |1.0 |1.0 |[0.0170508265586195,0.9829491734413806]| |1.0 |0.0 |[0.6122241729292978,0.3877758270707023]| +-----+----------+---------------------------------------+ MACHINE LEARNING WITH PYSPARK

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Data Preparation Data Preparation Types of Data and Basic statistics Discretization of

Data Preparation Data Preparation Types of Data and Basic statistics Discretization of

Data Preparation Data Preparation Introduction to Data Preparation Data Preparation Data Preparation Types of Data and Basic statistics Discretization of Continuous Variables Working in the R environment (Data

290 views • 24 slides
Data Preparation Discretization Data cleaning (Data pre-processing) Data

Data Preparation Discretization Data cleaning (Data pre-processing) Data

Data Preparation Why prepare the data? Data Preparation Discretization Data cleaning (Data pre-processing) Data integration and transformation Data reduction, Feature selection 2 Why Prepare Data? Why Prepare Data?

426 views • 17 slides
Data Preprocessing Data Mining and Exploration: Preprocessing Data preparation is a big issue for

Data Preprocessing Data Mining and Exploration: Preprocessing Data preparation is a big issue for

Data Preprocessing Data Mining and Exploration: Preprocessing Data preparation is a big issue for data mining. Cabena et al (1998) Amos Storkey, School of Informatics estimate that data preparation accounts for 60% of the effort in a

243 views • 3 slides
Choosing a Focus Analyzing Qualitative Data Assembled by Uta Hinrichs data preparation &amp;

Choosing a Focus Analyzing Qualitative Data Assembled by Uta Hinrichs data preparation &amp;

Choosing a Focus Analyzing Qualitative Data Assembled by Uta Hinrichs data preparation &amp; familiarization first thing is first Data preparation &amp; being familiar with the collected data is a pre-requisite for finding a theme/question

487 views • 13 slides
Data Preparation Data cleaning Data integration and transformation (Data

Data Preparation Data cleaning Data integration and transformation (Data

Data Preprocessing Why preprocess the data? Data Preparation Data cleaning Data integration and transformation (Data preprocessing) Data reduction, Feature selection Discretization and concept hierarchy generation 2

891 views • 21 slides
Choosing a Focus Analyzing Qualitative Data Uta Hinrichs data preparation &amp; familiarization

Choosing a Focus Analyzing Qualitative Data Uta Hinrichs data preparation &amp; familiarization

Choosing a Focus Analyzing Qualitative Data Uta Hinrichs data preparation &amp; familiarization first thing is first Data preparation &amp; being familiar with the collected data is a pre-requisite for finding a theme/question to focus the

612 views • 15 slides
Data Preparation Data cleaning Discretization (Data preprocessing) Data

Data Preparation Data cleaning Discretization (Data preprocessing) Data

Data Preprocessing Why preprocess the data? Data Preparation Data cleaning Discretization (Data preprocessing) Data integration and transformation Data reduction, Feature selection 2 Why Prepare Data? Why Prepare

462 views • 15 slides
Data preparation &amp; presentation Gary Collins EQUATOR Network, Centre for Statistics in

Data preparation &amp; presentation Gary Collins EQUATOR Network, Centre for Statistics in

Data preparation &amp; presentation Gary Collins EQUATOR Network, Centre for Statistics in Medicine NDORMS, University of Oxford EQUATOR Network OUCAGS training course 24 October 2015 Data preparation Prior to ANY data analysis it is

583 views • 23 slides
asking Why ? Preparation of the Gifts : Changes for Congregation: Suscipiat Dominus Preparation

asking Why ? Preparation of the Gifts : Changes for Congregation: Suscipiat Dominus Preparation

Preparation of the Gifts : INTRODUCTION Identify the changes to the words used (very few) Bring the period of the Preparation of the Gifts to life by reviewing what we do and asking Why ? Preparation of the Gifts : Changes for

538 views • 37 slides
Data Preparation for Web Usage Mining Reference :

Data Preparation for Web Usage Mining Reference :

Data Preparation for Web Usage Mining Reference : http://maya.cs.depaul.edu/~classes/ect584/papers/cms-kais.pdf Dr. Ahmed Rafea Outline A General Overview Preprocessing Data Cleaning User Identification Session

235 views • 21 slides
Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and mapping 4) Virtual Data Integration 5) Data Exchange 6) Data Warehousing 7) Big Data Analytics 8) Data Provenance 1 CS520 - 1) Introduction Overview

1.63k views • 96 slides
Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and mapping 4) Virtual Data Integration 5) Data Exchange 6) Data Warehousing 7) Big Data Analytics 8) Data Provenance 1 CS520 - 1) Introduction 2.

1.02k views • 87 slides
Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema mappings and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema mappings and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema mappings and Virtual Data Integration 4) Data Exchange 5) Data Warehousing 6) Big Data Analytics 7) Data Provenance 1 CS520 - Course Info About me I am a database

781 views • 34 slides
BIOL110L-12-Module 1-Data Preparation Oral presentations: Come prepared to describe your data

BIOL110L-12-Module 1-Data Preparation Oral presentations: Come prepared to describe your data

BIOL110L-12-Module 1-Data Preparation Oral presentations: Come prepared to describe your data (using lab terminology) and to provide interpretations of the data. The presentation is informal; the purpose is to start preparing to write Lab Report

341 views • 4 slides
Integration and Automation of Data Preparation and Data Mining Yanhui Geng Huawai Technologies

Integration and Automation of Data Preparation and Data Mining Yanhui Geng Huawai Technologies

Integration and Automation of Data Preparation and Data Mining Yanhui Geng Huawai Technologies Agenda Introduction Karma Data Modeling and Integration Prediction Task Data collection Preparing the mode of transportation

652 views • 34 slides
Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and mapping 4) Virtual Data Integration 5) Data Exchange 6) Data Warehousing 7) Big Data Analytics 8) Data Provenance 1 CS520 - 3) Matching and Mapping 4.

630 views • 50 slides
Fast dynamic and partial reconfiguration Data Path with low Hardware overhead on Xilinx FPGAs

Fast dynamic and partial reconfiguration Data Path with low Hardware overhead on Xilinx FPGAs

Fast dynamic and partial reconfiguration Data Path with low Hardware overhead on Xilinx FPGAs with low Hardware overhead on Xilinx FPGAs Michael Hbner 1 , Diana Ghringer 2 , Juanjo Noguera 3 , Jrgen Becker 1 1 Karlsruhe Institute of

370 views • 16 slides
A Synthesizable Datapath-Oriented Embedded FPGA Fabric Steven J.E. Wilton 1 , Chun Hok Ho 2 ,

A Synthesizable Datapath-Oriented Embedded FPGA Fabric Steven J.E. Wilton 1 , Chun Hok Ho 2 ,

A Synthesizable Datapath-Oriented Embedded FPGA Fabric Steven J.E. Wilton 1 , Chun Hok Ho 2 , Philip Leong 3 , Wayne Luk 2 , Brad Quinton 1 1 University of British Columbia 2 Imperial College London 3 Chinese University of Hong Kong (This work was

491 views • 9 slides
CSCI341 Lecture 30, Building a Datapath RECALL... The datapath is a representation of

CSCI341 Lecture 30, Building a Datapath RECALL... The datapath is a representation of

CSCI341 Lecture 30, Building a Datapath RECALL... The datapath is a representation of the flow of information (data, instructions) through the CPU Implemented as combination of circuitry and combinatorial &amp; sequential chips

404 views • 17 slides
CPU Organization Scope We will build a CPU to implement our subset of the MIPS ISA Memory

CPU Organization Scope We will build a CPU to implement our subset of the MIPS ISA Memory

5.1 5.2 CPU Organization Scope We will build a CPU to implement our subset of the MIPS ISA Memory Reference Instructions: Load Word (LW) EE 457 Unit 5 Store Word (SW) Arithmetic and Logic Instructions: ADD, SUB, AND, OR,

274 views • 9 slides
Programming by Example: Challenges and Opportunities Anish Doshi What this talk will cover

Programming by Example: Challenges and Opportunities Anish Doshi What this talk will cover

Programming by Example: Challenges and Opportunities Anish Doshi What this talk will cover What programming by example (PBE) is Algorithms for solving the PBE problem Integrating it into Trifacta, a production data application How

632 views • 33 slides
- - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - = 1 1 2 2 2 1 = 1 2 2 100,000 100 = 99,900 log10 5 log10 2 = 5 2 = 3

628 views • 39 slides
Introduction to Data Science: Common observation to be religion, income, frequency where sex and

Introduction to Data Science: Common observation to be religion, income, frequency where sex and

Tidying data Common problems in messy data Tidy data and the ER model Common problems in messy data Common problems in messy data Common problems in messy data Common problems in messy data Common problems in messy data Common problems in

680 views • 21 slides
Exploration of declarative languages applicability to development of large-scale data processing

Exploration of declarative languages applicability to development of large-scale data processing

Exploration of declarative languages applicability to development of large-scale data processing systems December 2016 Slavik Derevyanko, Anil Pacaci Declarative languages for distributed systems A research group at UC Berkeley lead by Prof.

593 views • 16 slides

More recommend