Data-Intensive Distributed Computing CS 431/631 451/651 (Winter - - PowerPoint PPT Presentation

Data-Intensive Distributed Computing

Part 6: Data Mining (1/4)

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States See http://creativecommons.org/licenses/by-nc-sa/3.0/us/ for details

CS 431/631 451/651 (Winter 2019) Ali Abedi October 29, 2019

These slides are available at https://www.student.cs.uwaterloo.ca/~cs451

1

Structure of the Course

“Core” framework features and algorithm design

Analyzing Text Analyzing Graphs Analyzing Relational Data Data Mining

2

Descriptive vs. Predictive Analytics

3

Frontend Backend

users

Frontend Backend

users

Frontend Backend

external APIs

“Traditional” BI tools SQL on Hadoop Other tools

Data Warehouse “Data Lake” data scientists OLTP database ETL

(Extract, Transform, and Load)

OLTP database OLTP database

4

Supervised Machine Learning

The generic problem of function induction given sample instances of input and output

Classification: output draws from finite discrete labels Regression: output is a continuous value

This is not meant to be an exhaustive treatment of machine learning!

5

Source: Wikipedia (Sorting)

Classification

6

Applications

Spam detection Sentiment analysis Content (e.g., topic) classification Link prediction Document ranking Object recognition And much much more! Fraud detection

7

training Model Machine Learning Algorithm testing/deployment

?

Supervised Machine Learning

8

Objects are represented in terms of features:

“Dense” features: sender IP, timestamp, # of recipients, length of message, etc. “Sparse” features: contains the term “Viagra” in message, contains “URGENT” in subject, etc.

Feature Representations

9

Applications

Spam detection Sentiment analysis Content (e.g., genre) classification Link prediction Document ranking Object recognition And much much more! Fraud detection

10

Components of a ML Solution

Data Features Model Optimization

11

(Banko and Brill, ACL 2001) (Brants et al., EMNLP 2007)

No data like more data!

12

Limits of Supervised Classification?

Why is this a big data problem?

Isn’t gathering labels a serious bottleneck?

Solutions

Crowdsourcing Bootstrapping, semi-supervised techniques Exploiting user behavior logs

The virtuous cycle of data-driven products

13

a useful service analyze user behavior to extract insights transform insights into action

$

(hopefully)

• Google. Facebook. Twitter. Amazon. Uber.

data science data products

Virtuous Product Cycle

14

What’s the deal with neural networks?

Data Features Model Optimization

15

Supervised Binary Classification

Restrict output label to be binary

Yes/No 1/0

Binary classifiers form primitive building blocks for multi-class problems…

16

Binary Classifiers as Building Blocks

Example: four-way classification One vs. rest classifiers

A or not? B or not? C or not? D or not? A or not? B or not? C or not? D or not?

Classifier cascades

17

The Task

Given:

(sparse) feature vector label

Induce:

Such that loss is minimized loss function

Typically, we consider functions of a parametric form:

model parameters

18

Key insight: machine learning as an optimization problem!

(closed form solutions generally not possible)

19

Gradient Descent: Preliminaries

Rewrite: Compute gradient:

“Points” to fastest increasing “direction”

So, at any point:

20

Gradient Descent: Iterative Update

Start at an arbitrary point, iteratively update: We have:

21

Old weights Update based on gradient New weights

Intuition behind the math…

22

Gradient Descent: Iterative Update

Start at an arbitrary point, iteratively update: Lots of details:

Figuring out the step size Getting stuck in local minima Convergence rate …

We have:

23

Repeat until convergence:

Gradient Descent

Note, sometimes formulated as ascent but entirely equivalent

24

Gradient Descent

Source: Wikipedia (Hills)

25

Even More Details…

Gradient descent is a “first order” optimization technique

Often, slow convergence

Newton and quasi-Newton methods:

Intuition: Taylor expansion Requires the Hessian (square matrix of second order partial derivatives): impractical to fully compute

26

Source: Wikipedia (Hammer)

Logistic Regression

27

Logistic Regression: Preliminaries

Given: Define: Interpretation:

28

Relation to the Logistic Function

After some algebra: The logistic function:

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

• 8
• 7
• 6
• 5
• 4
• 3
• 2
• 1

1 2 3 4 5 6 7 8

logistic(z) z

29

Training an LR Classifier

Maximize the conditional likelihood: Define the objective in terms of conditional log likelihood: We know: So: Substituting:

30

LR Classifier Update Rule

Take the derivative: General form of update rule: Final update rule:

31

Want more details? Take a real machine-learning course!

Lots more details…

Regularization Different loss functions …

32

mapper mapper mapper mapper reducer

compute partial gradient single reducer mappers update model iterate until convergence

MapReduce Implementation

33

Shortcomings

Hadoop is bad at iterative algorithms

High job startup costs Awkward to retain state across iterations

High sensitivity to skew

Iteration speed bounded by slowest task

Potentially poor cluster utilization

Must shuffle all data to a single reducer

Some possible tradeoffs

Number of iterations vs. complexity of computation per iteration E.g., L-BFGS: faster convergence, but more to compute

34

val points = spark.textFile(...).map(parsePoint).persist() var w = // random initial vector for (i <- 1 to ITERATIONS) { val gradient = points.map{ p => p.x * (1/(1+exp(-p.y*(w dot p.x)))-1)*p.y }.reduce((a,b) => a+b) w -= gradient } mapper mapper mapper mapper reducer

compute partial gradient update model

Spark Implementation

35