Lecture 1: Linear Regression Princeton University COS 495 - - PowerPoint PPT Presentation

Machine Learning Basics Lecture 1: Linear Regression

Princeton University COS 495 Instructor: Yingyu Liang

Machine learning basics

What is machine learning?

• “A computer program is said to learn from experience E with respect

to some class of tasks T and performance measure P, if its performance at tasks in T as measured by P, improves with experience E.”

• ------ Machine Learning, Tom Mitchell, 1997

Example 1: image classification

Task: determine if the image is indoor or outdoor Performance measure: probability of misclassification

Example 1: image classification

indoor

• utdoor

Experience/Data: images with labels Indoor

Example 1: image classification

• A few terminologies
• Training data: the images given for learning
• Test data: the images to be classified
• Binary classification: classify into two classes

Example 1: image classification (multi-class)

ImageNet figure borrowed from vision.standford.edu

Example 2: clustering images

Task: partition the images into 2 groups Performance: similarities within groups Data: a set of images

Example 2: clustering images

• A few terminologies
• Unlabeled data vs labeled data
• Supervised learning vs unsupervised learning

Math formulation

Color Histogram

Red Green Blue

Indoor

Feature vector: 𝑦𝑗 Label: 𝑧𝑗 Extract features

Math formulation

Color Histogram

Red Green Blue

• utdoor

1

Feature vector: 𝑦𝑘 Label: 𝑧𝑘 Extract features

Math formulation

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜
• Find 𝑧 = 𝑔(𝑦) using training data
• s.t. 𝑔 correct on test data

What kind of functions?

Math formulation

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜
• Find 𝑧 = 𝑔(𝑦) ∈ 𝓘 using training data
• s.t. 𝑔 correct on test data

Hypothesis class

Math formulation

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜
• Find 𝑧 = 𝑔(𝑦) ∈ 𝓘 using training data
• s.t. 𝑔 correct on test data

Connection between training data and test data?

Math formulation

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜 i.i.d. from distribution 𝐸
• Find 𝑧 = 𝑔(𝑦) ∈ 𝓘 using training data
• s.t. 𝑔 correct on test data i.i.d. from distribution 𝐸

They have the same distribution i.i.d.: independently identically distributed

Math formulation

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜 i.i.d. from distribution 𝐸
• Find 𝑧 = 𝑔(𝑦) ∈ 𝓘 using training data
• s.t. 𝑔 correct on test data i.i.d. from distribution 𝐸

What kind of performance measure?

Math formulation

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜 i.i.d. from distribution 𝐸
• Find 𝑧 = 𝑔(𝑦) ∈ 𝓘 using training data
• s.t. the expected loss is small

𝑀 𝑔 = 𝔽 𝑦,𝑧 ~𝐸[𝑚(𝑔, 𝑦, 𝑧)]

Various loss functions

Math formulation

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜 i.i.d. from distribution 𝐸
• Find 𝑧 = 𝑔(𝑦) ∈ 𝓘 using training data
• s.t. the expected loss is small

𝑀 𝑔 = 𝔽 𝑦,𝑧 ~𝐸[𝑚(𝑔, 𝑦, 𝑧)]

• Examples of loss functions:
• 0-1 loss: 𝑚 𝑔, 𝑦, 𝑧 = 𝕁[𝑔 𝑦 ≠ 𝑧] and 𝑀 𝑔 = Pr[𝑔 𝑦 ≠ 𝑧]
• 𝑚2 loss: 𝑚 𝑔, 𝑦, 𝑧 = [𝑔 𝑦 − 𝑧]2 and 𝑀 𝑔 = 𝔽[𝑔 𝑦 − 𝑧]2

Math formulation

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜 i.i.d. from distribution 𝐸
• Find 𝑧 = 𝑔(𝑦) ∈ 𝓘 using training data
• s.t. the expected loss is small

𝑀 𝑔 = 𝔽 𝑦,𝑧 ~𝐸[𝑚(𝑔, 𝑦, 𝑧)]

How to use?

Math formulation

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜 i.i.d. from distribution 𝐸
• Find 𝑧 = 𝑔(𝑦) ∈ 𝓘 that minimizes ෠

𝑀 𝑔 =

1 𝑜 σ𝑗=1 𝑜

𝑚(𝑔, 𝑦𝑗, 𝑧𝑗)

• s.t. the expected loss is small

𝑀 𝑔 = 𝔽 𝑦,𝑧 ~𝐸[𝑚(𝑔, 𝑦, 𝑧)]

Empirical loss

Machine learning 1-2-3

• Collect data and extract features
• Build model: choose hypothesis class 𝓘 and loss function 𝑚
• Optimization: minimize the empirical loss

Wait…

• Why handcraft the feature vectors 𝑦, 𝑧?
• Can use prior knowledge to design suitable features
• Can computer learn the features on the raw images?
• Learn features directly on the raw images: Representation Learning
• Deep Learning ⊆ Representation Learning ⊆ Machine Learning ⊆ Artificial

Intelligence

Wait…

• Does MachineLearning-1-2-3 include all approaches?
• Include many but not all
• Our current focus will be MachineLearning-1-2-3

Example: Stock Market Prediction

2013 2014 2015 2016

Stock Market (Disclaimer: synthetic data/in another parallel universe)

Orange MacroHard Ackermann

Sliding window over time: serve as input 𝑦; non-i.i.d.

Linear regression

Real data: Prostate Cancer by Stamey et al. (1989)

Figure borrowed from The Elements of Statistical Learning

𝑧: prostate

specific antigen

(𝑦1, … , 𝑦8): clinical measures

Linear regression

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜 i.i.d. from distribution 𝐸
• Find 𝑔

𝑥 𝑦 = 𝑥𝑈𝑦 that minimizes ෠

𝑀 𝑔

𝑥 = 1 𝑜 σ𝑗=1 𝑜

𝑥𝑈𝑦𝑗 − 𝑧𝑗 2

𝑚2 loss; also called mean square error

Hypothesis class 𝓘

Linear regression: optimization

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜 i.i.d. from distribution 𝐸
• Find 𝑔

𝑥 𝑦 = 𝑥𝑈𝑦 that minimizes ෠

𝑀 𝑔

𝑥 = 1 𝑜 σ𝑗=1 𝑜

𝑥𝑈𝑦𝑗 − 𝑧𝑗 2

• Let 𝑌 be a matrix whose 𝑗-th row is 𝑦𝑗

𝑈, 𝑧 be the vector 𝑧1, … , 𝑧𝑜 𝑈

෠ 𝑀 𝑔

𝑥 = 1

𝑜 ෍

𝑗=1 𝑜

𝑥𝑈𝑦𝑗 − 𝑧𝑗 2 = 1 𝑜 ⃦𝑌𝑥 − 𝑧 ⃦2

2

Linear regression: optimization

• Set the gradient to 0 to get the minimizer

𝛼

𝑥 ෠

𝑀 𝑔

𝑥 = 𝛼 𝑥

1 𝑜 ⃦𝑌𝑥 − 𝑧 ⃦2

2 = 0

𝛼

𝑥[ 𝑌𝑥 − 𝑧 𝑈(𝑌𝑥 − 𝑧)] = 0

𝛼

𝑥[ 𝑥𝑈𝑌𝑈𝑌𝑥 − 2𝑥𝑈𝑌𝑈𝑧 + 𝑧𝑈𝑧] = 0

2𝑌𝑈𝑌𝑥 − 2𝑌𝑈𝑧 = 0 w = 𝑌𝑈𝑌 −1𝑌𝑈𝑧

Linear regression: optimization

• Algebraic view of the minimizer
• If 𝑌 is invertible, just solve 𝑌𝑥 = 𝑧 and get 𝑥 = 𝑌−1𝑧
• But typically 𝑌 is a tall matrix

𝑌 𝑥 = 𝑧 𝑌𝑈𝑌 𝑥 = 𝑌𝑈𝑧 Normal equation: w = 𝑌𝑈𝑌 −1𝑌𝑈𝑧

Linear regression with bias

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜 i.i.d. from distribution 𝐸
• Find 𝑔

𝑥,𝑐 𝑦 = 𝑥𝑈𝑦 + 𝑐 to minimize the loss

• Reduce to the case without bias:
• Let 𝑥′ = 𝑥; 𝑐 , 𝑦′ = 𝑦; 1
• Then 𝑔

𝑥,𝑐 𝑦 = 𝑥𝑈𝑦 + 𝑐 = 𝑥′ 𝑈(𝑦′)

Bias term