Diagnosing ML System Shih-Yang Su Virginia Tech ECE-5424G / - - PowerPoint PPT Presentation

▶
diagnosing ml system
SMART_READER_LITE
LIVE PREVIEW

Diagnosing ML System Shih-Yang Su Virginia Tech ECE-5424G / - - PowerPoint PPT Presentation

Aug 25, 2022 44 likes •477 views

Diagnosing ML System Shih-Yang Su Virginia Tech ECE-5424G / CS-5824 Spring 2019 Today's Lectures Advice on how getting learning algorithms to different applications How to fix your learning algorithm Basically ZERO MATH Debugging a

slide-1
SLIDE 1

Diagnosing ML System

Shih-Yang Su Virginia Tech

ECE-5424G / CS-5824 Spring 2019

slide-2
SLIDE 2

Today's Lectures

  • Advice on how getting learning algorithms to different applications
  • How to fix your learning algorithm
  • Basically ZERO MATH
slide-3
SLIDE 3

Debugging a learning algorithm

  • You have built you awesome linear regression model predicting price
  • Work perfectly on you testing data

Source: Andrew Ng

slide-4
SLIDE 4

Debugging a learning algorithm

  • You have built you awesome linear regression model predicting price
  • Work perfectly on you testing data
  • Then it fails miserably when you test it on the new data you collected

Source: Andrew Ng

slide-5
SLIDE 5

Debugging a learning algorithm

  • You have built you awesome linear regression model predicting price
  • Work perfectly on you testing data
  • Then it fails miserably when you test it on the new data you collected
  • What to do now?

Source: Andrew Ng

slide-6
SLIDE 6

Things You Can Try

  • Get more data
  • Try different features
  • Try tuning your hyperparameter
slide-7
SLIDE 7

Things You Can Try

  • Get more data
  • Try different features
  • Try tuning your hyperparameter
  • But which should I try first?
slide-8
SLIDE 8

Diagnosing Machine Learning System

  • Figure out what is wrong first
  • Diagnosing your system takes time, but it can save your time as well
  • Ultimate goal: low generalization error
slide-9
SLIDE 9

Diagnosing Machine Learning System

  • Figure out what is wrong first
  • Diagnosing your system takes time, but it can save your time as well
  • Ultimate goal: low generalization error

Source: reddit?

slide-10
SLIDE 10

Diagnosing Machine Learning System

  • Figure out what is wrong first
  • Diagnosing your system takes time, but it can save your time as well
  • Ultimate goal: low generalization error

Source: reddit?

slide-11
SLIDE 11

Problem: Fail to Generalize

  • Model does not generalize to unseen data
  • Fail to predict things that are not in training sample
  • Pick a model that has lower generalization error
slide-12
SLIDE 12

Evaluate Your Hypothesis

Price ($) Size (ft) Price ($) Size (ft) Price ($) Size (ft) Source: Andrew Ng

slide-13
SLIDE 13

Evaluate Your Hypothesis

Price ($) Size (ft) Price ($) Size (ft) Price ($) Size (ft) Underfit Overfit Just right Source: Andrew Ng

slide-14
SLIDE 14

Evaluate Your Hypothesis

Price ($) Size (ft) Price ($) Size (ft) Price ($) Size (ft) Underfit Overfit Just right

  • What if the feature dimension is too high?

Source: Andrew Ng

slide-15
SLIDE 15

Model Selection

  • Model does not generalize to unseen data
  • Fail to predict things that are not in training sample
  • Pick a model that has lower generalization error
slide-16
SLIDE 16

Model Selection

  • Model does not generalize to unseen data
  • Fail to predict things that are not in training sample
  • Pick a model that has lower generalization error
  • How to evaluate generalization error?
slide-17
SLIDE 17

Model Selection

  • Model does not generalize to unseen data
  • Fail to predict things that are not in training sample
  • Pick a model that has lower generalization error
  • How to evaluate generalization error?
  • Split your data into train, validation, and test set.
  • Use test set error as an estimator of generalization error
slide-18
SLIDE 18

Model Selection

  • Training error
  • Validation error
  • Test error
slide-19
SLIDE 19

Model Selection

  • Training error
  • Validation error
  • Test error

Procedure: Step 1. Train on training set Step 2. Evaluate validation error Step 3. Pick the best model based on Step 2. Step 4. Evaluate the test error

slide-20
SLIDE 20

Bias/Variance Trade-off

Price ($) Size (ft) Price ($) Size (ft) Price ($) Size (ft) Underfit Overfit Just right Source: Andrew Ng

slide-21
SLIDE 21

Bias/Variance Trade-off

Price ($) Size (ft) Price ($) Size (ft) Price ($) Size (ft) Underfit High bias Overfit High Variance Just right Source: Andrew Ng

slide-22
SLIDE 22

Bias/Variance Trade-off

Price ($) Size (ft) Price ($) Size (ft) Price ($) Size (ft) Underfit High bias Too simple Overfit High Variance Too Complex Just right Source: Andrew Ng

slide-23
SLIDE 23

Linear Regression with Regularization

Price ($) Size (ft) Price ($) Size (ft) Price ($) Size (ft) Underfit High bias Too simple Too much regularization Overfit High Variance Too Complex Too little regularization Just right Source: Andrew Ng

slide-24
SLIDE 24

Bias / Variance Trade-off

  • Training error
  • Cross-validation error

Loss Degree of Polynomial Source: Andrew Ng

slide-25
SLIDE 25

Bias / Variance Trade-off

  • Training error
  • Cross-validation error

Loss Degree of Polynomial High bias High Variance

slide-26
SLIDE 26

Bias / Variance Trade-off with Regularization

  • Training error
  • Cross-validation error

Loss λ Source: Andrew Ng

slide-27
SLIDE 27

Bias / Variance Trade-off with Regularization

  • Training error
  • Cross-validation error

Loss High bias High Variance Source: Andrew Ng λ

slide-28
SLIDE 28

Problem: Fail to Generalize

  • Should we get more data?
slide-29
SLIDE 29

Problem: Fail to Generalize

  • Should we get more data?
  • Getting more data does not always help
slide-30
SLIDE 30

Problem: Fail to Generalize

  • Should we get more data?
  • Getting more data does not always help
  • How do we know if we should collect more data?
slide-31
SLIDE 31

Learning Curve

m=1 m=3 m=5 m=2 m=4 m=6

slide-32
SLIDE 32

Learning Curve

m=1 m=3 m=5 m=2 m=4 m=6

slide-33
SLIDE 33

Learning Curve

slide-34
SLIDE 34

Learning Curve

Underfit High bias Overfit High Variance

slide-35
SLIDE 35

Learning Curve

Underfit High bias

Does adding more data help?

Price ($) Size (ft)

slide-36
SLIDE 36

Learning Curve

Underfit High bias

Does adding more data help?

Price ($) Size (ft)

slide-37
SLIDE 37

Learning Curve

Underfit High bias

Does adding more data help?

Price ($) Size (ft)

slide-38
SLIDE 38

Learning Curve

Does adding more data help?

Price ($) Size (ft) Price ($) Size (ft)

More data doesn't help when your model has high bias

slide-39
SLIDE 39

Learning Curve

Does adding more data help?

Overfit High Variance Price ($) Size (ft)

slide-40
SLIDE 40

Learning Curve

Does adding more data help?

Overfit High Variance Price ($) Size (ft)

slide-41
SLIDE 41

Learning Curve

Does adding more data help? More data is likely to help when your model has high variance

Price ($) Size (ft) Price ($) Size (ft)

slide-42
SLIDE 42

Things You Can Try

  • Get more data
  • When you have high variance
  • Try different features
  • Adding feature helps fix high bias
  • Using smaller sets of feature fix high variance
  • Try tuning your hyperparameter
  • Decrease regularization when bias is high
  • Increase regularization when variance is high
slide-43
SLIDE 43

Things You Can Try

  • Get more data
  • When you have high variance
  • Try different features
  • Adding feature helps fix high bias
  • Using smaller sets of feature fix high variance
  • Try tuning your hyperparameter
  • Decrease regularization when bias is high
  • Increase regularization when variance is high

Analyze your model before you act