Decision Tree Mahdi Roozbahani Lecturer, Computational Science and - - PowerPoint PPT Presentation

▶
decision tree
SMART_READER_LITE
LIVE PREVIEW

Decision Tree Mahdi Roozbahani Lecturer, Computational Science and - - PowerPoint PPT Presentation

Mar 10, 2024 175 likes •654 views

Class Website CX4242: Decision Tree Mahdi Roozbahani Lecturer, Computational Science and Engineering, Georgia Tech These slides are adopted from Polo, Andrew w. Moore, and Vivek Srikumar 2 1 Visual Introduction to Decision Tree

slide-1
SLIDE 1

Class Website

CX4242:

Decision Tree

Mahdi Roozbahani Lecturer, Computational Science and Engineering, Georgia Tech

These slides are adopted from Polo, Andrew w. Moore, and Vivek Srikumar

slide-2
SLIDE 2

𝑌1 𝑌2

slide-3
SLIDE 3

3

Visual Introduction to Decision Tree

Building a tree to distinguish homes in New York from homes in San Francisco

slide-4
SLIDE 4

Decision Tree: Example (2)

4

Will I play tennis today?

slide-5
SLIDE 5

The classifier: fT(x): majority class in the leaf in the tree T containing x Model parameters: The tree structure and size

5

Outlook?

Decision trees (DT)

slide-6
SLIDE 6

6

Pieces:

  • 1. Find the best attribute to split on
  • 2. Find the best split on the chosen attribute
  • 3. Decide on when to stop splitting

Decision trees

slide-7
SLIDE 7

Label

Categorical or Discrete attributes

slide-8
SLIDE 8
slide-9
SLIDE 9

Attribute

slide-10
SLIDE 10

Continuous attributes

slide-11
SLIDE 11

Test data

slide-12
SLIDE 12
slide-13
SLIDE 13
slide-14
SLIDE 14
slide-15
SLIDE 15
slide-16
SLIDE 16
slide-17
SLIDE 17

Information Content

Coin flip

Which coin will give us the purest information?

Entropy ~ Uncertainty Lower uncertainty, higher information gain

slide-18
SLIDE 18
slide-19
SLIDE 19
slide-20
SLIDE 20
slide-21
SLIDE 21
slide-22
SLIDE 22
slide-23
SLIDE 23
slide-24
SLIDE 24
slide-25
SLIDE 25
slide-26
SLIDE 26
slide-27
SLIDE 27
slide-28
SLIDE 28
slide-29
SLIDE 29
slide-30
SLIDE 30
slide-31
SLIDE 31
slide-32
SLIDE 32
slide-33
SLIDE 33
slide-34
SLIDE 34
slide-35
SLIDE 35
slide-36
SLIDE 36
slide-37
SLIDE 37
slide-38
SLIDE 38
slide-39
SLIDE 39

different

slide-40
SLIDE 40
slide-41
SLIDE 41
slide-42
SLIDE 42
slide-43
SLIDE 43
slide-44
SLIDE 44

What will happen if a tree is too large?

Overfitting High variance Instability in predicting test data

slide-45
SLIDE 45

How to avoid overfitting?

  • Acquire more training data
  • Remove irrelevant attributes (manual process – not always

possible)

  • Grow full tree, then post-prune
  • Ensemble learning
slide-46
SLIDE 46

Reduced-Error Pruning

Split data into training and validation sets Grow tree based on training set Do until further pruning is harmful:

  • 1. Evaluate impact on validation set of pruning each possible

node (plus those below it)

  • 2. Greedily remove the node that most improves validation set

accuracy

slide-47
SLIDE 47

How to decide to remove it a node using pruning

  • Pruning of the decision tree is done by replacing a whole

subtree by a leaf node.

  • The replacement takes place if a decision rule establishes that

the expected error rate in the subtree is greater than in the single leaf.