Characterizing Uncertainty in Decision Trees through Imprecise - - PowerPoint PPT Presentation

SLIDE 1

Characterizing Uncertainty in Decision Trees through Imprecise Splitting Rules

ISIPTA 2019 Malte Nalenz & Thomas Augustin July 6, 2019

Malte Nalenz & Thomas Augustin Characterizing Uncertainty in Decision Trees through Imprecise Splitting Rules July 6, 2019 1 / 5

SLIDE 2

−2 −1 1 2 −3 −2 −1 1 2 x1 x2

Malte Nalenz & Thomas Augustin Characterizing Uncertainty in Decision Trees through Imprecise Splitting Rules July 6, 2019 2 / 5

SLIDE 3

−2 −1 1 2 −3 −2 −1 1 2 x1 x2

Oberservations close to the decision boundarie(s) more uncertain, as small perturbations lead to different prediction

Malte Nalenz & Thomas Augustin Characterizing Uncertainty in Decision Trees through Imprecise Splitting Rules July 6, 2019 2 / 5

SLIDE 4

Instead of using a single splitpoint x ≤ t0 we propose to consider a neighbourhood: T = {t−k = x−k, · · · , t0, · · · , tk = xk}, where t0 is the candidate split and tk and t−k the k’th datapoints with higher and lower ordered covariate values as reasonable alternative splitting values.

Malte Nalenz & Thomas Augustin Characterizing Uncertainty in Decision Trees through Imprecise Splitting Rules July 6, 2019 3 / 5

SLIDE 5

• 2
• 1

1 2 3

• 2
• 1

1 2 3 x1 x2 Malte Nalenz & Thomas Augustin Characterizing Uncertainty in Decision Trees through Imprecise Splitting Rules July 6, 2019 4 / 5

SLIDE 6

For details and discussion please come to the poster!

◗✉❛♥t✐❢②✐♥❣❯♥❝❡rt❛✐♥t②✐♥❉❡❝✐s✐♦♥❚r❡❡s t❤r♦✉❣❤✐♠♣r❡❝✐s❡s♣❧✐tt✐♥❣❘✉❧❡s

▼❛❧t❡ ◆❛❧❡♥③✱ ❚❤♦♠❛s ❆✉❣✉st✐♥ ♠❛❧t❡✳♥❛❧❡♥③❅st❛t✳✉♥✐✲♠✉❡♥❝❤❡♥✳❞❡✱ t❤♦♠❛s✳❛✉❣✉st✐♥❅st❛t✳✉♥✐✲♠✉❡♥❝❤❡♥✳❞❡ ▼♦t✐✈❛t✐♦♥

❉❡❝✐s✐♦♥ tr❡❡ ✭❉❚✮ ✐♥❞✉❝❡rs s✉❝❝❡ss✐✈❡❧② ♣❛rt✐✲ t✐♦♥ t❤❡ ❝♦✈❛r✐❛t❡ s♣❛❝❡ x ✐♥t♦ s♠❛❧❧❡r s✉❜s♣❛❝❡s t❤❛t ❛r❡ ♣✉r❡r ✇✐t❤ r❡s♣❡❝t t♦ t❤❡✐r t❛r❣❡t ✈❛❧✲ ✉❡s y✳ ❆ ♠❛❥♦r ❞♦✇♥s✐❞❡ ♦❢ ❞❡❝✐s✐♦♥ tr❡❡s ✐s t❤❡✐r ✐♥st❛❜✐❧✐t② ✇✐t❤ r❡s♣❡❝t t♦ s♠❛❧❧ ♣❡rt✉❜❛t✐♦♥s ♦❢ t❤❡ tr❛✐♥✐♥❣ ❞❛t❛✳

❇❛s✐❝ ❈♦♥❝❡♣ts

❆t ❡❛❝❤ ♥♦❞❡ ❛ ❉❚ ✐♥❞✉❝❡r s❡❛r❝❤❡s ❢♦r t❤❡ s♣❧✐t x ≤ t t❤❛t ♠✐♥✐♠✐③❡s t❤❡ ✐♠♣✉r✐t②✱ ♠❡❛ss✉r❡❞ ❜② ❡✳❣✳ t❤❡ ❙❤❛♥♥♦♥✲❊♥tr♦♣② H(y) = −

k p(y = k)log2(p(y = k)) ♦❢ t❤❡

t✇♦ r❡s✉❧t✐♥❣ ❝❤✐❧❞♥♦❞❡s✳ ▼♦st ❝♦♠♠♦♥❧② ✉s❡❞ ❛❧❣♦r✐t❤♠s ✉s❡ ✉♥✐✈❛r✐❛t❡ s♣❧✐ts ❛♥❞ ❡①❤❛✉st✐✈❡❧② s❡❛r❝❤ ♦✈❡r ❛❧❧ ❝♦✈❛r✐❛t❡s ❢♦r t❤❡ ❜❡st ♣♦ss✐❜❧❡ s♣❧✐t✳ ■❢ t❤❡ ✇❡✐❣❤t❡❞ ✐♠♣✉r✐t② ♦❢ t❤❡ ❝❤✐❧❞♥♦❞❡s ✐s ❧♦✇❡r t❤❛♥ t❤❡ ❝✉rr❡♥t ✐♠♣✉r✐t② t❤❡ ♥♦❞❡ ✐s s♣❧✐t ❛♥❞ ♦❜s❡r✈❛t✐♦♥s ✇✐t❤ x ≤ t ♠♦✈❡❞ ❛♥❞t♦ t❤❡ r✐❣❤t ❝❤✐❧❞♥♦❞❡ ♦t❤❡r✇✐s❡✳ ❚❤✐s ♣r♦❝❡ss ✐s r❡♣❡❛t❡❞ ✉♥t✐❧ ❛ st♦♣♣✐♥❣ ❝r✐t❡r✐❛ ✐s ♠❡t✳ ❬✷❪

■♠♣r❡❝✐s❡ ❙♣❧✐tt✐♥❣ ❘✉❧❡s

■♥st❡❛❞ ♦❢ ✉s✐♥❣ ❛ s✐♥❣❧❡ s♣❧✐t ✇❡ ♣r♦♣♦s❡ t♦ ✐♥✲ st❡❛❞ ❝♦♥s✐❞❡r t❤❡ ♥❡✐❣❤❜♦✉r❤♦♦❞✿ T = {t−k = x−k, · · · , t0, · · · , tk = xk}✱ ✇❤❡r❡ t0 ✐s t❤❡ ❝❛♥❞✐❞❛t❡ s♣❧✐t ❛♥❞ tk ❛♥❞ t−k t❤❡ ❦✬t❤ ❞❛t❛♣♦✐♥ts ✇✐t❤ ❤✐❣❤❡r ❛♥❞ ❧♦✇❡r ♦r❞❡r❡❞ ❝♦✈❛r✐✲ ❛t❡ ✈❛❧✉❡s ❛s r❡❛s♦♥❛❜❧❡ ❛❧t❡r♥❛t✐✈❡ s♣❧✐tt✐♥❣ ✈❛❧✲ ✉❡s✳ ❚❤✐s ♥♦♥✲♣❛r❛♠❡tr✐❝ ❛♣♣r♦❛❝❤ ♠❛❦❡s ♥♦ ❛s✲ s✉♠♣t✐♦♥ ❛❜♦✉t t❤❡ ❞✐str✐❜✉t✐♦♥ ♦❢ x ♦r t ❜✉t ✐♥❢❡rs r❡❛s♦♥❛❜❧❡ ✈❛❧✉❡s ❞✐r❡❝t❧② ❢r♦♠ t❤❡ ❞❛t❛✳

❋✐♥❞✐♥❣ st❛❜❧❡ ❙♣❧✐ts

❲❤❡♥ ❡✈❛❧✉❛t✐♥❣ ❛ ♣♦t❡♥t✐❛❧ s♣❧✐t✱ ✇❡ ❝♦♥✲ s✐❞❡r t❤❡ ✇❤♦❧❡ ♥❡✐❣❤❜♦r❤♦♦❞ ✐♥st❡❛❞ ♦❢ ❛ s✐♥✲ ❣❧❡ ❝❛♥❞✐❞❛t❡ ❛♥❞ ❝❛❧❝✉❧❛t❡ ❛♥ ❛❣❣r❡❣❛t❡❞ ♠❡✲ ❛ss✉r❡ ¯ H(x, y, T ) =

1 |T |

• t∈T H(x, y, t)✱ ✇❤✐❝❤

♣r❡❢❡rs r❡❣✐♦♥s t❤❛t ❛r❡ ♠♦r❡ st❛❜❧❡ t♦✇❛r❞s s♠❛❧❧ ❝❤❛♥❣❡s ✐♥ x✳

◆♦♥✲❜✐♥❛r② ❙♣❧✐tt✐♥❣

■♥ ❝♦♥tr❛st t♦ tr❛❞✐t✐♦♥❛❧ ❉❚ t❤❡ ❞❡❝✐s✐♦♥ ❛t ❡❛❝❤ ♥♦❞❡ ✐s ♥♦ ❧♦♥❣❡r ❜✐♥❛r②✳ ❆t ❡❛❝❤ ♥♦❞❡ ❛♥ ♦❜s❡r✈❛t✐♦♥ ✐s ♠♦✈❡❞ t♦ t❤❡ ❧❡❢t ❝❤✐❧❞♥♦❞❡ ✇✐t❤ wl =

1 |T |

• t∈T I(x ≤ t) ❛♥❞ wr = 1 − wl t♦ t❤❡

r✐❣❤t✳ ❚❤✐s ❛❧❧♦✇s t♦ ✉s❡ ✬❝❧♦s❡ ❝❛❧❧✬ ♦❜s❡r✈❛t✐♦♥s ✐♥ ❜♦t❤ ❜r❛♥❝❤❡s ♦❢ t❤❡ tr❡❡✱ ♠❛❦✐♥❣ t❤❡ ❡①❛❝t ♣♦s✐t✐♦♥ ♦❢ t❤❡ s♣❧✐t ❧❡ss ✐♥✢✉❡♥t✐❛❧✳

■♥t❡r♣r❡t❛t✐♦♥

❈❛♥ ❜❡ ✐♥t❡r♣r❡t❡❞ ✐♥ t✇♦ ❡q✉✐✈❛❧❡♥t ✇❛②s✿ ✶✳ ❍♦✇ ❞♦ t❤❡ ♣r❡❞✐❝t✐♦♥s ❝❤❛♥❣❡ ✐❢ ✇❡ ❝❤♦♦s❡ x ≤ t0 + ǫ ❛s ❛ s❧✐❣❤t❧② ❞✐✛❡r❡♥t s♣❧✐tt✐♥❣ ♣♦✐♥t ✷✳ ❍♦✇ ❞♦ t❤❡ ♣r❡❞✐❝t✐♦♥s ❝❤❛♥❣❡ ✐❢ t❤❡ x ✈❛❧✲ ✉❡s ❛r❡ s❧✐❣❤t❧② ♣❡rt✉r❜❡❞ x − ǫ ≤ t0 ❖t❤❡r✇✐s❡ s✐♠✐❧❛r ✐♥t❡r♣r❡t❛t✐♦♥ ❛s ❝❧❛ss✐❝ ❉❚✳

❆❝❦♥♦✇❧❡❞❣❡♠❡♥ts

❲❡ t❤❛♥❦ t❤❡ r❡✈✐❡✇❡rs ♦❢ t❤❡ ♣♦st❡r ❛❜str❛❝t ❢♦r t❤❡✐r ✐♥✲ t❡r❡st✐♥❣ ❛♥❞ ✈❛❧✉❛❜❧❡ ❝♦♠♠❡♥ts t❤❛t ❤❡❧♣❡❞ t♦ ✐♠♣r♦✈❡ t❤❡ ❝♦♥t❡♥t ♦❢ t❤✐s ✇♦r❦✳

■❧❧✉str❛t✐♦♥ ♦❢ ■♠♣r❡❝✐s❡ s♣❧✐tt✐♥❣ ❙✐♠✉❧❛t❡❞ ❉❛t❛

• ❡♥❡r❛t❡ ❚✇♦✲❉✐♠❡♥s✐♦♥❛❧ ❉❛t❛ ❢♦r i = 1, · · · , 100✿

yi ∼ B(0.5), xi|yi = 1 ∼ N2( −1

−1

• ,

0.752

0.752

• , xi|yi = 0 ∼ N2(

1

1

• ,

0.752

0.752

• ))

✲✷ ✲✶ ✵ ✶ ✷ ✸ ✲✷ ✲✶ ✵ ✶ ✷ ✸ ①✶ ①✷

❚❤❡ s❡t ♦❢ s♣❧✐ts ❢♦✉♥❞ ❜② ♦✉r ♠❡t❤♦❞ ✭t0 ✭❜❧❛❝❦✮ T ✭❞❛s❤❡❞ ❣r❡②✮✮ ❛r❡ ♠♦r❡ ❝♦♥s❡r✈❛t✐✈❡ t❤❛♥ ♦♣t✐♠❛❧ s♣❧✐ts ❢r♦♠ tr❛❞✐t✐♦♥❛❧ ❉❚ ✭❜r♦✇♥✮✳ x2 ≤ T1 = {t−5 = −0.29, · · · , t0 = −0.12, · · · , t5 = 0.18}✱ x1 ≤ T2 = {t−5 = −0.59, · · · , t0 = 0.11, · · · , t5 = 0.75} ❛♥❞ x1 ≤ T3 = {t−5 = −0.65, · · · , t0 = −0.16, · · · , t5 = 0.14}✳ ❈❧❛ss✐❝ ❉❚ s♣❧✐ts✿ x2 ≤ 0.05✱ x1 ≤ −0.61 ❛♥❞ x1 ≤ 0.44✳

Pr❡❞✐❝t✐♦♥s

❚❡st ❝❛s❡s ❡♥❞ ✉♣ ✐♥ ❡✈❡r② t❡r♠✐♥❛❧ ♥♦❞❡ ✇✐t❤ ✈❛r②✐♥❣ ✇❡✐❣❤ts✳ ▲❡t η s♣❡❝✐❢② t❤❡ s❡t ♦❢ K t❡r♠✐♥❛❧ ♥♦❞❡s✳ ❋♦r ❡①❛♠♣❧❡ ♦♥❡ ♣r❡❞✐❝t✐♦♥ ❢r♦♠ t❤❡ s✐♠✉❧❛t❡❞ ❞❛t❛ ✐s✿ P(y = 1|η) = {0.38, 0.98, 0.02, 0.38}, P(η|x) = w = {0, 0.54, 0.16, 0.28}. ❋♦r ❝❛s❡s t❤❛t ❛r❡ ❢❛r ❛✇❛② ❢r♦♠ t❤❡ ❞❡❝✐s✐♦♥ ❜♦✉♥❞❛r✐❡s t②♣✐❝❛❧❧② ♠♦st ✇❡✐❣❤ts ❛r❡ 0✱ ✇❤✐❝❤ ❛♣✲ ♣r♦❛❝❤❡s t❤❡ ♣r❡❞✐❝t✐♦♥ ❢r♦♠ ❛ ❝❧❛ss✐❝ ❉❚✳

❙✐♥❣❧❡ Pr❡❞✐❝t✐♦♥

❋♦r ❛ ✜♥❛❧ s✐♥❣❧❡ ♣r❡❞✐❝t✐♦♥ t❤❡ Pr❡❞✐❝t✐♦♥ s❡t ❝❛♥ ❜❡ ❛❣❣r❡❣❛t❡❞ ❢♦r ❡①❛♠♣❧❡ ✉s✐♥❣ t❤❡ ✇❡✐❣❤t❡❞ ♠❡❛♥✿ ¯ P(y = 1|x) = K

j=1 P(y = 1|ηj) · P(ηj|x)✱ ❜✉t

♠❛① ♦r ♠❡❞✐❛♥ ❛❧s♦ r❡❛s♦♥❛❜❧❡ ❝❤♦✐❝❡s✳

Pr❡❞✐❝t✐♦♥ s♣r❡❛❞

❚❤❡ ✈❛r✐❛♥❝❡ ♦❢ t❤❡ ♣r♦❜❛❜✐❧✐t② s❡t V (P(y = 1|x)) ✐s ❛ ❣♦♦❞ ✐♥❞✐❝❛t♦r ❢♦r t❤❡ ❡①✲ ♣❡❝t❡❞ ❧♦ss✳ ❖✉r ❝❧❛ss✐✜❡r ♣❡r❢♦r♠s s✐❣♥✐✜❝❛♥t❧② ❜❡tt❡r ♦♥ t❤❡ ✬s✉r❡✬ ♣r❡❞✐❝t✐♦♥s ✇❤✐❝❤ ❤❛✈❡ ❛ ❧♦✇ ♣r♦❜❛❜✐❧✐t② s♣r❡❛❞✳

❉✐s❝✉ss✐♦♥

• ❋✐rst ❡♠♣✐r✐❝❛❧ r❡s✉❧ts s✉❣❣❡st t❤❛t ♦✉r ♠❡t❤♦❞ ♦✉t♣❡r❢♦r♠s ❝❧❛ss✐❝ ❉❚ ❛♥❞ ♦♥ s♦♠❡ ❞❛t❛s❡ts

❝❧♦s❡ t♦ r❛♥❞♦♠ ❢♦r❡sts✱ ✇❤✐❧❡ st✐❧❧ ✐♥t❡r♣r❡t❛❜❧❡✳

• ■♥❞✐❝❛t✐♦♥ ♦❢ ❡①♣❡❝t❡❞ ❧♦ss ♠✐❣❤t ❜❡ ❡①tr❡♠❧② ✉s❡❢✉❧ ❢♦r ❡♥s❡♠❜❧❡ ❧❡❛r♥✐♥❣✱ ❛s ✐t ❛❧❧♦✇s ❝❛s❡✲✇✐s❡

✇❡✐❣❤t✐♥❣ ♦❢ t❤❡ ♣r❡❞✐❝t✐♦♥s ❢r♦♠ t❤❡ ✐♥❞✐✈✐❞✉❛❧ tr❡❡s✳ ❯s✉❛❧ ❉❚ ❜❧✐♥❞ t♦✇❛r❞s ✐t✬s ✉♥❝❡rt❛✐♥t②✳

• ❖✉r ♠❡t❤♦❞ ♦♥❧② ✈❛❧✐❞ ❢♦r ♦r❞❡r❡❞ ❝♦✈❛r✐❛t❡s✱ ❜✉t ❡❛s② t♦ ❝♦♠❜✐♥❡ ✇✐t❤ ♦t❤❡r ♠❡t❤♦❞s✳ ❋♦r

❝❛t❡❣♦r✐❝❛❧ ❝♦✈❛r✐❛t❡s ❝❧❛ss✐❝ ❉❚ ❛♣♣r♦❛❝❤❡s ❝♦✉❧❞ ❜❡ ✉s❡❞ ♦r t❤❡ ■❉▼ ❬✶❪✳

• ❊①t❡♥s✐♦♥ ❢r♦♠ ❜✐♥❛r② ❝❧❛ss✐✜❝❛t✐♦♥ t♦ ♠✉❧t✐♥♦♠✐❛❧ ♦r ♥✉♠❡r✐❝ t❛r❣❡t str❛✐❣❤t❢♦r✇❛r❞✳
• ❈♦♠♣✉t❛t✐♦♥❛❧❧② s❛♠❡ ♦r❞❡r ♦❢ ❝♦♠♣❧❡①✐t② ❛s tr❛❞✐t✐♦♥❛❧ ❉❚✳
• ■♥tr♦❞✉❝❡s ❝❡rt❛✐♥ ❞❡❣r❡❡ ♦❢ s♠♦♦t❤♥❡ss✳

❘❡❢❡r❡♥❝❡s

❬✶❪ ▼❛♥t❛s✱ ❈❛r❧♦s ❏ ❛♥❞ ❆❜❡❧❧á♥✱ ❏♦❛q✉í♥ ❈r❡❞❛❧✲❈✹✳ ✺✿ ❉❡❝✐s✐♦♥ tr❡❡ ❜❛s❡❞ ♦♥ ✐♠♣r❡❝✐s❡ ♣r♦❜❛❜✐❧✐t✐❡s t♦ ❝❧❛ss✐❢② ♥♦✐s② ❞❛t❛✱ ❊①♣❡rt ❙②st❡♠s ✇✐t❤ ❆♣♣❧✐❝❛t✐♦♥s ✹✶ ✭✷✵✶✹✮ ❬✷❪ ◗✉✐♥❧❛♥✱ ❏ ❘♦ss✿ ❈✹✳ ✺✿ ♣r♦❣r❛♠s ❢♦r ♠❛❝❤✐♥❡ ❧❡❛r♥✐♥❣✱ ❊❧s❡✈✐❡r✭✷✵✶✶✮

Malte Nalenz & Thomas Augustin Characterizing Uncertainty in Decision Trees through Imprecise Splitting Rules July 6, 2019 5 / 5