Instance Based Learning [Read Ch. 8] k -Nearest Neigh b - - PDF document
▶
SMART_READER_LITE
LIVE PREVIEW
Instance Based Learning [Read Ch. 8] k -Nearest Neigh b - - PDF document
Apr 28, 2023 373 likes •542 views
Instance Based Learning [Read Ch. 8] k -Nearest Neigh b or Lo cally w eigh ted regression Radial basis functions Case-based reasoning Lazy and eager learning 199 lecture slides for textb o ok
SLIDE 1Instance Based Learning [Read Ch. 8]
k
Nearest
Neigh b
r
Lo
cally w eigh ted regression
Radial
basis functions
Case-based
reasoning
Lazy
and eager learning 199 lecture slides for textb
k
Machine L e arning, c
T
m
M. Mitc hell, McGra w Hill, 1997
SLIDE 2Instance-Based Learning Key idea: just store all training examples hx i ; f (x i )i Nearest neigh b
r:
Giv
en query instance x q , rst lo cate nearest training example x n , then estimate ^ f (x q ) f (x n ) k
Nearest
neigh b
r:
Giv
en x q , tak e v
te
among its k nearest n brs (if discrete-v alued target function)
tak
e mean
f
f v alues
f
k nearest n brs (if real-v alued) ^ f (x q ) P k i=1 f (x i ) k 200 lecture slides for textb
k
Machine L e arning, c
T
m
M. Mitc hell, McGra w Hill, 1997
SLIDE 3When T
Consider
Nearest Neigh b
r
Instances
map to p
in
ts in < n
Less
than 20 attributes p er instance
Lots
f
training data Adv an tages:
T
raining is v ery fast
Learn
complex target functions
Don't
lose information Disadv an tages:
Slo
w at query time
Easily
fo
led
b y irrelev an t attributes 201 lecture slides for textb
k
Machine L e arning, c
T
m
M. Mitc hell, McGra w Hill, 1997
SLIDE 4V
ronoi
Diagram
+ + − − − + − − + xq
202 lecture slides for textb
k
Machine L e arning, c
T
m
M. Mitc hell, McGra w Hill, 1997
SLIDE 5Beha vior in the Limit Consider p(x) denes probabilit y that instance x will b e lab eled 1 (p
sitiv
e) v ersus (negativ e). Nearest neigh b
r:
As
n um b er
f
training examples ! 1, approac hes Gibbs Algorithm Gibbs: with probabilit y p(x) predict 1, else k
Nearest
neigh b
r:
As
n um b er
f
training examples ! 1 and k gets large, approac hes Ba y es
ptimal
Ba y es
ptimal:
if p(x) > :5 then predict 1, else Note Gibbs has at most t wice the exp ected error
f
Ba y es
ptimal
203 lecture slides for textb
k
Machine L e arning, c
T
m
M. Mitc hell, McGra w Hill, 1997
SLIDE 6Distance-W eigh ted k NN Migh t w an t w eigh t nearer neigh b
rs
more hea vily ... ^ f (x q ) P k i=1 w i f (x i ) P k i=1 w i where w i
1
d(x q ; x i ) 2 and d(x q ; x i ) is distance b et w een x q and x i Note no w it mak es sense to use al l training examples instead
f
just k ! Shepard's metho d 204 lecture slides for textb
k
Machine L e arning, c
T
m
M. Mitc hell, McGra w Hill, 1997
SLIDE 7Curse
f
Dimensionali t y Imagine instances describ ed b y 20 attributes, but
nly
2 are relev an t to target function Curse
f
dimensionality: nearest n br is easily mislead when high-dimensional X One approac h:
Stretc
h j th axis b y w eigh t z j , where z 1 ; : : : ; z n c hosen to minimize prediction error
Use
cross-v alidati
n
to automatically c ho
se
w eigh ts z 1 ; : : : ; z n
Note
setting z j to zero eliminates this dimension altogether see [Mo
re
and Lee, 1994] 205 lecture slides for textb
k
Machine L e arning, c
T
m
M. Mitc hell, McGra w Hill, 1997
SLIDE 8Lo cally W eigh ted Regression Note k NN forms lo cal appro ximation to f for eac h query p
in
t x q Wh y not form an explici t appro ximation ^ f (x) for region surrounding x q
Fit
linear function to k nearest neigh b
rs
Fit
quadratic, ...
Pro
duces \piecewise appro ximation" to f Sev eral c hoices
f
error to minimize:
Squared
error
v
er k nearest neigh b
rs
E 1 (x q )
1
2 X x2 k near est nbr s
f
x q (f (x)
^
f (x)) 2
Distance-w
eigh ted squared error
v
er all n brs E 2 (x q )
1
2 X x2D (f (x)
^
f (x)) 2 K (d(x q ; x))
:
: : 206 lecture slides for textb
k
Machine L e arning, c
T
m
M. Mitc hell, McGra w Hill, 1997
SLIDE 9Radial Basis F unction Net w
rks
Global
appro ximation to target function, in terms
f
linear com bination
f
lo cal appro ximations
Used,
e.g., for image classicati
n
A
dieren t kind
f
neural net w
rk
Closely
related to distance-w eigh ted regression, but \eager" instead
f
\lazy" 207 lecture slides for textb
k
Machine L e arning, c
T
m
M. Mitc hell, McGra w Hill, 1997
SLIDE 10Radial Basis F unction Net w
rks
... ...
f(x) w
1
w0 wk 1
1
a (x)
2
a (x)
n
a (x)
where a i (x) are the attributes describing instance x, and f (x) = w + k X u=1 w u K u (d(x u ; x)) One common c hoice for K u (d(x u ; x)) is K u (d(x u ; x)) = e
1
2 2 u d 2 (x u ;x) 208 lecture slides for textb
k
Machine L e arning, c
T
m
M. Mitc hell, McGra w Hill, 1997
SLIDE 11T raining Radial Basis F unction Net- w
rks
Q1: What x u to use for eac h k ernel function K u (d(x u ; x))
Scatter
uniformly throughout instance space
Or
use training instances (reects instance distribution) Q2: Ho w to train w eigh ts (assume here Gaussian K u )
First
c ho
se
v ariance (and p erhaps mean) for eac h K u { e.g., use EM
Then
hold K u xed, and train linear
utput
la y er { ecien t metho ds to t linear function 209 lecture slides for textb
k
Machine L e arning, c
T
m
M. Mitc hell, McGra w Hill, 1997
SLIDE 12Case-Based Reasoning Can apply instance-based learning ev en when X 6= < n ! need dieren t \distance" metric Case-Based Reasoning is instance-based learning applied to instances with sym b
