E (h) o r out in ( h 1 ) E out ( h 1 ) | > o r in ( h 2 ) - - PowerPoint PPT Presentation

▶
e
SMART_READER_LITE
LIVE PREVIEW

E (h) o r out in ( h 1 ) E out ( h 1 ) | > o r in ( h 2 ) - - PowerPoint PPT Presentation

Aug 11, 2023 367 likes •631 views

Review of Leture 2 Sine g has to b e one of h 1 , h 2 , , h M , w e onlude that Is Lea rning feasible? Y es, in a p robabilisti sense. If: in ( g ) E out ( g ) | > Hi Then: | E E (h) o r out in (

slide-1
SLIDE 1 Review
  • f
Le ture 2 Is Lea rning feasible? Y es, in a p robabilisti sense.

Hi Hi

E E

(h)

  • ut

in(h)

P [ |E

in(h) − E
  • ut(h)| > ǫ ] ≤ 2e−2ǫ2N
Sin e g has to b e
  • ne
  • f h1, h2, · · · , hM
, w e
  • n lude
that If:

|E

in(g) − E
  • ut(g)| > ǫ
Then:

|E

in(h1) − E
  • ut(h1)| > ǫ
  • r

|E

in(h2) − E
  • ut(h2)| > ǫ
  • r

· · · |E

in(hM) − E
  • ut(hM)| > ǫ
This gives us an added M fa to r.
slide-2
SLIDE 2 Lea rning F rom Data Y aser S. Abu-Mostafa Califo rnia Institute
  • f
T e hnology Le ture 3: Linea r Mo dels I Sp
  • nso
red b y Calte h's Provost O e, E&AS Division, and IST
  • T
uesda y , Ap ril 10, 2012
slide-3
SLIDE 3 Outline
  • Input
rep resentation
  • Linea
r Classi ation
  • Linea
r Regression
  • Nonlinea
r T ransfo rmation

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 2/23
slide-4
SLIDE 4 A real data set

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 3/23
slide-5
SLIDE 5 Input rep resentation `ra w' input x = (x0,x1, x2, · · · , x256) linea r mo del:

(w0, w1, w2, · · · , w256)

F eatures: Extra t useful info rmation, e.g., intensit y and symmetry x = (x0,x1, x2) linea r mo del:

(w0, w1, w2)

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 4/23
slide-6
SLIDE 6 Illustration
  • f
features

x = (x0,x1, x2) x1

: intensit y

x2

: symmetry PSfrag repla ements 2 4 6 8 10 12 14 16 18 2 4 6 8 10 12 14 16 18 2 4 6 8 10 12 14 16 18 5 10 15 5 10 15 5 10 15 PSfrag repla ements 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
  • 8
  • 7
  • 6
  • 5
  • 4
  • 3
  • 2
  • 1
PSfrag repla ements 2 4 6 8 10 12 14 16 18 2 4 6 8 10 12 14 16 18 2 4 6 8 10 12 14 16 18 5 10 15 5 10 15 5 10 15

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 5/23
slide-7
SLIDE 7 What PLA do es Evolution
  • f E
in and E
  • ut
Final p er eptron b
  • unda
ry PSfrag repla ements

E

  • ut

E

in 250 500 750 1000 1% 10% 50% PSfrag repla ements A verage Intensit y Symmetry 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4
  • 8
  • 7
  • 6
  • 5
  • 4
  • 3
  • 2
  • 1
1

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 6/23
slide-8
SLIDE 8 The `p
  • k
et' algo rithm PLA: P
  • k
et: PSfrag repla ements

E

  • ut

E

in 250 500 750 1000 1% 10% 50% PSfrag repla ements

E

  • ut

E

in 250 500 750 1000 1% 10% 50%

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 7/23
slide-9
SLIDE 9 Classi ation b
  • unda
ry
  • PLA
versus P
  • k
et PLA: P
  • k
et: PSfrag repla ements A verage Intensit y Symmetry 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4
  • 8
  • 7
  • 6
  • 5
  • 4
  • 3
  • 2
  • 1
1 PSfrag repla ements A verage Intensit y Symmetry 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4
  • 8
  • 7
  • 6
  • 5
  • 4
  • 3
  • 2
  • 1
1

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 8/23
slide-10
SLIDE 10 Outline
  • Input
rep resentation
  • Linea
r Classi ation
  • Linea
r Regression regression ≡ real-valued
  • utput
  • Nonlinea
r T ransfo rmation

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 9/23
slide-11
SLIDE 11 Credit again Classi ation: Credit app roval (y es/no) Regression: Credit line (dolla r amount) Input: x = age 23 y ea rs annual sala ry $30,000 y ea rs in residen e 1 y ea r y ea rs in job 1 y ea r urrent debt $15,000

· · · · · ·

Linea r regression
  • utput: h(x) =

d

  • i=0

wi xi = w

Tx

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 10/23
slide-12
SLIDE 12 The data set Credit
  • ers
de ide
  • n
redit lines:

(x1, y1), (x2, y2), · · · , (xN, yN) yn ∈ R

is the redit line fo r ustomer xn . Linea r regression tries to repli ate that.

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 11/23
slide-13
SLIDE 13 Ho w to measure the erro r Ho w w ell do es h(x) = w Tx app ro ximate f(x) ? In linea r regression, w e use squa red erro r (h(x) − f(x))2 in-sample erro r: E in(h) = 1

N

N

  • n=1

(h(xn) − yn)2

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 12/23
slide-14
SLIDE 14 Illustration
  • f
linea r regression PSfrag repla ements

x y

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 PSfrag repla ements

x1 x2 y

0.5 1 0.2 0.4 0.6 0.8 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 PSfrag repla ements

x1 x2 y

0.5 1 0.2 0.4 0.6 0.8 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 13/23
slide-15
SLIDE 15 The exp ression fo r E in

E

in(w) =

1 N

N

  • n=1

(w

Txn − yn)2

= 1 NXw − y2

where

X =     

x1 T
  • x2
T
  • .
. . xN T

    , y =      y1 y2

. . .

yN     

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 14/23
slide-16
SLIDE 16 Minimizing E in

E

in(w) = 1

NXw − y2

∇E

in(w) = 2

NX

T(Xw − y) = 0

X

TXw = X Ty

w = X†y

where X† = (X TX)−1X T

X†

is the `pseudo-inverse'
  • f X

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 15/23
slide-17
SLIDE 17 The pseudo-inverse

X† = (X

TX)−1X T

        

  • d+1× N

         

N × d+1

        

−1

  • d+1 × N
  • d+1 × N
  • d+1 × d+1

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 16/23
slide-18
SLIDE 18 The linea r regression algo rithm 1: Constru t the matrix X and the ve to r y from the data set

(x1, y1), · · · , (xN, yN)

as follo ws

X =     

x T

1

  • x
T

2

  • .
. . x T

N

   

  • input
data matrix

, y =      y1 y2

. . .

yN     

  • ta
rget ve to r

.

2: Compute the pseudo-inverse X† = (X TX)−1X T . 3: Return w = X†y .

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 17/23
slide-19
SLIDE 19 Linea r regression fo r lassi ation Linea r regression lea rns a real-valued fun tion y = f(x) ∈ R Bina ry-valued fun tions a re also real-valued! ±1 ∈ R Use linea r regression to get w where w Txn ≈ yn = ±1 In this ase, sign(w Txn) is lik ely to agree with yn = ±1 Go
  • d
initial w eights fo r lassi ation

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 18/23
slide-20
SLIDE 20 Linea r regression b
  • unda
ry PSfrag repla ements A verage Intensit y Symmetry 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55
  • 8
  • 7
  • 6
  • 5
  • 4
  • 3
  • 2
  • 1

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 18/23
slide-21
SLIDE 21 Outline
  • Input
rep resentation
  • Linea
r Classi ation
  • Linea
r Regression
  • Nonlinea
r T ransfo rmation

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 19/23
slide-22
SLIDE 22 Linea r is limited Data: Hyp
  • thesis:
PSfrag repla ements

−1 1 −1 1

PSfrag repla ements

−1 1 −1 1

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 20/23
slide-23
SLIDE 23 Another example Credit line is ae ted b y `y ea rs in residen e' but not in a linea r w a y! Nonlinea r [[xi < 1]] and [[xi > 5]] a re b etter. Can w e do that with linea r mo dels?

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 21/23
slide-24
SLIDE 24 Linea r in what? Linea r regression implements

d

  • i=0

wi xi

Linea r lassi ation implements sign

d

  • i=0

wi xi

  • Algo
rithms w
  • rk
b e ause
  • f
linea rit y in the w eights

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 22/23
slide-25
SLIDE 25 T ransfo rm the data nonlinea rly

(x1, x2)

Φ

− → (x2

1, x2 2)

PSfrag repla ements

−1 1 −1 1

PSfrag repla ements

0.5 1 0.5 1

A

M L

Creato r: Y aser Abu-Mostafa
  • LFD
Le ture 3 23/23