Review Linear separability (and use of features) Class - - PowerPoint PPT Presentation

▶
review
SMART_READER_LITE
LIVE PREVIEW

Review Linear separability (and use of features) Class - - PowerPoint PPT Presentation

Nov 06, 2022 270 likes •618 views

Review Linear separability (and use of features) Class probabilities for linear discriminants sigmoid (logistic) function Applications: USPS, fMRI ' figure from book 1 #$& 2 #$% ) 0.5 #$! #$" 0 # ! ! ! "

slide-1
SLIDE 1

Review

  • Linear separability (and use of features)
  • Class probabilities for linear discriminants

sigmoid (logistic) function

  • Applications: USPS, fMRI

!! !" # " ! # #$" #$! #$% #$& ' ( ) φ1 φ2 0.5 1 0.5 1

figure from book

1

slide-2
SLIDE 2

Review

  • Generative vs. discriminative

maximum conditional likelihood

  • Logistic regression
  • Weight space

each example adds a penalty to all weight vectors that misclassify it penalty is approximately piecewise linear

!! !" !# $ # " ! $ % # & " ' ! ()*+ !,-./0)1/2/*+

2

slide-3
SLIDE 3

Example

!! !" # " ! $ # #%! #%& #%' #%( "

3

slide-4
SLIDE 4

–log(P(Y1..3 | X1..3, W))

!" !# !$ !% " % $ & ' !& !$ !% " %

4

slide-5
SLIDE 5

Generalization: multiple classes

  • One weight vector per class: Y {1,2,…,C}

P(Y=k) = Zk =

  • In 2-class case:

5

slide-6
SLIDE 6

Multiclass example

6 4 2 2 4 6 6 4 2 2 4 6

figure from book

6

slide-7
SLIDE 7

Priors and conditional MAP

  • P(Y | X, W) =

Z =

  • As in linear regression, can put prior on W

common priors: L2 (ridge), L1 (sparsity)

  • maxw P(W=w | X, Y)

7

slide-8
SLIDE 8

Software

  • Logistic regression software is easily

available: most stats packages provide it

e.g., glm function in R

  • r, http://www.cs.cmu.edu/~ggordon/IRLS-example/
  • Most common algorithm: Newton’s method
  • n log-likelihood (or L2-penalized version)

called “iteratively reweighted least squares” for L1, slightly harder (less software available)

8

slide-9
SLIDE 9

Historical application: Fisher iris data

petal length P(I. virginica)

9

slide-10
SLIDE 10

10

slide-11
SLIDE 11

Bayesian regression

  • In linear and logistic regression, we’ve

looked at

conditional MLE: maxw P(Y | X, w) conditional MAP: maxw P(W=w | X, Y)

  • But of course, a true Bayesian would turn

up nose at both

why?

11

slide-12
SLIDE 12

Sample from posterior

!" !# !$ " $ #" !% !& !' " ' &

12

slide-13
SLIDE 13

Predictive distribution

!!" !#" " #" !" " "$! "$% "$& "$' #

13

slide-14
SLIDE 14

Overfitting

  • Overfit: training likelihood test likelihood
  • ften a result of overconfidence
  • Overfitting is an indicator that the MLE or

MAP approximation is a bad one

  • Bayesian inference rarely overfits

may still lead to bad results for other reasons! e.g., not enough data, bad model class, …

14

slide-15
SLIDE 15

So, we want the predictive distribution

  • Most of the time…

Graphical model is big and highly connected Variables are high-arity or continuous

  • Can’t afford exact inference

Inference reduces to numerical integration (and/

  • r summation)

We’ll look at randomized algorithms

15

slide-16
SLIDE 16

Numerical integration

!! !"#$ " "#$ ! !! !"#% !"#& !"#' !"#( " "#( "#' "#& "#% ! " ! ( ) ' $ * + ,-+.*/

16

slide-17
SLIDE 17

2D is 2 easy!

  • We care about high-D problems
  • Often, much of the mass is hidden in a tiny

fraction of the volume

simultaneously try to discover it and estimate amount

17

slide-18
SLIDE 18

Application: SLAM

18

slide-19
SLIDE 19

Integrals in multi-million-D

Eliazar and Parr, IJCAI-03

19

slide-20
SLIDE 20

Simple 1D problem

!! !"#$ " "#$ ! " !" %" &" '" $" (" )"

20

slide-21
SLIDE 21

Uniform sampling

!! !"#$ " "#$ ! " !" %" &" '" $" (" )"

21

slide-22
SLIDE 22

Uniform sampling

  • So, is desired integral
  • But standard deviation can be big
  • Can reduce it by averaging many samples
  • But only at rate 1/N

E(f(X)) =

22

slide-23
SLIDE 23

Importance sampling

  • Instead of X ~ uniform, use X ~ Q(x)
  • Q =
  • Should have Q(x) large where f(x) is large
  • Problem:

23

slide-24
SLIDE 24

Importance sampling

  • Instead of X ~ uniform, use X ~ Q(x)
  • Q =
  • Should have Q(x) large where f(x) is large
  • Problem:

EQ(f(X)) =

  • Q(x)f(x)dx

23

slide-25
SLIDE 25

Importance sampling

h(x) ≡ f(x)/Q(x) EQ(h(X)) =

  • Q(x)h(x)dx

=

  • Q(x)f(x)/Q(x)dx

=

  • f(x)dx

24

slide-26
SLIDE 26

Importance sampling

  • So, take samples of h(X) instead of f(X)
  • Wi = 1/Q(Xi) is importance weight
  • Q = 1/V yields uniform sampling

25

slide-27
SLIDE 27

Importance sampling

!! !"#$ " "#$ ! " !" %" &" '" $" (" )"

26

slide-28
SLIDE 28

Variance

  • How does this help us control variance?
  • Suppose:

f big Q small

  • Then h = f/Q:
  • Variance of each weighted sample is
  • Optimal Q?

27

slide-29
SLIDE 29

Importance sampling, part II

  • Suppose we want
  • Pick N samples Xi from proposal Q(X)
  • Average Wi g(Xi), where importance weight is

Wi =

  • f(x)dx =
  • P(x)g(x)dx = EP (g(X))

28

slide-30
SLIDE 30

Importance sampling, part II

  • Suppose we want
  • Pick N samples Xi from proposal Q(X)
  • Average Wi g(Xi), where importance weight is

Wi =

  • f(x)dx =
  • P(x)g(x)dx = EP (g(X))

EQ(Wg(X)) =

  • Q(x)[P(x)/Q(x)]g(x)dx =
  • P(x)g(x)dx

28

slide-31
SLIDE 31

Two variants of IS

  • Same algorithm, different terminology

want f(x) dx vs. EP(f(X)) W = 1/Q vs. W = P/Q

29

slide-32
SLIDE 32

Parallel importance sampling

  • Suppose we want
  • But P(x) is unnormalized (e.g., represented

by a factor graph)—know only Z P(x)

  • f(x)dx =
  • P(x)g(x)dx = EP (g(X))

30

slide-33
SLIDE 33

Parallel IS

  • Pick N samples Xi from proposal Q(X)
  • If we knew Wi = P(Xi)/Q(Xi), could do IS
  • Instead, set

and,

  • Then:

31

slide-34
SLIDE 34

Parallel IS

  • Final estimate:

32