Bayesian Method for Repeated Threshold Estimation Alexander Petrov - - PowerPoint PPT Presentation

Bayesian Method for Repeated Threshold Estimation

Alexander Petrov

Department of Psychology Ohio State University

5/6/06 http://alexpetrov.com/pub/vss06/

2

Motivation: Perceptual Learning

Non-stationary thresholds Dynamics of learning is important Must use naïve observers Low motivation high lapsing rates Slow learning many sessions Large volume of low-quality binary data

5/6/06 http://alexpetrov.com/pub/vss06/

3

Objective: Data Reduction

5/6/06 http://alexpetrov.com/pub/vss06/

4

Isn’t This a Solved Problem?

Up/down (Levitt, 1970) PEST (Taylor & Creelman, 1967) BEST PEST (Pentland, 1980) QUEST (Watson & Pelli, 1979) ML-Test (Harvey, 1986) Ideal (Pelli, 1987) YAAP (Treutwein, 1989) and many others…

5/6/06 http://alexpetrov.com/pub/vss06/

5

We Solve a Different Problem

Standard methods:

Adaptive stimulus placement Stopping criterion Threshold estimation

Our method:

Threshold estimation Integrate information across blocks

5/6/06 http://alexpetrov.com/pub/vss06/

6

Weibull Psychometric Function

Threshold log α Slope β Guessing rate γ Lapsing rate λ

( ; , ) 1 exp( exp((log log ) )) ( ; , , , ) (1 ) ( ; , ) W x x P x W x α β α β α β γ λ γ γ λ α β = − − − = + − −

γ

1-λ logα logx

5/6/06 http://alexpetrov.com/pub/vss06/

7

Two Kinds of Parameters

Threshold log α Slope β Guessing rate γ Lapsing rate λ

Parameters

• f interest θ

Nuisance parameters φ

The nuisance parameters are harder to estimate but change more slowly than the threshold parameter.

5/6/06 http://alexpetrov.com/pub/vss06/

8

Get the Best of Both Worlds

Use long data sequences to constrain the nuisance parameters; use short sequences to estimate the thresholds.

5/6/06 http://alexpetrov.com/pub/vss06/

9

Joint Posterior of θk,φ

1 1 1

( , | ; , ) ( | , ) ( ) ( ) ( ) ( | , )

k k k k n k k k i i i i i k

p p p p p p d θ φ θ φ θ φ θ θ φ θ

− + ≠

=

∏∫

y y y y y y y … …

Likelihood of current data Priors Information about φ extracted from the other data sets Modified prior for the current block

5/6/06 http://alexpetrov.com/pub/vss06/

10

Pass 1: for each block i, calculate Pass 2: for each block k, calculate

(

Two-Pass Algorithm

| ) ( ) ( | , )

i i

p p p d φ θ θ φ θ = ∫ y y ( , | ) ( | , ) ( ) ( ) ( | )

k k k k k i i k

p p p p p θ φ θ φ θ φ φ

≠

=

∏∫

y y y

5/6/06 http://alexpetrov.com/pub/vss06/

11

Posterior Thresholds

1

( ) (.75; , ) ( , | )

k k k k k

p T P p d d θ φ θ φ θ φ

−

= ∫ y

• 3
• 2.5
• 2
• 1.5
• 1
• 0.5

75% threshold Posterior density posterior normal

5/6/06 http://alexpetrov.com/pub/vss06/

12

Vaguely informative priors: Implemented on a grid: logα x β x λ Assume γ= .5 for 2AFC data MATLAB software available at

http://alexpetrov.com/softw/

Some Details

( ) N( , ) p

β β

β μ σ ∝ (log ) N( , ) p

α α

α μ σ ∝ ( ) Beta( , ) p a b

λ λ

λ ∝

5/6/06 http://alexpetrov.com/pub/vss06/

13

Simulation 1: Stationary

1.5 β = log 1.204 const α = − = .10 λ =

75

1.217 T = −

5/6/06 http://alexpetrov.com/pub/vss06/

14

Stimulus Placement

20 40 60 80 100

• 2
• 1.5
• 1
• 0.5

trial number log intensity

2 interleaved

staircases

100 trials/block

10 catch 40 x 3down/1up 50 x 2down/1up

100 runs of 12

blocks each

5/6/06 http://alexpetrov.com/pub/vss06/

15

• 3
• 2.5
• 2
• 1.5
• 1
• 0.5

Estimated threshold Frequency ML median mean true

Threshold Estimators

Estimator Mean Med Std ML

• 1.24
• 1.23

.27 .28 .31

• Std. dev.

0.41 0.36 .15 Median

• 1.26
• 1.23

Mean

• 1.30
• 1.27

1200 Monte Carlo estimates True 75% threshold = -1.217

5/6/06 http://alexpetrov.com/pub/vss06/

16

β x λ Distribution from Pass 1

• 4
• 3
• 2
• 1

0.5 0.6 0.7 0.8 0.9 1 log intensity P(correct) true β and λ ML β and λ Lapsing rate LAMBDA Slope BETA 0.05 0.1 0.15 0.2 1 2 3 4 5

5/6/06 http://alexpetrov.com/pub/vss06/

17

• 3
• 2.5
• 2
• 1.5
• 1
• 0.5

Estimated threshold Frequency ML median mean true

Catch Trials Are Worthwhile

Estimator Mean Med Std ML

• 1.24
• 1.22

.31 .30 .34 .16 Median

• 1.29
• 1.26

Mean

• 1.36
• 1.33
• Std. dev.

0.58 0.57 1200 Monte Carlo estimates No catch trials presented True 75% threshold = -1.217

5/6/06 http://alexpetrov.com/pub/vss06/

18

Simulation 2: With Learning

1.5 β =

/ 800

log 0.693 ( 2)

t

e α

−

= − − .10 λ =

1000 2000 3000 4000 5000 6000

• 2
• 1.5
• 1
• 0.5

T75

5/6/06 http://alexpetrov.com/pub/vss06/

19

Group Learning Curve, N= 100

10 20 30 40 50 60

• 2
• 1.8
• 1.6
• 1.4
• 1.2
• 1
• 0.8
• 0.6
• 0.4
• 0.2

Block number ML threshold True learning curve Reconstruction ± CI95

5/6/06 http://alexpetrov.com/pub/vss06/

20

More Realistic Sample, N= 10

10 20 30 40 50 60

• 2
• 1.8
• 1.6
• 1.4
• 1.2
• 1
• 0.8
• 0.6
• 0.4
• 0.2

Block number ML threshold True learning curve Reconstruction ± CI95

5/6/06 http://alexpetrov.com/pub/vss06/

21

Individual Runs

• 2
• 1
• 2
• 1
• 2
• 1
• 2
• 1
• 2
• 1
• 2
• 1

5/6/06 http://alexpetrov.com/pub/vss06/

22

• 1.5
• 1
• 0.5

0.5 1 Estimated threshold - true threshold Frequency ML median mean true

The Method Performs Well

Estimator Mean Med Std ML

• 0.03
• 0.02

.28 .29 .32

• Std. dev.

0.42 0.39 .15 Median

• 0.05
• 0.03

Mean

• 0.08
• 0.05

6000 Monte Carlo estimates Similar to the stationary case No systematic bias over time

5/6/06 http://alexpetrov.com/pub/vss06/

23

Example: Actual Data, N= 8

2 4 6 8 10 12 14 16

• 3
• 2.5
• 2
• 1.5
• 1
• 0.5

0.5 1 Block number 75% threshold

In high noise In no noise Jeter, Dosher, Petrov, & Lu (2005)

5/6/06 http://alexpetrov.com/pub/vss06/

24

Future Work

Sensitivity to priors? Compare with standard ML methods Individual differences Estimate slope in addition to threshold Non-stationary β and λ? Recommended stimulus placement? Hierarchical models

5/6/06 http://alexpetrov.com/pub/vss06/

25

The End