A comparison of efficient designs for choices between two options - - PowerPoint PPT Presentation

A comparison of efficient designs for choices between two options

Heiko Großmann Queen Mary, University of London (joint work with H. Holling, U. Graßhoff and R. Schwabe)

mODa 8

Almagro, 7 June 2007

Outline

◮ Background ◮ Relationship between nonlinear choice and linear paired comparison

models

◮ Comparison of designs for main effects models ◮ Designs for choice experiments with main effects and interactions

1/25

Choice experiments

Where are they used?

◮ Marketing: New product development, consumer research ◮ Health economics, medicine: Preferences for health care

interventions

◮ Psychology, behavioral economics: Judgement and decision making,

attitude research

◮ Environmental sciences: Valuation of ecosystems, willingness-to-pay

for public goods

◮ ...

2/25

Preference measurement

3/25

How can preferences be measured?

Decision attributes

◮ Number of windows ◮ Type of floor ◮ Size ◮ Layout

4/25

How can preferences be measured?

5/25

Choice experiments

6/25

Choice experiments

7/25

Choice experiments

8/25

Modeling choices between two options

The multinomial logit (MNL) model

Choice probabilities P(Y (s, t) = 1) = eV (s) eV (s) + eV (t) where V (x) = f(x)⊤β for every option x

9/25

Modeling choices between two options

Information matrix for the MNL model

Normalized Fisher information matrix for exact design ξN with N pairs (sn, tn), n = 1, . . . , N M(ξN; β) = 1 N

N

• n=1

X⊤

n (Dn − pnp⊤ n )Xn

where

◮ Dn = diag(πn, 1 − πn), pn = (πn, 1 − πn)⊤ ◮ πn = P(Y (sn, tn) = 1) ◮ Xn =

f(sn)⊤ f(tn)⊤

• 10/25

Relation between MNL and linear model for pairs

A basis for comparing choice designs

◮ Most optimal designs in the MNL model for pairs have been derived

under the assumption that πn = 1/2, or equivalently that β = 0 (Burgess & Street, 2005; Street et al. 2001; Street & Burgess, 2004)

◮ In this case, for every exact design ξN

M(ξN; β) = 1 4M(ξN) where M(ξN) = 1

N X⊤X is the normalized information matrix in the

linear paired comparison model ˜ Y (s, t) = (f(s) − f(t))⊤ ˜ β + ε

◮ Implication: Optimal designs for the linear model are also optimal

for the MNL model and vice versa, if β = 0 is presumed

11/25

General setting

Notation

◮ K factors with levels in Xk = {1, . . . , vk}, k = 1, . . . , K ◮ Options: s, t ∈ X1 × . . . × XK ◮ Approximate design on set of pairs (s, t):

ξ = (s1, t1) · · · (sJ, tJ) w1 · · · wJ

• where

J

• j=1

wj = 1 with information matrix in linear paired comparison model given by M(ξ) =

J

• j=1

wj[f(sj) − f(tj)][f(sj) − f(tj)]⊤

12/25

Main effects model

Parametrization

◮ Standard parametrization (effects coding)

f = (f ⊤

1 , . . . , f ⊤ K)⊤, ˜

β = ( ˜ β⊤

1 , . . . , ˜

β⊤

K)⊤, ˜

βk = (˜ β(k)

1 , . . . , ˜

β(k)

vk−1)⊤

fk(xk) =        xkth unit vector of length vk − 1 xk ∈ {1, . . . , vk − 1} −1vk−1 xk = vk

◮ Number of parameters: p = K k=1 vk − K

13/25

Optimality results: main effects model

Theorem (Graßhoff et al., 2004)

For K factors with levels v1, . . . , vK an approximate design ξ∗ is D-optimal, if and only if M(ξ∗) =    M1 ... MK    where for k = 1, . . . , K the (vk − 1) × (vk − 1) matrix Mk is given by Mk = 2 vk − 1

          2 1 · · · · · · 1 1 ... ... . . . . . . ... ... ... . . . . . . ... ... 1 1 · · · · · · 1 2          

14/25

Optimal exact designs: main effects model

Available results

Same number of levels for all factors, that is vk = v, k = 1, . . . , K

◮ v = 2: Foldover construction based on regular fractions of resolution

III or higher (Street & Burgess, 2004) Pairs: 2K−m or 2K−m−1 where m is largest number for which fraction of required resolution exists

◮ v ≥ 2: Hadamard matrix construction (Graßhoff et al., 2004)

Pairs: HKv(v − 1)/2 where HK ≤ K + 3

15/25

Optimal exact designs: main effects model

Available results

Different numbers of levels allowed

◮ Method based on ‘difference vectors’ (Burgess & Street, 2005)

Pairs: Multiple of K

k=1 vk ◮ Construction using asymmetric orthogonal arrays of strength 2

(Graßhoff et al., 2004) Pairs: Size N of smallest OA(N; m1, . . . , mK; 2) where mk is a multiple of vk(vk − 1)/2 if vk is odd and a multiple of vk(vk − 1) if vk is even

16/25

Examples

Factors with different numbers of levels

◮ K = 7, v1 = . . . = v6 = 3, v7 = 4

Burgess & Street (2005): 36 × 4 × 3 = 8748 pairs Grasshoff et al. (2004): 18 pairs

◮ K1 ≤ 11 two-level and K2 ≤ 12 three-level factors, K1 + K2 = K

Burgess & Street (2005): 2K1 × 3K2 pairs Grasshoff et al. (2004): 36 pairs

17/25

Design comparison for main effects model

Remarks

◮ If all factors have v = 2 levels, the Hadamard matrix construction

and the foldover construction yield designs of the same size (for K ≤ 8) provided that whenever possible the m defining contrasts needed for the latter construction have even wordlength

◮ For factors with different numbers of levels, the method of Grasshoff

et al. (2004) yields smaller optimal designs than the construction in Burgess & Street (2005)

◮ The method of Burgess & Street (2005) is more widely applicable

but the reduction achieved is often not very large when compared to the product-type optimal design with 1/2 K

k=1 Nk pairs, where

Nk = vk(vk − 1) for vk even and Nk = vk(vk − 1)/2 for vk odd

18/25

Model including main effects and interactions

Parametrization

◮ Consider only the case vk = 2, k = 1, . . . , K ◮ Standard parametrization (effects coding)

f(x) = (g1(x1), . . . , gK(xK), g1(x1)g2(x2), . . . , gK−1(xK−1)gK(xK))⊤ where gk(1) = 1 and gk(2) = −1 for k = 1, . . . , K

◮ Number of parameters: p = K + K(K − 1)/2

19/25

Optimality results: main and interaction effects

Theorem (van Berkum, 1987; Street et al., 2001)

◮ If K is odd, then the approximate design ξ∗ which is uniform on the

set of all pairs differing in exactly d∗ = (K + 1)/2 factors is D-optimal with information matrix M(ξ∗) = 4d∗/KIp

◮ If K is even, then the approximate design ξ∗ which is uniform on the

set of all pairs differing in exactly d∗ = K/2 or d∗ + 1 factors is D-optimal with information matrix M(ξ∗) = 4(d∗ + 1)/(K + 1)Ip

◮ Generalization when common number of levels is larger than 2:

Graßhoff et al. (2003)

20/25

Performance of uniform designs on sets of pairs which differ in fixed number of factors

D-efficiencies

K 3 4 5 6 7 8 9 10 d∗ 2 2 3 3 4 4 5 5 effD(¯ ξd∗) 1.000 0.990 1.000 0.997 1.000 0.999 1.000 0.999 effD(¯ ξd∗−1) 0.707 0.632 0.874 0.816 0.931 0.891 0.956 0.928 effD(¯ ξd∗+1) 0.000 0.980 0.840 0.995 0.922 0.998 0.953 0.999 Note: For every d the uniform design on the set of pairs which differ in exactly d factors is denoted by ¯ ξd. The information matrix M(¯ ξd) is diagonal

21/25

Exact designs: main and interaction effects

Available results

◮ Method based on two-level fractional factorials of resolution V or

higher in K factors and ‘difference vectors’ (Street & Burgess, 2004) Pairs: Multiple of size of the fractional factorial

◮ Construction for pairs which differ in exactly d factors using BIBDs,

Hadamard matrices and two-level fractional factorials of resolution III or higher in K − d factors (Großmann et al., 2007) Pairs: bmn where n ≤ d + 3, m is the size of a fractional factorial in K − d factors and b the number of blocks of a BIBD(K, b, r, d, λ)

22/25

Design comparison: main and interaction effects

GSG07 SB04 K d n BIBD(K, b, r, d, λ) Fact. Pairs D-eff. Pairs D-eff. 3 2 2 BIBD(3, 3, 2, 2, 1) 21 12 1.000 12 1.000 4 2 2 BIBD(4, 6, 3, 2, 1) 22 48 0.990 48 0.990 5 3 4 BIBD(5, 10, 6, 3, 3) 22 160 1.000 160 1.000 6 3 4 BIBD(6, 10, 5, 3, 2) 23−1

III

160 0.997 224 1.000 7 4 4 BIBD(7, 7, 4, 4, 2) 23−1

III

112 1.000 224 1.000 8 4 4 BIBD(8, 14, 7, 4, 3) 24−1

IV

448 0.999 1056 0.999

23/25

Concluding remarks

Main points

◮ When the parameter vector in a multinomial logit model is presumed

to be equal to zero the optimal design problem for choices between two options is equivalent to the corresponding problem for linear paired comparisons

◮ Optimal and efficient linear paired comparison designs often require

a considerably smaller number of pairs than choice designs available in the literature

◮ Construction methods developed by Burgess and Street have a wider

range of applicability than corresponding methods for linear paired comparisons

◮ Designs considered here can be adapted to other contexts such as

two-colour microarray experiments

24/25

References

◮ Burgess, L. & Street, D.J. (2005). Optimal designs for choice experiments with

asymmetric attributes. Journal of Statistical Planning and Inference 134, 288–301.

◮ Graßhoff, U., Großmann, H., Holling, H. & Schwabe, R. (2003). Optimal paired

comparison designs for first-order interactions. Statistics 37, 373–386.

◮ Graßhoff, U., Großmann, H., Holling, H. & Schwabe, R. (2004). Optimal designs

for main effects in linear paired comparison models. Journal of Statistical Planning and Inference 126, 361–376.

◮ Großmann, H., Schwabe, R. & Gilmour S.G. (2007). Designs for first-order

interactions in choice experiments with binary attributes. Unpublished manuscript.

◮ Street, D.J. & Burgess, L. (2004). Optimal and near-optimal pairs for the

estimation of effects in 2-level choice experiments. Journal of Statistical Planning and Inference 118, 185–199.

◮ Street, D.J., Bunch, D.S. & Moore, B.S. (2001). Optimal designs for 2k paired

comparison experiments. Communications in Statistics—Theory and Methods 30, 2149–2171.

◮ van Berkum, E. E. M. (1987). Optimal Paired Comparison Designs for Factorial

• Experiments. CWI Tract 31. Amsterdam: Centrum voor Wiskunde en

Informatica.

25/25

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

H4 =     1 −1 −1 −1 1 −1 1 1 1 1 −1 1 1 1 1 −1    

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

H4 =     1 −1 −1 −1 1 −1 1 1 1 1 −1 1 1 1 1 −1     A =     −1 −1 −1 −1 1 1 1 −1 1 1 1 −1    

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

H4 =     1 −1 −1 −1 1 −1 1 1 1 1 −1 1 1 1 1 −1     A =     −1 −1 −1 −1 1 1 1 −1 1 1 1 −1     a =   (1, 2) (2, 3) (3, 1)  

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

H4 =     1 −1 −1 −1 1 −1 1 1 1 1 −1 1 1 1 1 −1     A =     −1 −1 −1 −1 1 1 1 −1 1 1 1 −1     a =   (1, 2) (2, 3) (3, 1)   a− =   (2, 1) (3, 2) (1, 3)  

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

a =   (1, 2) (2, 3) (3, 1)   a− =   (2, 1) (3, 2) (1, 3)   A =     − 1 − 1 − 1 − 1 1 1 1 − 1 1 1 1 − 1    

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

a =   (1, 2) (2, 3) (3, 1)   a− =   (2, 1) (3, 2) (1, 3)   A =     − 1 − 1 − 1 − 1 1 1 1 − 1 1 1 1 − 1                         −1 −1 −1 −1 1 1 1 −1 1 1 1 −1                    

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

a =   (1, 2) (2, 3) (3, 1)   a− =   (2, 1) (3, 2) (1, 3)   A =     − 1 − 1 − 1 − 1 1 1 1 − 1 1 1 1 − 1                         (2, 1) (3, 2) −1 −1 (1, 3) −1 1 1 1 −1 1 1 1 −1                    

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

a =   (1, 2) (2, 3) (3, 1)   a− =   (2, 1) (3, 2) (1, 3)   A =     − 1 − 1 − 1 − 1 1 1 1 − 1 1 1 1 − 1                         (2, 1) (3, 2) −1 −1 (1, 3) −1 1 1 (1, 2) (2, 3) −1 1 (3, 1) 1 1 −1                    

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

a =   (1, 2) (2, 3) (3, 1)   a− =   (2, 1) (3, 2) (1, 3)   A =     − 1 − 1 − 1 − 1 1 1 1 − 1 1 1 1 − 1     D =                     (2, 1) (2, 1) (2, 1) (3, 2) (3, 2) (3, 2) (1, 3) (1, 3) (1, 3) (2, 1) (1, 2) (1, 2) (3, 2) (2, 3) (2, 3) (1, 3) (3, 1) (3, 1) (1, 2) (2, 1) (1, 2) (2, 3) (3, 2) (2, 3) (3, 1) (1, 3) (3, 1) (1, 2) (1, 2) (2, 1) (2, 3) (2, 3) (3, 2) (3, 1) (3, 1) (1, 3)                    

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

a =   (1, 2) (2, 3) (3, 1)   a− =   (2, 1) (3, 2) (1, 3)   A =     − 1 − 1 − 1 − 1 1 1 1 − 1 1 1 1 − 1     D =                     (2, 1) (2, 1) (2, 1) (3, 2) (3, 2) (3, 2) (1, 3) (1, 3) (1, 3) (2, 1) (1, 2) (1, 2) (3, 2) (2, 3) (2, 3) (1, 3) (3, 1) (3, 1) (1, 2) (2, 1) (1, 2) (2, 3) (3, 2) (2, 3) (3, 1) (1, 3) (3, 1) (1, 2) (1, 2) (2, 1) (2, 3) (2, 3) (3, 2) (3, 1) (3, 1) (1, 3)                    

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

a =   (1, 2) (2, 3) (3, 1)   a− =   (2, 1) (3, 2) (1, 3)   A =     − 1 − 1 − 1 − 1 1 1 1 − 1 1 1 1 − 1     D =                     (2, 1) (2, 1) (2, 1) (3, 2) (3, 2) (3, 2) (1, 3) (1, 3) (1, 3) (2, 1) (1, 2) (1, 2) (3, 2) (2, 3) (2, 3) (1, 3) (3, 1) (3, 1) (1, 2) (2, 1) (1, 2) (2, 3) (3, 2) (2, 3) (3, 1) (1, 3) (3, 1) (1, 2) (1, 2) (2, 1) (2, 3) (2, 3) (3, 2) (3, 1) (3, 1) (1, 3)                    

⇒

                    (2, 2, 2) (3, 3, 3) (1, 1, 1) (2, 1, 1) (3, 2, 2) (1, 3, 3) (1, 2, 1) (2, 3, 2) (3, 1, 3) (1, 1, 2) (2, 2, 3) (3, 3, 1)                    

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

a =   (1, 2) (2, 3) (3, 1)   a− =   (2, 1) (3, 2) (1, 3)   A =     − 1 − 1 − 1 − 1 1 1 1 − 1 1 1 1 − 1     D =                     (2, 1) (2, 1) (2, 1) (3, 2) (3, 2) (3, 2) (1, 3) (1, 3) (1, 3) (2, 1) (1, 2) (1, 2) (3, 2) (2, 3) (2, 3) (1, 3) (3, 1) (3, 1) (1, 2) (2, 1) (1, 2) (2, 3) (3, 2) (2, 3) (3, 1) (1, 3) (3, 1) (1, 2) (1, 2) (2, 1) (2, 3) (2, 3) (3, 2) (3, 1) (3, 1) (1, 3)                    

⇒

                    (1, 1, 1) (2, 2, 2) (3, 3, 3) (1, 2, 2) (2, 3, 3) (3, 1, 1) (2, 1, 2) (3, 2, 3) (1, 3, 1) (2, 2, 1) (3, 3, 2) (1, 1, 3)                    

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

a =   (1, 2) (2, 3) (3, 1)   a− =   (2, 1) (3, 2) (1, 3)   A =     − 1 − 1 − 1 − 1 1 1 1 − 1 1 1 1 − 1     D =                     (2, 1) (2, 1) (2, 1) (3, 2) (3, 2) (3, 2) (1, 3) (1, 3) (1, 3) (2, 1) (1, 2) (1, 2) (3, 2) (2, 3) (2, 3) (1, 3) (3, 1) (3, 1) (1, 2) (2, 1) (1, 2) (2, 3) (3, 2) (2, 3) (3, 1) (1, 3) (3, 1) (1, 2) (1, 2) (2, 1) (2, 3) (2, 3) (3, 2) (3, 1) (3, 1) (1, 3)                    

⇒

                    ((2, 2, 2), (1, 1, 1)) ((3, 3, 3), (2, 2, 2)) ((1, 1, 1), (3, 3, 3)) ((2, 1, 1), (1, 2, 2)) ((3, 2, 2), (2, 3, 3)) ((1, 3, 3), (3, 1, 1)) ((1, 2, 1), (2, 1, 2)) ((2, 3, 2), (3, 2, 3)) ((3, 1, 3), (1, 3, 1)) ((1, 1, 2), (2, 2, 1)) ((2, 2, 3), (3, 3, 2)) ((3, 3, 1), (1, 1, 3))                    

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

Option 1 Option 2 Option 1 Option 2 Pair A B C A B C Pair A B C A B C 1 2 2 2 1 1 1 7 1 2 1 2 1 2 2 3 3 3 2 2 2 8 2 3 2 3 2 3 3 1 1 1 3 3 3 9 3 1 3 1 3 1 4 2 1 1 1 2 2 10 1 1 2 2 2 1 5 3 2 2 2 3 3 11 2 2 3 3 3 2 6 1 3 3 3 1 1 12 3 3 1 1 1 3

Hadamard matrix construction

Example: K = 3 factors with v = 3 levels

M(ξ∗) =         2 1 1 2 2 1 1 2 2 1 1 2        

Hadamard matrix construction

Algorithmic description

Optimal paired comparison design for K factors, each at v levels

• 1. Select Hadamard matrix Hu of order u ≥ K
• 2. Choose u × K submatrix A of Hu
• 3. Let a be the vector of length v(v − 1)/2 whose components are the
• rdered pairs (i, j) corresponding to the subsets of size two of

{1, . . . , v} listed in some order

• 4. Let a− be the vector with each pair (i, j) in a replaced by (j, i)
• 5. Replace every 1 in A with a and every −1 with a− to obtain D
• 6. The (n, k)-element of D is an ordered pair (xn,k, yn,k)
• 7. For the n-th choice set use the pair (sn, tn) with options

sn = (xn,1, . . . , xn,K) and tn = (yn,1, . . . , yn,K)