SLIDE 1 Bayesian D-optimal designs for rank-order conjoint choice experiments
Bart Vermeulen
Katholieke Universiteit Leuven, Faculty of Economics and Applied Economics
Peter Goos
University of Antwerp, Faculty of Applied Economics
Martina Vandebroek
Katholieke Universiteit Leuven Faculty of Economics and Applied Economics University Center of Statistics
SLIDE 2 Outline
⋄ Rank-order conjoint choice experiments ⋄ Optimal Design for rank-order conjoint choice experiments
- 2. Rank-order multinomial logit model
- 3. Bayesian D-optimal designs for rank-order conjoint choice
experiments
- 4. Performance of the Bayesian D-optimal designs
- 5. Improvement in the accuracy of the estimates and predictions
SLIDE 3
Introduction: rank-order conjoint choice experiment What is a rank-order conjoint choice experiment?
⋄ A rank-order conjoint choice experiment aims to estimate the
values respondents attach to the features of a product...
⋄ ... by asking him to rank a number of alternatives of several
choice sets instead of to choose the most preferred one as is done in a classical choice experiment
Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 3
SLIDE 4
Introduction: rank-order conjoint choice experiment Example: Rank the following laptops in decreasing order of preference: Screen:15.4 inch Screen:17 inch Screen:15.4 inch Screen:17 inch HD:80 GB HD:60 GB HD:120 GB HD:80 GB Battery:5h Battery:6h Battery:4h Battery:6h
Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 4
SLIDE 5 Introduction: Optimal design for rank-order conjoint choice experiments Goal: Performing a rank-order conjoint choice experiment in a statistically efficient way: with a small number of choice sets
- btaining a maximum amount of information
→Leads to precise estimates of the parameters with a minimum
variance
⇓
Problem: A large number of possible candidate alternatives to include in a ranking experiment and how to group them into choice sets
⇓
Solution: Constructing D-optimal design for rank-order conjoint choice experiments
Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 5
SLIDE 6
Introduction: Optimal design for rank-order conjoint choice experiments
Questions
⋄ How to construct D-optimal designs for rank-order conjoint
choice experiments?
⇒ Development of an expression for the information matrix for
rank-order conjoint choice experiments
⋄ Do these designs outperform other benchmark designs? ⋄ What is the improvement in estimation and prediction accuracy
if we include extra ranking steps in an experiment?
Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 6
SLIDE 7 The rank-ordered multinomial logit model The rank-ordered multinomial logit model (MLM) = extension of the multinomial logit model (Begs et al., 1981)
֒ → Ranking alternatives = sequential and conditional choice task ⇒ If ranking alternative i, i’ and i”, the probability of assigning rank 1
to alternative i in this choice set is:
Pik1 = exp(x
′
ikβ)
′
jkβ)
(1)
⇒ Probability of assigning rank 2 to alternative i’ is then: Pi′k2 = exp(x
′
i′kβ)
′
jkβ)
(2)
Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 7
SLIDE 8 The rank-ordered multinomial logit model
⇒ The joint probability of ranking alternative i first, alternative i’
second and alternative i” third is:
Pii′i′′k = exp(x
′
ikβ)
′
jkβ) ·
exp(x
′
i′kβ)
′
jkβ)
(3)
⇒ This leads to the following log-likelihood function for one choice
set and for one respondent:
ln(L) = Yii′i′′k ln(Pii′i′′k) + Yii′′i′k ln(Pii′′i′k) + Yi′ii′′k ln(Pi′ii′′k) + Yi′i′′ik ln(Pi′i′′ik) + Yi′′ii′k ln(Pi′′ii′k) + Yi′′i′ik ln(Pi′′i′ik)
(4)
Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 8
SLIDE 9 Bayesian D-optimal design for the rank-ordered MLM Aim: maximize information coming from the experiment
→ Bayesian D-optimality criterion: maximizes the expected
determinant of the Fisher Information matrix over the prior distribution of the unknown parameters
Db − error =
- ℜp (det (I(X, β)))−1/p · f(β) · dβ
However: Problem: expression for the information matrix of the rank-ordered multinomial logit model? Cases:
⋄ ranking three alternatives ⋄ ranking four alternatives Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 9
SLIDE 10 Bayesian D-optimal design for the rank-ordered MLM In case of ranking three alternatives
(for one choice set and one respondent)
Irank(β) = X
′
k(Pk − pkp
′
k)Xk + 3
Pik1X′
(i)k(P(i)k − p(i)kp′ (i)k)X(i)k
= Ichoice(β) +
3
Pik1X′
(i)k(P(i)k − p(i)kp′ (i)k)X(i)k
(5) So:
Irank(β) − Ichoice(β) =
3
Pik1X
′
(i)k(P(i)k − p(i)kp′ (i)k)X(i)k
(6)
֒ → extra information can be quantified!
Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 10
SLIDE 11 Bayesian D-optimal design for the rank-ordered MLM In case of ranking four alternatives
(for one choice set and one respondent)
Irank(β) = Ichoice(β) +
4
Pik1X′
(i)k(P(i)k − p(i)kp′ (i)k)X(i)k
+
4
4
(Pik1Pi′k2 + Pi′k1Pik2)X
′
(ii′)k(P(ii′)k − p(ii′)kp′ (ii′)k)X(ii′)k
(7)
֒ → extra information can be quantified!
Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 11
SLIDE 12
Bayesian D-optimal design for the rank-ordered MLM
⋄ Bayesian D-optimality criterion ⋄ Prior distribution: p-variate normal distribution with mean
[−0.5; 0; −0.5; 0; −0.5; 0; −0.5; −0.5] and variance 0.5 ∗ Ip for
three three-level and two two-level attributes
⋄ Adaptive algorithm (Kessels, Jones, Goos and Vandebroek, 2006):
– coordinate exchange algorithm – use systematic sample of 100 draws of the prior distribution which were generated by the Halton sequence
⋄ Design: 33 · 22/4/9 Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 12
SLIDE 13 Performance of the Bayesian D-optimal designs Simulation study
⋄ 75 parameter sets drawn from a p-variate normal distribution as
the true parameters
⋄ 1000 simulations for each true parameter set ⋄ ranking 9 sets of 4 alternatives by 50 respondents ⋄ Three scenarios:
- 1. Ranking the three most preferred alternatives (full choice set)
- 2. Ranking the two most preferred alternatives
- 3. Choosing the most preferred alternative (conjoint choice
experiment)
Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 13
SLIDE 14 Performance of the Bayesian D-optimal designs Benchmark designs:
⋄ Bayesian D-optimal choice design ⋄ Sawtooth Software:
- 1. Complete enumeration: orthogonal designs with minimal
attribute level overlap within a choice set
- 2. Random design
- 3. Balanced overlap: no strong level overlap between the
attributes in a single task
Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 14
SLIDE 15 Performance of the Bayesian D-optimal designs Evaluation criteria
⋄ Estimation accuracy
EMSE ˆ
β(β) = 1 R
1000
r=1 (ˆ
βr − β)
′(ˆ
βr − β)
⋄ Prediction accuracy
EMSEˆ
p(β) = 1 R
1000
r=1 (ˆ
p(ˆ βr) − p(β))
′(ˆ
p(ˆ βr) − p(β)) ֒ → The smaller EMSE ˆ
β and EMSEˆ p(β), the more precise the
parameter estimates and predictions respectively.
Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 15
SLIDE 16
Performance of the Bayesian D-optimal designs Ranking the three most preferred alternatives
Estimation accuracy Prediction accuracy Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 16
SLIDE 17
Performance of the Bayesian D-optimal designs Ranking the two most preferred alternatives
Estimation accuracy Prediction accuracy Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 17
SLIDE 18
Performance of the Bayesian D-optimal designs Choosing the most preferred alternative
Estimation accuracy Prediction accuracy Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 18
SLIDE 19 Improvement on the accuracy of the estimates and the predictions Case: 9 sets of 4 alternatives
Improvement in EMSE ˆ
β
EMSEˆ
p
after assigning after assigning after assigning after assigning second rank third rank second rank third rank D-opt. rank.
62.29% 30.45% 57.96% 30.73%
D-opt. choice
58.07% 23.09% 54.75% 23.70%
Balanced overlap
63.07% 34.23% 58.41% 31.66%
64.78% 31.35% 58.57% 27.79%
Random
61.69% 30.70% 66.72% 27.80%
Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 19
SLIDE 20 Improvement on the accuracy of the estimates and the predictions Case: 12 sets of 3 alternatives Improvement in
EMSE ˆ
β
EMSEˆ
p
after assigning after assigning second rank second rank D-opt. rank.
54.91% 52.45%
D-opt. choice
50.12% 48.02%
Balanced overlap
53.59% 49.26%
54.47% 52.71%
Random
49.06% 46.37%
Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 20
SLIDE 21 Conclusions
⋄ Bayesian D-optimal designs for rank-order conjoint choice
experiments
⋄ Development of the information matrix for the rank-ordered
multinomial logit model
→ More information than a classical conjoint choice experiment
⋄ If the respondent does not rank all alternatives:
֒ → Bayesian D-optimal choice design gives slightly more accurate
parameter estimates and predictions If the respondent ranks all alternatives:
֒ → Bayesian D-optimal ranking design gives the most accurate
parameter estimates and predictions
⋄ Second ranking step gives an improvement in estimation and
prediction accuracy of about 50 - 60%; a third ranking step still
- ffers considerable improvements of about 30%
Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 21
