Bayesian D-optimal designs for rank-order conjoint choice - PowerPoint PPT Presentation

Bayesian D-optimal designs for rank-order conjoint choice experiments Bart Vermeulen Katholieke Universiteit Leuven, Faculty of Economics and Applied Economics Peter Goos University of Antwerp, Faculty of Applied Economics Martina Vandebroek Katholieke Universiteit Leuven Faculty of Economics and Applied Economics University Center of Statistics

Outline 1. Introduction ⋄ Rank-order conjoint choice experiments ⋄ Optimal Design for rank-order conjoint choice experiments 2. Rank-order multinomial logit model 3. Bayesian D-optimal designs for rank-order conjoint choice experiments 4. Performance of the Bayesian D-optimal designs 5. Improvement in the accuracy of the estimates and predictions

Introduction: rank-order conjoint choice experiment What is a rank-order conjoint choice experiment? ⋄ A rank-order conjoint choice experiment aims to estimate the values respondents attach to the features of a product... ⋄ ... by asking him to rank a number of alternatives of several choice sets instead of to choose the most preferred one as is done in a classical choice experiment Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 3

Introduction: rank-order conjoint choice experiment Example: Rank the following laptops in decreasing order of preference: Screen:15.4 inch Screen:17 inch Screen:15.4 inch Screen:17 inch HD:80 GB HD:60 GB HD:120 GB HD:80 GB Battery:5h Battery:6h Battery:4h Battery:6h Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 4

Introduction: Optimal design for rank-order conjoint choice experiments Goal: Performing a rank-order conjoint choice experiment in a statistically efficient way: with a small number of choice sets obtaining a maximum amount of information → Leads to precise estimates of the parameters with a minimum variance ⇓ Problem: A large number of possible candidate alternatives to include in a ranking experiment and how to group them into choice sets ⇓ Solution: Constructing D-optimal design for rank-order conjoint choice experiments Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 5

Introduction: Optimal design for rank-order conjoint choice experiments Questions ⋄ How to construct D-optimal designs for rank-order conjoint choice experiments? ⇒ Development of an expression for the information matrix for rank-order conjoint choice experiments ⋄ Do these designs outperform other benchmark designs? ⋄ What is the improvement in estimation and prediction accuracy if we include extra ranking steps in an experiment? Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 6

The rank-ordered multinomial logit model The rank-ordered multinomial logit model (MLM) = extension of the multinomial logit model (Begs et al., 1981) ֒ → Ranking alternatives = sequential and conditional choice task ⇒ If ranking alternative i, i’ and i”, the probability of assigning rank 1 to alternative i in this choice set is: ′ exp ( x ik β ) P ik 1 = (1) ′ � j ∈{ i,i ′ ,i ′′ } exp ( x jk β ) ⇒ Probability of assigning rank 2 to alternative i’ is then: ′ exp ( x i ′ k β ) P i ′ k 2 = (2) ′ � j ∈{ i ′ ,i ′′ } exp ( x jk β ) Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 7

The rank-ordered multinomial logit model ⇒ The joint probability of ranking alternative i first, alternative i’ second and alternative i” third is: ′ ′ exp ( x ik β ) exp ( x i ′ k β ) P ii ′ i ′′ k = (3) jk β ) · ′ ′ � j ∈{ i,i ′ ,i ′′ } exp ( x � j ∈{ i ′ ,i ′′ } exp ( x jk β ) ⇒ This leads to the following log-likelihood function for one choice set and for one respondent: ln ( L ) = Y ii ′ i ′′ k ln( P ii ′ i ′′ k ) + Y ii ′′ i ′ k ln( P ii ′′ i ′ k ) + Y i ′ ii ′′ k ln( P i ′ ii ′′ k ) (4) + Y i ′ i ′′ ik ln( P i ′ i ′′ ik ) + Y i ′′ ii ′ k ln( P i ′′ ii ′ k ) + Y i ′′ i ′ ik ln( P i ′′ i ′ ik ) Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 8

Bayesian D-optimal design for the rank-ordered MLM Aim: maximize information coming from the experiment �→ Bayesian D-optimality criterion: maximizes the expected determinant of the Fisher Information matrix over the prior distribution of the unknown parameters ℜ p ( det ( I ( X , β ))) − 1 /p · f ( β ) · d β � D b − error = However: Problem: expression for the information matrix of the rank-ordered multinomial logit model? Cases: ⋄ ranking three alternatives ⋄ ranking four alternatives Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 9

Bayesian D-optimal design for the rank-ordered MLM In case of ranking three alternatives (for one choice set and one respondent) 3 ′ ′ � P ik 1 X ′ ( i ) k ( P ( i ) k − p ( i ) k p ′ I rank ( β ) = X k ( P k − p k p k ) X k + ( i ) k ) X ( i ) k i =1 (5) 3 � P ik 1 X ′ ( i ) k ( P ( i ) k − p ( i ) k p ′ = I choice ( β ) + ( i ) k ) X ( i ) k i =1 So: 3 ′ � ( i ) k ( P ( i ) k − p ( i ) k p ′ I rank ( β ) − I choice ( β ) = P ik 1 X ( i ) k ) X ( i ) k (6) i ֒ → extra information can be quantified! Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 10

Bayesian D-optimal design for the rank-ordered MLM In case of ranking four alternatives (for one choice set and one respondent) I rank ( β ) = I choice ( β ) 4 � P ik 1 X ′ ( i ) k ( P ( i ) k − p ( i ) k p ′ + ( i ) k ) X ( i ) k i =1 4 4 ′ � � ( ii ′ ) k ( P ( ii ′ ) k − p ( ii ′ ) k p ′ + ( P ik 1 P i ′ k 2 + P i ′ k 1 P ik 2 ) X ( ii ′ ) k ) X ( ii ′ ) k i =1 i ′ = i (7) ֒ → extra information can be quantified! Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 11

Bayesian D-optimal design for the rank-ordered MLM ⋄ Bayesian D-optimality criterion ⋄ Prior distribution: p-variate normal distribution with mean [ − 0 . 5; 0; − 0 . 5; 0; − 0 . 5; 0; − 0 . 5; − 0 . 5] and variance 0 . 5 ∗ I p for three three-level and two two-level attributes ⋄ Adaptive algorithm (Kessels, Jones, Goos and Vandebroek, 2006) : – coordinate exchange algorithm – use systematic sample of 100 draws of the prior distribution which were generated by the Halton sequence ⋄ Design: 3 3 · 2 2 / 4 / 9 Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 12

Performance of the Bayesian D-optimal designs Simulation study ⋄ 75 parameter sets drawn from a p-variate normal distribution as the true parameters ⋄ 1000 simulations for each true parameter set ⋄ ranking 9 sets of 4 alternatives by 50 respondents ⋄ Three scenarios: 1. Ranking the three most preferred alternatives (full choice set) 2. Ranking the two most preferred alternatives 3. Choosing the most preferred alternative (conjoint choice experiment) Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 13

Performance of the Bayesian D-optimal designs Benchmark designs : ⋄ Bayesian D-optimal choice design ⋄ Sawtooth Software: 1. Complete enumeration: orthogonal designs with minimal attribute level overlap within a choice set 2. Random design 3. Balanced overlap: no strong level overlap between the attributes in a single task Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 14

Performance of the Bayesian D-optimal designs Evaluation criteria ⋄ Estimation accuracy � 1000 r =1 (ˆ ′ (ˆ β ( β ) = 1 EMSE ˆ β r − β ) β r − β ) R ⋄ Prediction accuracy � 1000 p (ˆ p (ˆ ′ (ˆ p ( β ) = 1 EMSE ˆ r =1 (ˆ β r ) − p ( β )) β r ) − p ( β )) R ֒ → The smaller EMSE ˆ β and EMSE ˆ p ( β ) , the more precise the parameter estimates and predictions respectively. Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 15

Performance of the Bayesian D-optimal designs Ranking the three most preferred alternatives Estimation accuracy Prediction accuracy Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 16

Performance of the Bayesian D-optimal designs Ranking the two most preferred alternatives Estimation accuracy Prediction accuracy Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 17

Performance of the Bayesian D-optimal designs Choosing the most preferred alternative Estimation accuracy Prediction accuracy Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 18

Improvement on the accuracy of the estimates and the predictions Case: 9 sets of 4 alternatives Improvement in EMSE ˆ EMSE ˆ p β after assigning after assigning after assigning after assigning second rank third rank second rank third rank 62.29% 30.45% 57.96% 30.73% D-opt. rank. 58.07% 23.09% 54.75% 23.70% D-opt. choice 63.07% 34.23% 58.41% 31.66% Balanced overlap 64.78% 31.35% 58.57% 27.79% Compl. enum. 61.69% 30.70% 66.72% 27.80% Random Bayesian D-optimal designs for rank-order conjoint choice experiments MODA8 / 19