S u r v e y s in marketing research SU R VE Y AN D ME ASU R E ME N - - PowerPoint PPT Presentation

Surveys in marketing research

SU R VE Y AN D ME ASU R E ME N T D E VE L OP ME N T IN R

George Mount

Data analytics educator

SURVEY AND MEASUREMENT DEVELOPMENT IN R

"On a scale of 1 to 5..."

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Anatomy of a survey

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Steps to survey development

Step 1: Item generation Do the experts agree we have strong items?

Hinkin, Timothy R. "A brief tutorial on the development of measures for use in survey questionnaires." _Organizational research methods_ 1.1 (1998).

1

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Inter-rater reliability: base agreement

Base agreement

library(irr) agree(experts) Percentage agreement (Tolerance=0) Subjects = 10 Raters = 2 %-agree = 50

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Inter-rater reliability: Cohen's kappa

Account for chance: Cohen's kappa

library(psych) cohen.kappa(experts) Cohen Kappa and Weighted Kappa correlation coefficients and confidence boundaries lower estimate upper unweighted kappa -0.202 0.24 0.69 weighted kappa -0.078 0.42 0.92 Number of subjects = 10

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Interpreting Cohen's kappa

Weighted kappa interpretation 0 - .39 Poor agreement .4 - .59 Moderate agreement .6 - .79 Substantial .8 - 1 Very strong

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Content validity

How essential is each item to measuring the concept of interest? Lawshe's Content Validity Ratio (CVR): What percent of experts judge that item "essential" to what's being measured?

CV R = N = total number of experts E = number who rated the item as essential N/2 E − (N/2)

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Measuring CVR in R

Calculating CVR in R :

library(psychometric) # Expert 1: Essential # Expert 2: Essential # Expert 3: Not necessary # Expert 4: Useful # Expert 5: Essential CVratio(NTOTAL = 5, NESSENTIAL = 3) [1] 0.2

SURVEY AND MEASUREMENT DEVELOPMENT IN R

CVR interpretation

CVR value Interpretation

• 1

Perfect consensus against No consensus 1 Perfect consensus for # of experts CVR threshold 5 .99 15 .49 40 .29

Expert assessment: let's practice!

SU R VE Y AN D ME ASU R E ME N T D E VE L OP ME N T IN R

Measurement, validity & reliability

SU R VE Y AN D ME ASU R E ME N T D E VE L OP ME N T IN R

George Mount

Data analytics educator

SURVEY AND MEASUREMENT DEVELOPMENT IN R

What is measurement?

A process of observing and recording Done with an instrument Is our instrument "calibrated?"

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Measurement reliability

Are we measuring consistently? Can we reproduce our measurements?

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Measurement validity

Are we measuring what we claim/intend to measure?

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Step 2.5: exploratory data analysis

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Step 2.5: exploratory data analysis

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Survey EDA: response frequencies

Checking for: Response frequencies

# c_sat is a customer satisfaction survey # Convert items to factors library(dplyr) library(likert) c_sat_likert <- c_sat %>% mutate_if(is.integer, as.factor) %>% likert() # Plot response frequencies plot(c_sat_likert)

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Survey EDA: reverse coding

Checking for: Reverse-coded items

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Survey EDA: reverse coding

library(car) # Recode item 6:"difficult to use" # Not dplyr::recode()! # A "3" response will stay a "3"! c_sat$difficult_to_use_r <- car::recode(c_sat$difficult_to_use, "1 = 5; 2 = 4; 4 = 2; 5 = 1")

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Survey EDA: response frequencies/reverse coding

library(psych) library(dplyr) # Confirm reverse coding c_sat %>% select(difficult_to_use, difficult_to_use_r) %>% response.frequencies() %>% round(2) 1 2 3 4 5 miss difficult_to_use 0.02 0.18 0.57 0.21 0.03 0 difficult_to_use_r 0.03 0.21 0.57 0.18 0.02 0

Survey EDA: let's practice!

SU R VE Y AN D ME ASU R E ME N T D E VE L OP ME N T IN R

Describing survey results

SU R VE Y AN D ME ASU R E ME N T D E VE L OP ME N T IN R

George Mount

Data analytics educator

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Missingness

If < 5% non-response, omit missing cases. If > 5%, back to item generation.

# Total number of incomplete cases nrow(na.omit(bad_survey)) # Number of incomplete cases by item colSums(is.na(bad_survey)) item_1 item_2 item_3 item_4 item_5 82 98 78 98 82 75

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Missingness

library(Hmisc) # Hierarchical plot -- what values # are missing together? plot(naclus(bad_survey))

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Item correlations

Do groups of items correlate highly with each other... ...and not so high with those outside the group?

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Item correlations

library(psych) corr.test(brand_qual[,1:3]) Call:corr.test(x = brand_qual[, 1:3]) Correlation matrix trendy latest tired_r trendy 1.00 0.76 0.55 latest 0.76 1.00 0.57 tired_r 0.55 0.57 1.00 Sample Size [1] 505 Probability values (Entries above the diagonal are adjusted for multiple tests.) trendy latest tired_r trendy 0 0 0 latest 0 0 0 tired_r 0 0 0 To see confidence intervals of the correlations, print with the short=FALSE option

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Item correlations

# Visualize item correlations library(corrplot) corrplot(cor(brand_qual), method = "circle")

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Spotting "groups" in our correlation

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Spotting "groups" in our correlation

SURVEY AND MEASUREMENT DEVELOPMENT IN R

Spotting "groups" in our correlation

Let's practice!

SU R VE Y AN D ME ASU R E ME N T D E VE L OP ME N T IN R