Incentives Research in HE Those that found incentives effective - PDF document

5/29/2014 Incentives Research in HE Those that found incentives effective Survey Incentives and Institutional Response Rates: Parsons and Maniere (2013) An Exploratory Analysis • Significant increase in response rate for the experimental group compared to the control (49.4% versus 37.6%, respectively). Laguilles et al (2011) Across four surveys covering different topics, use of lottery ‐ based Shimon Sarraf • incentives significantly boosted responses rates between 5% and James Cole, Ph.D. 9%. Indiana University Center for Postsecondary Research Those that found incentives were ineffective Porter and Whitcomb (2003) One exp group responded at a significantly higher rate than the • control group (16.9% versus 13.9%, respectively). No differences between other exp groups and control. Overall incentives had “minimal impact” Theory Background Social Exchange Theory (Dillman, 1978) • For years response rates have been in decline. Three factors • As a result, incentives are increasingly used to boost 1) Reward – what the respondent expect to gain from the survey? or hold steady response rates. 2) Cost – how much to obtain the reward? • Since 2010, use of incentive at NSSE institutions has 3) Trust – expectation that the reward will outweigh the cost increased from 35% (2010) to 54% (2014). Leverage ‐ Salience Theory (Groves et al., 2000) • However, the scant research on the efficacy is mixed A decision ‐ making theory that considers the “subjective with some studies indicating minimal, while others weight” of demonstrate that incentives are effective. 1) Leverage – importance 2) Salience – topic interest 3) Survey and invitation attributes 2 1

5/29/2014 Theory Research Questions Social Exchange Theory (Dillman, 1978) 1. Are different lottery incentive types Three factors associated with higher NSSE response 1) Reward – incentive type and value rates after controlling for institutional 2) Cost – effort, time characteristics? 3) Trust – promotion message; survey attributes 2. Does incentive type effectiveness vary by class level Leverage ‐ Salience Theory (Groves et al., 2000) (first ‐ year students versus seniors)? 3. Are certain types of incentives more strongly related to A decision ‐ making theory that considers the “subjective institutional response rates than others? Does the weight” of amount spent by institutions on incentives matter? 1) Leverage – incentive type and value 4. What is the relationship between campus promotional 2) Salience – negative or positive interest efforts and response rates, above and beyond incentives? Does the relationship vary by class level? How do schools 3) Survey and invitation attributes – sponsorship, design, etc promote NSSE (posters, social media) and who is involved? 7 Research Questions Research Questions 1. Do different types of lottery incentives correlate with 1. Are different lottery incentive types associated with higher NSSE response rates after controlling for various institutional higher NSSE response rates after controlling for characteristics? institutional characteristics? 2. Does the relationship between incentive 2. Does the relationship between incentive type and average institutional response rates vary by first ‐ year and type and average institutional response senior NSSE administrations? rates vary by first ‐ year and senior NSSE 3. Are certain types of incentives more strongly related to administrations? institutional response rates than others? Does the amount spent by institutions on incentives matter? 3. Are certain types of incentives more strongly related to institutional response rates than others? Does the amount 4. What is the relationship between campus promotional spent by institutions on incentives matter? efforts and response rates, above and beyond incentives? 4. What is the relationship between campus promotional Does the relationship vary by class level? How do schools efforts and response rates, above and beyond incentives? promote NSSE (posters, social media) and who is Does the relationship vary by class level? How do schools involved? promote NSSE (posters, social media) and who is involved? 6 8 2

5/29/2014 Research Questions Original Sample 1. Are different lottery incentive types associated with higher NSSE • National Survey of Student Engagement (NSSE) response rates after controlling for various institutional – 2013 administration (n=621) characteristics? 2. Does the relationship between incentive type and average institutional response rates vary by first ‐ year and senior NSSE administrations? 3. Are certain types of incentives more strongly related to institutional response rates than others? Does the amount spent by institutions on incentives matter? 4. What is the relationship between campus promotional efforts and response rates, above and beyond incentives? Does the relationship vary by class level? How do schools promote NSSE (posters, social media) and who is involved? 9 11 Research Questions Incentive Types Used (NSSE 2013) 1. Are different lottery incentive types associated with higher NSSE response rates after controlling for various institutional characteristics? 2. Does the relationship between incentive type and average institutional response rates vary by first ‐ year and senior NSSE administrations? 3. Are certain types of incentives more strongly related to institutional response rates than others? Does the amount spent by institutions on incentives matter? 4. What is the relationship between campus promotional efforts and response rates, above and beyond incentives? Does the relationship vary by class level? How do schools promote NSSE (posters, social media) and who is N = 621 involved? 10 12 3

5/29/2014 Incentive Type: Lottery or Guaranteed Analytical Sample Prize (NSSE 2013) • 531 out of 621 NSSE 2013 institutions • Institutions excluded 1) Used more than one incentive type 2) Used a guaranteed incentive 3) International institutions (Canadians retained) 4) Halted administrations 5) Influential outliers • NSSE incentive data set (229 institutions) • Quick Response Panel (230 institutions) 13 15 How many different types of Key Variables incentive combinations? (NSSE 2013) Dependent Variables Incentive Type (1/0) Institutional Response Rates Cash • First –Year Gift Card – General • Senior Gift Card – Specific Technology Other (school benefits, Promotional Effort (1/0) souvenirs, and other High random approaches) Medium reference : no incentive used Low (reference) Expenditures Dollars spent per NSSE sample member 14 4

5/29/2014 Measuring Promotional Effort with Methods NSSE Quick Response Panel • First ‐ year and senior OLS Regression models Apart from participation incentives, how would you describe your NSSE 2013 • Natural log of response rate used to address promotional activities? regression assumptions (constant residual • We did not have any special promotional variance) activities. Students only received our official – Coefficients transformed and interpreted as recruitment messages. [Low] percentages per log ‐ linear model • We promoted the survey using one or two simple methods (e.g., posters, advertisements) but did – Note: a 15% coefficient does NOT mean response not invest much otherwise. [Medium] rate is 15% points greater • We promoted the survey in many different ways • 51% of variation explained for first ‐ year students; and invested significant effort in getting the word 47% for seniors out. [High] 17 19 Other Variables School Characteristics Campus Proportion of … Results Full ‐ time students Female African ‐ American Latino Canadian institution (1/0) Public institution (1/0) Undergraduate enrollment (000s) 20 5

5/29/2014 Question #1 Question #2 Do different types of lottery incentives Does incentive type effectiveness vary correlate with higher NSSE response by class level (first ‐ year students rates after controlling for institutional versus seniors)? characteristics? 21 23 Response Rate % Increase by Incentive Type Response Rate % Increase by Incentive Type (reference group: no incentive used) (reference group: no incentive used) Percentage Increase in Response Rate (vs reference) Response Rate Percentage Increase (vs reference) 25% 25% 7 8 3 5 20% 20% 20% 20% 19% 19% 15% 15% 15% 15% 13% 13% 12% 12% 12% 12% 11% 11% First ‐ Year First ‐ Year 10% 10% Senior Senior 7% 7% 5% 5% + + *** ** ** * * * ** *** ** ** * * * ** 0% 0% Technology Gift Card ‐ Cash Gift Card ‐ Technology Gift Card ‐ Cash Gift Card ‐ General Specific General Specific Incentive Type 22 Incentive Type 24 +p<.10; *p<.05; **p<.01; ***p<.001 +p<.10; *p<.05; **p<.01; ***p<.001 6