Assessing the Effectiveness of Youth Engagement Initiatives - PowerPoint PPT Presentation

Assessing the Effectiveness of Youth Engagement Initiatives Elections Canada 2013 BC Provincial General Election Conference University of British Columbia – Centre for the Study of Democratic Institutions January 31st – February 1st, 2014 VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 1

Why do evaluations? • Accountability • Learning – What’s the theory/logic of the project/program? – Was the project/program delivered as intended? – What went well? What could be improved? • Capacity-building – To do / to use evaluations (think evaluatively) • Knowledge-building – Contribute to broader learning VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 2

Logic Model Inputs Activities Outputs Immediate Intermediate Ultimate Outcomes Outcomes Outcomes Resources: Products # orders Learning Sustained - Staff developed Teaching interest - money # students Knowledge methods Promotions reached Participation Interest Voting Events # forms Voting distributed / Discussion completed Registering External Context VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 3

Student Vote Evaluation VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 4

Student Vote Federal Participation 2004-2011 *While targeted to Grades 5 and higher, all grades may participate, and students as young as kindergarten do. Teachers in middle school (intermediate) select the curriculum appropriate for their class. Since 2006, over 60% of registered schools have been elementary or middle schools. VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 5

Evaluation objectives • Assess the overall impact of the SVP in the context of the 41st federal general election • Provide findings that will enable Student Vote to improve the SVP for future parallel elections VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 6

Evaluation Design Students Teachers Parents Pre- Post- Pre- Post- Pre- Post- Treatment 14,754 3,780 1,341 341 Control 1,071 106 • Treatment: Students, teachers, parents who had participated in the 2011 SVP • Control: Students, teachers who had not participated in 2011 SVP, but had taken/taught a class related to civics during that school year VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 7

Teachers • Knowledge – Number of years teaching civics is significant (+) – Elementary teachers feel less informed than other teachers (secondary or middle) • Confidence – 85% participants say confidence increased – Only 65% of control group report feeling confident teaching civics • Intent to use SVP in future – 95% state they are very likely – High praise for materials and support received VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 8

Parents • Knowledge: – 61% of parents felt that their child’s participation in the SVP increased their own knowledge of politics and elections – 84% felt that it provided their family with more opportunities to learn about politics • Engagement/Discussion: – 65% of parents felt that it increased their own interest – 86% believe the SVP motivated their child to discuss politics • Intention to vote: – Of the 73% who reported voting, 20% said that their child’s participation in SVP positively influenced their decision to vote VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 9

Students • Knowledge and understanding – Number of correct responses (pre- vs. post) • Elementary: + 15 percentage points • Secondary: + 11 percentage points – Number of correct responses (participants vs. control) • Elementary: +11 percentage points • Secondary: no significant difference VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 10

Students • Engagement in political discussion – discussion with parents increased significantly as a result of SVP participation – especially in secondary students • Appreciation (It is “a civic duty to vote”) • Increase in secondary students • Political interest • Experiential learning had a positive effect VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 11

Intention to vote (frequency results) Question: If you had been eligible to vote in the federal general election that was held on May 2nd, would you have voted? Pre-Program 84% 16% Participant Post-Program 10% 90% Participant Non-Participant 82% 18% (Control) Yes No VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 12

Intention to vote • Multiple regression yielded mixed results • Methodological issues: – Confusing question wording (“eligible”) – Many responded “not old enough” to open-ended follow up question – “Real” versus “parallel” • Do we need better methods to evaluate impact on future voting behaviour? ? ? • Is it appropriate to try to measure this? ? VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 13

Mixed results • Confidence expressing views – Pre- vs. post: no significant difference – Confidence much higher among control group (over-confident?) ? – But: Parallel election had positive impact ? • Critical thinking – Frequencies (pre/post/control): no difference – Regression: mixed results • Methodological issues? VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 14

Methodological issues • Question specificity • Surveys too long and some questions too complex for younger students • Very strong control group • Quantitative surveys may not be best instrument for assessing all outcomes (need mixed methods approach) VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 15

Key recommendations • To strengthen the program: – Expand efforts to reach civics and social studies teachers – Engage official parties • To strengthen civics in general: – Experiential civics between official election periods • To strengthen the evaluation: – Review survey instruments – Consider research to determine impact on voting through a ‘longitudinal’ study VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 16

Apathy is Boring youth registration pilot project VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 17

Project Overview - 1 • Elections BC - Apathy is Boring: pilot project • Goal: to increase the number of youth accurately registered to vote in advance of the 2013 BC general election • Two Components: 1. Education for Civic and Community Organizations 2. Youth Voter Registration Campaign VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 18

Project Overview - 2 • Elections Canada – Apathy is Boring: evaluation/assessment • Goal: evaluate the effectiveness of key tactics using quasi-experimental methods and qualitative interviewing • Two components: 1. Group Registration/Turnout Comparison 2. Interviews with local organizations VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 19

Evaluation Project Components 1. Implement a quasi-experiment to test the effectiveness of registration drives conducted by Street Teams at concerts and events in March and April 2013 2. Conduct qualitative interviews with organizations seeking to engage or mobilize youth during 2013 BC election – to get a deeper sense of their experiences in youth electoral engagement VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 20

1. Registration Drives quasi-experiment • Treatment group: youth who register as a result of Apathy is Boring Street Teams activities • Control: youth who were not reached by this initiative • Analysis: registration and turnout data provided by Elections BC VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 21

Preliminary Results: Registration Drives • March 8 – April 19, 2013 • Key Outputs: – 51 events attended (target: 40) – 10,511 interactions (target: 14,000) – 390 “Register to Vote” packages given out – 511 “BC Has More Ways to Vote” packages given out – 572 completed 200A forms (target: 8000) VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 23

Preliminary Results: Registration Drives • Analysis of 200A registration transactions • 508 forms completed by individuals 18-39 Create Confirm Update Total 164 40 102 306 18-24 36 18 91 145 25-29 8 13 22 43 30-34 3 6 5 14 35-39 211 77 220 508 Total VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 24

Voter turnout by age and transaction type • 60% turnout among those who completed form Create Confirm Update Total 18-24 Y : 90 N : 74 Y : 28 N : 12 Y : 68 N : 34 Y : 186 N : 120 25-29 Y : 13 N : 23 Y : 13 N : 5 Y : 50 N : 41 Y : 76 N : 69 30-34 Y : 5 N : 3 Y : 9 N : 4 Y : 20 N : 2 Y : 34 N : 9 Y : 1 N : 2 Y : 6 N : 0 Y : 3 N : 2 Y : 10 N : 4 35-39 Total Y : 109 N : 102 Y : 56 N : 21 Y : 141 N : 79 Y : 306 N : 202 VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 25

Next steps • Obtain / analyse data for control group(s) • Analysis of qualitative interview data • Final report and presentation to EC (to be shared with E-BC) VOTE. SHAPE YOUR WORLD. VOTER, C’EST CHOISIR SON MONDE. 26