Removing barriers: Financial transparency, peer effects and prior - - PowerPoint PPT Presentation

Removing barriers: Financial transparency, peer effects and prior attainment

Simon Burgess, Julia Carey at EFM, University of Bristol Catherine Dilnot, Lindsey Macmillan at UCL/IOE Raj Chande, Michael Sanders at Behavioural Insights Team Also Abbi Stoneman, Christine Spencer and Maggi Walton, Project

administrators

Kate Guest from Teach First ran the mentor training Amy Butterworth, Deputy Head of Admissions at UoB helped

with understanding data and provided admissions data

All the student mentors.

UoB WP Workshop, Oct. 2016 2

Team:

This talk

Issue ‘Model’ and Interventions Aims Operations Analysis:

Methodology Data Preliminary results

Timetable Conclusions

UoB WP Workshop, Oct. 2016 3

Issue

Despite huge rise in numbers attending university, the

rate of return is still high for most students.

How can we help more students from disadvantaged

backgrounds attend the best universities?

It may all be about bursaries … But recent research

suggests that other factors may matter as much.

UoB WP Workshop, Oct. 2016 4

‘Model’ and Interventions

We believe there are other important barriers to access:

Not understanding the true financial costs and benefits of

going to a Russell Group university

Not believing that a Russell Group uni is “for people like me” Not having the very top grades to get in

Our interventions address these We train UoB students to go into local schools to deliver:

‘Inspire’ talks Additional small-scale tutoring

UoB WP Workshop, Oct. 2016 5

Aims

The over-arching aim was to:

improve access to the University of Bristol … … and universities more broadly … … for disadvantaged children in local schools and colleges.

We will provide causal evidence on the efficacy of this

approach using an RCT methodology.

We have written a Handbook on managing this scheme.

UoB WP Workshop, Oct. 2016 6

Operations

Intervention from 2013/14 to 2015/16, finishing June 2016. Recruit local schools to take part

‘Local’ was broad including disadvantaged areas of Weston-super-Mare. Included regular state secondary schools and Further Education colleges

Recruit UoB students to act as mentors:

Fresher’s Fair and Volunteer’s Fair, and later through email, undertaken

each year

Work with Teach First to train the students for both roles

Teach First do this for free, undertaken each year.

Match mentors to schools, and organise the practicalities:

Funded the travel of the mentors but they were volunteers, so not paid.

UoB WP Workshop, Oct. 2016 7

Analysis

Methodology Data Preliminary Results

UoB WP Workshop, Oct. 2016 8

Methodology

RCT approach School-year groups are randomised into Control, ‘Inspire’,

‘Tutoring’.

Control = “business as usual”. ‘Inspire’ talks:

We treated whole school-year groups School-year group level analysis

Tutoring

Five pupils selected by school (not random) for tutoring Pupil-level analysis using matched pupils in Control schools

UoB WP Workshop, Oct. 2016 9

Data

‘Inspire’ talks: school-year-calendar year level data on

applications from UCAS:

UoB any RG any HE

Mentoring: pupil level data from NPD:

GCSE grades A level grades

UoB WP Workshop, Oct. 2016 10

Sample: 43 schools and colleges in the trial:

7 are 16+ colleges only (so only appear in KS5 data) 10 have no sixth form (so only appear in KS4 data) 26 schools are all through, so appear in both

UoB WP Workshop, Oct. 2016 11

Trial schools

UoB WP Workshop, Oct. 2016 12

State schools and colleges: In the Trial All in Bristol, Gloucestershire and Somerset In England KS4 KS5 KS4 KS5 KS4 KS5 Number of students 6,857 7,594 23,971 16,776 593,435 395,457 Number of schools/colleges 36 33 174 98 3,775 2,392 % female 48.1 53.7 48.6 52.5 48.8 53.2 FSM (% of non-missing) 12.7 7.0 10.3 5.7 14.9 10.7 SEN (% of non-missing) 13.9 8.3 14.3 8.2 17.0 10.4 SSEN (% of non-missing) 2.2 0.9 3.4 0.7 3.8 1.1 EAL (% of non-missing) 7.5 6.3 4.7 4.9 12.7 15.1 Ethnicity: % Asian (of non-missing) 3.6 2.9 2.0 2.3 8.2 10.3 Ethnicity: % Black (of non-missing) 4.0 2.6 1.9 1.8 4.8 5.7 Ethnicity: % Mixed (of non-missing) 4.2 3.9 3.1 3.3 3.8 4.0 Ethnicity: % White (of non-missing) 87.5 89.2 91.8 91.0 80.6 76.9 Ethnicity: % AOEG/Chinese/Unclear (of non-missing) 0.8 1.4 1.2 1.6 2.6 3.1 GCSE mean capped points score including equivalents 333 370 332 377 333 375 % with 5 A*-C – GCSE and GNVQ 53.5 79.4 56.8 82.2 55.1 78.3 % with A-levels as main qualification N/A 56.9 N/A 61.4 N/A 57.5 % with NVQ as main qualification N/A 41.3 N/A 36.7 N/A 39.9 % achieving 3 or more A*-E grades in A- levels/applied A-levels N/A 42.5 N/A 48.3 N/A 46.6 Mean points from A-levels (across all students) N/A 415 N/A 464 N/A 445 Mean number of A-level entries per student N/A 2.0 N/A 2.2 N/A 2.1 Mean points per A-level entry N/A 199 N/A 205 N/A 202

Trial schools – extract …

UoB WP Workshop, Oct. 2016 13

State schools and colleges: In the Trial All in Bristol, Glos & Somerset KS4 KS5 KS4 KS5 FSM (% of non-missing) 12.7 7.0 10.3 5.7 EAL (% of non-missing) 7.5 6.3 4.7 4.9 % with 5 A*-C – GCSE and GNVQ 53.5 79.4 56.8 82.2 % with A-levels as main qualification N/A 56.9 N/A 61.4 % with NVQ as main qualification N/A 41.3 N/A 36.7 % achieving 3 or more A*-E grades in A-levels/applied A-levels N/A 42.5 N/A 48.3

Preliminary results

Our research is ongoing and these are preliminary

findings.

Our determination to give the actual implementation

as long as possible to run, and the relative slowness of the process of accessing NPD data means that there is more to do.

UoB WP Workshop, Oct. 2016 14

Results 1: Tutoring

We provided tutoring to 108 students from year 9 to 11

• ver the three years. NPD matched 106.

Only 29 appear in KS4 data that we have so far. So – small numbers and very preliminary. Control group from same LAs as our schools, matched

using very local SES score, KS2 and gender.

Capped GCSE scores as a summary measure:

UoB WP Workshop, Oct. 2016 15

UoB WP Workshop, Oct. 2016 16

Treated GCSE score Controls CGSE score Difference Unmatched 373 302 71 ATT 373 342

31

So about an 8% increase in GCSE points PRELIMINARY

Results 2: ‘Inspire’ Talks

Outcome measure is UCAS “applications” to UoB, RG,

All HE.

Analysis uses the randomisation plus a difference-in-

difference approach

Include school fixed effects and application-year

effects.

UoB WP Workshop, Oct. 2016 17

All HE Applications

UoB WP Workshop, Oct. 2016 18 (1) HE applications (total) b/se treat 14.948 (65.521) application_cycle=2013 0.000 (.) application_cycle=2014 50.441 (48.704) application_cycle=2015

• 4.546

(51.549) application_cycle=2016 29.831 (52.507) Constant 801.487*** (34.819) N 95

(About 2% baseline; if each kid applies to 5 unis, that’s almost 3 more kids)

Russell Group Applications

UoB WP Workshop, Oct. 2016 19 (1) RG applications (total) b/se treat 0.528 (11.030) application_cycle=2013 0.000 (.) application_cycle=2014 21.462* (8.199) application_cycle=2015 4.872 (8.678) application_cycle=2016 21.933* (8.839) Constant 124.829*** (5.862) N 95

UoB Applications

UoB WP Workshop, Oct. 2016 20

(1)

UoB applications (total) b/se treat 1.440 (2.411) Application cycle=2013 0.679 (1.829) Application cycle=2014 1.804 (1.829) Application cycle=2015 0.000 (.) Application cycle=2016 3.363+ (1.772) Constant 22.439*** (1.358) N 94

UoB Apps: Longer run of data

UoB WP Workshop, Oct. 2016 21

UoB Apps: “before” and “after”

UoB WP Workshop, Oct. 2016 22

Commentary 1

Preliminary results No statistical significance, but that might change Positive results, very worthwhile but not huge:

‘Inspire’ Talks on

All HE applications ‘Local RG’ university = University of Bristol

Tutoring on

GCSE points

UoB WP Workshop, Oct. 2016 23

Commentary 2

Policy Lessons? To the extent that UoB WP own practices are somewhat

similar to ours, suggests that they may be effective too.

But two caveats:

Control condition was “business as usual” so that includes the

activities of the University’s own WP team. Our intervention raised applications above and beyond their activities, suggesting pupils don’t see them as the same.

No power to isolate the impact of the different components of

• ur intervention. So cannot know whether the most effective

components are or are not part of UoB existing practices.

UoB WP Workshop, Oct. 2016 24

Timetable

Sharpening results on Inspire Talks

Now through December 2016. Revising results with the 2016/17 Admissions data, Spring 2017.

Undertaking pupil level analysis on mentoring

Spring 2017, using the final release pupil level data for 2016 KS4,

available next April.

Dissemination

Late Spring 2017 We have good links with: the Russell Group, individual leading

universities, the Cabinet Office, the Department for Education, the Social Mobility Commission

UoB WP Workshop, Oct. 2016 25

Conclusion: Beneficiaries of the scheme

Main focus: local schools and pupils (hopefully) University of Bristol students:

the addition to their cvs the training, experience and ‘warm glow’ of helping out in

their adopted city.

invaluable soft skills improving their future employability.

University of Bristol:

enhanced student experience capability to attract talent from a more diverse pool reputational terms both in the city and nationally delivered by volunteer students, this fits very well into the

developing ethos of the ‘Bristol Futures’ programme.

UoB WP Workshop, Oct. 2016 26

Discussant

Rachael Tooth, Head of Strategy and Change, Office for Fair Access

Lunch