Rethinking the value of advanced maths participation: Progress after - - PowerPoint PPT Presentation

SLIDE 1

Rethinking the value of advanced maths participation: Progress after 14 months

Andy Noyes & Mike Adkins, University of Nottingham

 http://www.revamp-nottingham.org  andrew.noyes@nottingham.ac.uk

SLIDE 2

Outline

Project rationale Introduction to the work packages Working with national datasets Major findings

1

Economic value: Wage premiums from A level mathematics at age 34

2

Valued by: Predicting completion of A level mathematics

3

Academic value: The role of A level mathematics in Biology and Chemistry degree outcomes

In progress

1

Political value: tracking the policy discourse surrounding the 10% premium

2

Perceived value: End user attitudes to post-16 mathematics

Future research avenues

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 2 / 41

SLIDE 3

Project Rationale

The level of participation in advanced mathematics courses has been raising concerns for several years. Recent international comparisons show England to have one of the lowest levels of post-16 mathematics

• engagement. This, together with sustained pressure from stakeholders, has

led to the Secretary of State’s call for most young people to be studying mathematics up to 18 by the end of the decade. REVAMP weaves together four strands of quantitative analysis and one qualitative policy analysis strand to understand the current and changing attitudes to, participation in, and value of A level mathematics.

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 3 / 41

SLIDE 4

REVAMP Work Packages

Research questions

WP1: Is there still a ‘return’ to A level mathematics? Do Dolton and Vignoles’ findings hold in more recent datasets? WP2: Who is doing A level Mathematics now? How have participation patterns changed; by social category, by school type, etc? WP3: What is the relationship between A level participation and attainment and degree outcomes? WP4: How have mathematics education reports/policy/etc, taken up the economic and other value discourses since Curriculum 2000? WP5: What do 17 year olds think is the value of post-16 advanced mathematical study and how does it relate to their current and future choices and aspirations?

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 4 / 41

SLIDE 5

Working with national datasets

Datasets: NCDS - 1958 National Child Development Study BCS - 1970 British Cohort Study NPD – National Pupil Database HESA – Higher Education Statistics Agency Considerations: Negotiating access, security and risk management Size of datasets & computing power (HPC) Familiarisation takes time Cleaning takes more time Recoding variables Cumulative analytical choices

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 5 / 41

SLIDE 6

Work Package 1: Economic Value

Background: Economic Return to Maths Discourse

Liz Truss(18th September 2013), Parliamentary Under Secretary of State for Education and Childcare, argued that ”Maths, for example, is the only school subject which has been proven to add to

earnings, by up to 10% at A level,

even when every other factor is taken into account. Pupils who are ahead of their peers in maths at age 10

tend to be earning 7% more at the age of 30. Those working in science or technological careers are

paid, on average, 19% more than other professions...” (CBI talk on improving education and curriculum reform)

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 6 / 41

SLIDE 7

Work Package 1: Economic Value

Economic Return to Maths Discourse

Nick Gibb (10th September 2014) and Nicky Morgan (10th November 2014) have continued this argument: ”Those who do Maths A level will go on to earn 10%

more” (IET Skills event)

”And yet maths, as we all know, is the subject that employers value most, helping young people develop skills which are vital to almost any career. And you don’t just have to take my word for it - studies

show that pupils who study maths to A level will earn 10% more over their lifetime” (Your Life Campaign Launch).

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 7 / 41

SLIDE 8

Work Package 1: Economic Value

So where does this finding come from?

Peter Dolton and Anna Vignoles undertook research on the economic return of A level subjects in the mid to late 1990s. Their study had the following features: Data was from the National Child Development Survey Wave 5 (1991), with A level data from the 1981 sweep, and ability scores from the 1974 sweep. This was further supplemented with data from the 1980 National Graduate and Diplomates survey carried out in 1986. The final sample size consisted of 462 males who were aged 33 in

• 1991. Too many women had dropped out of work by age 33 with this
• sample. For the 1980 graduates the size for males was 2523 and for

women was 1515 respondents. Modelling was via an OLS log-linear model of annual gross earnings with missing cases being initially handled via mean imputation and later missing case dummies only.

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 8 / 41

SLIDE 9

Work Package 1: Economic value

Dolton and Vignoles (2002) findings[1]

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 9 / 41

SLIDE 10

Work Package 1: Economic value

Dolton and Vignoles (2002) findings[2]

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 10 / 41

SLIDE 11

Work Package 1: Economic Value

Approach

Research question: Is there still a ‘return’ to A level mathematics? Do Dolton and Vignoles’ findings hold in more recent datasets? Data: We used the British Cohort Study wave 7 (age 34) along with education data from wave 6 (age 30) and ability scores from wave 3 (age 10) to estimate average earnings conditional on a range of demographic, education, work experience and ability score predictors. Sample: 1457 male and female respondents born in 1970 and age 34 in 2004. Selection criteria is that each individual must have completed at least one A level and be in full or part-time work at the time of the wave 7 survey. Multiple Imputation: Given the vast improvements in computing power since the original study, we chose improve the handling of missing data through the technique of multiple imputation chained equations.

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 11 / 41

SLIDE 12

Work Package 1: Economic Value

Repeat study econometric model

(log)yi = α + βFemalei

1

+ βMarriedi

2

+ βChildreni

3

+ βManagerial−Technicali

4

+ βSkilled_Non−Manuali

5

+ βSkilled−Manuali

6

+ βPart−Skilledi

7

+ βUnskilledi

8

+ βOthersi

9

+ βEast_Midsi

10

+ βEast_Englandi

11

+ βNorth_Easti

12

+ βNorth_Westi

13

+ βSouth_Easti

14

+ βSouth_Westi

15

+ βWest_Midsi

16

+ βYorkshirei

17

+ βScotlandi

18

+ βWalesi

19

+ βDegreei

20

+ βNVQi

21

+ βProfi

22

+ βHE_Diplomai

23

+ βMaths&Computingi

24

+ βSciencei

25

+ βHumanitiesi

26

+ βSocial_sciencei

27

+ βOtheri

28

+ βPart_timei

29

+ βWork_Expi

30

+ βWork_Exp2

i

31

+ βTenurei

32

+ βUnemploymenti

33

+ βAge10_Mathsi

34

+ βAge10_Readingi

35

+ εi εi ∼ N(0, σ2)

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 12 / 41

SLIDE 13

Work Package 1: Economic value

Major findings[1]

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 13 / 41

SLIDE 14

Work Package 1: Economic value

Major findings[2]

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 14 / 41

SLIDE 15

Work Package 1: Economic value

Major findings[3]

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 15 / 41

SLIDE 16

Work Package 1: Economic value

Conclusions[1] The results do suggest that while the economy is vastly different from that experienced by the National Child Development Survey and British Graduate and Diplomates survey, those with an A level in mathematics and computing do appear to earn on average approximately 11% more than those without, which appears to be unique when compared to other subjects. However, there are a whole set of caveats to go with this:

1

This is an an average with a very wide confidence interval stretching from approximately 4-22%

2

Omitted variable bias - while we have tried to control for ability, there are

• ther non-random processes at work - such as subject choice being

non-random etc

3

While we have followed the original strategy, combining all science subjects together may mask the effect of individual subjects.

4

The A level mathematics and computing effect size is reduced substantially and becomes statistically insignificant when interacted with female.

5

While this is substantially more up-to-date than the estimates from the 1991 survey, the analysis used data which is 11 years old and there is a question to address whether the findings still hold especially with 5-6 years of wage stagnation.

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 16 / 41

SLIDE 17

Work Package 1: Economic value

Conclusions[2]

The results also highlight a number of other interesting patterns:

1

Women on average still earn substantially less than men (approximately 16% on average) by the age of 34.

2

The regional differences in pay are staggering, with the majority of the regions showing an average drop in pay of between 30-40% of the baseline income figure.

3

Those with a degree and those with a professional qualification are also well rewarded in terms of pay.

4

Suprising, however, are the results for work experience. A shift from the mean level of work experience to two standard deviations produced little discernible effect. However, those who have worked two standard deviations longer in their current position were also well rewarded.

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 17 / 41

SLIDE 18

Work Package 2: Valued by

Background:

In December 2012, the Advisory Committee on Mathematics Education (ACME) launched their strategy to tackle mathematics provision and

• participation. In the foreword, Professor Stephen Sparks noted the

numbers studying AS and A level mathematics have been rising steadily (after the drop in participation post Curriculum 2000), along with numbers studying AS and A leel Further Mathematics. However, at least 250,000 achieving a grade in GCSE mathematics choose not to study any maths after GCSE.

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 18 / 41

SLIDE 19

Work Package 2: Valued by

Approach: Research question: Who is doing A level Mathematics now? How have participation patterns changed; by social category, by school type, etc? Data: Using the National Pupil Database, we took a cohort based approach and followed the 2002/3 to 2009/10 year groups from their KS4 results to their KS5 A level outcomes from 2003/4 to 2012/13 linking datasets through the unique anonymous pupil identification number. This was cleaned extensively over several months. Sample: The population of those taking A levels in England for the years of data was 2,112,823 from which we took a 2% sample of 42,257. Multilevel Structure: We envisage a four-level model in which individual students are nested in school years, and as such are then nested within KS5 Schools Sixth Form and College providers, which are themselves nested within regions. Model Fit: Bayesian multilevel/hierarchical modelling via Markov Chain Monte Carlo (STAN MC). Model has so far been run with 4 chains each with a total of 1000 iterations including a warmup of 500 iterations each, although this will increase with further model development. Missing Cases: To be confirmed. At the present time we have dealt with it via listwise deletion, although we are looking at the potential to include missing data submodels within the model specification.

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 19 / 41

SLIDE 20

Work Package 2: Valued by

Descriptives

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 20 / 41

SLIDE 21

Work Package 2: Valued by

Model [1]: Individual level

Pr(yi = 1) = logit−1(α + β

Femaleijkl 1jk

+ β

Ethnicity_Blackijkl 2

+ β

Ethnicity_Asianijkl 3

+ β

Ethnicity_Chineseijkl 4

+ β

Ethnicity_Mixedijkl 5

+ β

Ethnicity_Otherijkl 6

+ β

A_Level_Entriesijkl 7

+ β

SEN_Aijkl 8

+ β

SEN_Pijkl 9

+ β

SEN_Sijkl 10

+ β

GCSE_Math_Pointsijkl 11jkl

+ β

GCSE_English_Pointsijkl 12

+ β

Ave_GCSE_Pointsijkl 13

+ β

Diff _Maths−Ave_GCSE_Gradeijkl 14

+ β

IDACI_Scoreijkl 15

+ αSchool−Year

j

+ αKS5_School

k

+ αRegion

l

)

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 21 / 41

SLIDE 22

Work Package 2: Valued by

Model [2]: Group level Model

Group level:

  αj β1j β11j   ∼ N     µα µβ1 µβ11   ,   σ2

α

ρσβ1σβ11 ρσασβ11 ρσασβ1 σ2

β1

ρσασβ1 ρσασβ11 ρσβ1σβ11 σ2β11     , for j = 1, . . . , J   αk β1k β11k   ∼ N     µα µβ1 µβ11   ,   σ2

α

ρσβ1σβ11 ρσασβ11 ρσασβ1 σ2

β1

ρσασβ1 ρσασβ11 ρσβ1σβ11 σ2β11     , for k = 1, . . . , K αl β11l

• ∼ N

µα µβ11

• ,
• σ2

α

ρσασβ ρσασβ σ2β

• , for l = 1, . . . , L

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 22 / 41

SLIDE 23

Work Package 2: Valued by

Model [3]: Priors

Priors: This is still very much a work in progress. For the Bayesian pri-

• rs we have gone with a weakly informative approach which intentionally

includes less information than we have available, but provides enough to improve computation, allowing the data to speak for itself… α ∼ N(0, 5); β ∼ N(0, 5)forβ = 1, . . . , 15; σ2

year ∼ Chalf (0, 2.5);

σ2

KS5_School ∼ Chalf (0, 2.5);

σ2

Region ∼ Chalf (0, 2.5);

ρ ∼ lkj(1.5)

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 23 / 41

SLIDE 24

Work Package 2: Valued by

Improvements planned:

We have estimated a random intercept model, and we are currently expanding this to the model we have detailed in the earlier slides. Of particular interest is the female and ethnic minority participation values. From our model building so far, women are about 20% less likely whereas Black, Asian and Chinese students range from 18-35% more likely to complete A level mathematics. This seems to be a very interesting trend to explore via deep interactions. Group level variables: e.g. School type Linear trend in the probability of completion. Missing data sub-model: We may need to switch to JAGS to estimate as STAN currently can only handle continuous variables (real vs. integer values).

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 24 / 41

SLIDE 25

Work Package 3: Academic value

Background: Scientists Need Better Maths Skills? Three major reports have called for stronger maths skills amongst undergraduate scientists:

1

The Royal Society in their report on UK first degrees in STM produced a list of skills for UK graduates to develop which included: ...[the] ability to think mathematically, to process, present and quantitatively analyse numerical and other scientific data...(2006: 56)”

2

The Royal Society in their State of the Nation report argued: ”that while there was considerable variation in entry requirements, one powerful message coming through was that those who aspire to study university STEM qualifications need to take mathematics in addition to science subjects(2011:15)”.

3

The House of Lords Select Committee on Science and Technology from report stated that: ”the number of pupils studying maths post-16 is insufficient to meet the level of numeracy needed in modern society, and the level at which the subject is taught does not meet the requirements needed to study STEM subjects at undergraduate level”(2012:18).

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 25 / 41

SLIDE 26

Work Package 3: Academic value

Approach[1]

Research question: What is the relationship between A level Mathematics participation and attainment and degree outcomes? Data: We took a cohort based approach and followed the 2005/2006 GCSE group in England and linked this through the unique anonymous pupil identification number to KS5 A level outcomes and also the Higher Education Statistics Agency Data from 2008/2009 to 2012/13. This was cleaned extensively over several months and finally subset into different subjects Sample: 4462 Biology and 1543 Chemistry graduates with a first class

• r upper-second class honours degree from a UK university from the

2005/2006 GCSE cohort who completed their studies by 2012/13. Multilevel Structure: Students at level 1 are clustered in universities (level 2). There are approximately 95 level 2 units for the Biology model and 59 level 2 units for the Chemistry model.

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 26 / 41

SLIDE 27

Work Package 3: Academic value

Approach[2]

Model Fit: Bayesian multilevel/hierarchical modelling via Markov Chain Monte Carlo (STAN MC). Biology model has so far been run with 4 chains, each with a total of 20000 iterations including a warmup of 10000 iterations, which were then thinned by a factor of

• 20. Chemistry model converged more quickly so we ran it for 1000

iterations on 4 separate chains, including a warmup of 500 iterations. Missing cases: This is work in progress. We plan to use joint modelling multiple imputation via REALCOM-IMPUTE/STAT-JR, although both datasets are approximately 95% fully observed.

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 27 / 41

SLIDE 28

Work Package 3: Academic value

Binary Logistic Multilevel Model Specification [1]: Individual level

Pr(yi = 1) = logit−1(α + βFemaleij

1

+ βEthnicity_Blackij

2

+ βEthnicity_Asianij

3

+ βEthnicity_Chineseij

4

+ βEthnicity_Mixedij

5

+ βEthnicity_Otherij

6

+ βPost−16_Mathsij

7j

+ βGCSE_Math_Pointsij

8

+ βGCSE_English_Pointsij

9

+ βQCA_KS5_Pointsi

10

+ βAve_KS4_Pointsij

11

+ βBio_KS5_AGradeij

12

+ βBio_KS5_BGradeij

13

+ βBio_KS5_CDEGradeij

14

+ βChem_KS5_AGradeij

15

+ βChem_KS5_BGradeij

16

+ βChem_KS5_CDEGradeij

17

+ βIntegrated_Masters

18 ∗

+ αuniversity

j

) * Only for the Chemistry model

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 28 / 41

SLIDE 29

Work Package 3: Academic value

Model [2]: Group level and priors

Group level: αj β7j

• ∼ N

µα µβ7

• ,
• σ2

α

ρσασβ ρσασβ σ2β

• , forj = 1, . . . , J

Priors: This is still very much a work in progress. For the Bayesian pri-

• rs we have gone with a weakly informative approach which intentionally

includes less information than we have available, but provides enough to improve computation, allowing the data to speak for itself… α ∼ N(0, 5); β ∼ N(0, 5)forβ = 1, . . . , 17/18; σ2

α ∼ Chalf (0, 2.5);

σ2

β ∼ Chalf (0, 2.5);

ρ ∼ lkj(1.5)

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 29 / 41

SLIDE 30

Work Package 3: Academic value

Major findings [1]: Individual level estimates (Biology)

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 30 / 41

SLIDE 31

Work Package 3: Academic value

Major findings [2]: Individual level estimates (Chemistry)

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 31 / 41

SLIDE 32

Work Package 3: Academic value

Major findings [3]: University level variation in the Russell Group (Biology)

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 32 / 41

SLIDE 33

Work Package 3: Academic value

Major findings [4]: University level variation in the Russell Group (Chemistry)

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 33 / 41

SLIDE 34

Work Package 3: Academic value

Conclusions[1] While we have not been able to model pathways through degree courses - in particular, whether the individual has taken a university level course such as Maths for Biologists or Maths for Chemists for instance, and whether they have chose a path of undertaking modules with mathematical content within their disciplines, there is a clear pattern of A level mathematics participation having a small negative effect on the probability of obtaining a first (although there is more noise than signal with the case of Chemistry). For both subsets, there is a clear pattern of underachievement for ethnic minority

• students. For Black, Asian and Chinese students in Biology, and Black and

Chinese students in Chemistry, in terms of their predicted probabilities, there is a substantially lower probability of obtaining a first class degree. In terms of A level subject combinations, the predictor with the strongest impact for Biology is having an A in A level Chemistry, although this is not repeated for Chemistry due to a large amount of noise.

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 34 / 41

SLIDE 35

Work Package 3: Academic value

Conclusions[2] In terms of variability at the institutional level, there is far greater variability in the probability of gaining a first with Biology degrees. However, the picture from both subjects is very clear with regard to the impact of mathematics. There is very little variation between universities for mathematics and all negative. Clearly, then it is a much more complicated picture than that painted by the three reports mentioned in the background section. We would argue strongly against any idea that advanced mathematical skills were not important to the study of Biology and Chemistry. However, we would argue that A level may not be fit for purpose in supplying skills and experience relevant to these two sciences. What this analysis indicates is that this is where the new Core Mathematics qualification could come in to help develop more relevant mathematics skill sets for science (with the exception of Physics) and we would urge the disciplines to engage with the qualification.

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 35 / 41

SLIDE 36

In Progress

Work Package 4: Political value[1]

Tracing the emergence of research ideas in policy discourse: 1999: New Labour Government; Dearing review precipitates major reform of 16-18 advanced level qualifications; Dolton and Vignoles, based at the LSE, show wage premium for A level maths. 2000-2004: Curriculum 2000 has major impact on advanced qualification landscape; drastic reduction in mathematics participation. Reports on importance of STEM to economic security (Gago, Roberts, RS); Wolf cites Dolton and Vignoles in Does Education Matter (2002). 2004: Tomlinson report recommendations rejected. Smith report on post-14

• maths. Numbers gradually rising

2004-2008: Settled period; Maths Pathways project has little impact. REFORM report (2007). Numbers still rising 2010: New coalition govt.; Wolf report on skills commissioned. Nuffield Outliers report

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 36 / 41

SLIDE 37

In Progress

Work Package 4: Political value [2]

2011: Vorderman report. ACME Mathematical Needs project reports. Gove maths to 18 for vast majority. 2012: Elizabeth Truss becomes junior minister at the Department for Education; ET uses DV result in a variety of places 2012-2015: 10% finding becomes commonplace, and transformed

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 37 / 41

SLIDE 38

In Progress

Work Package 5: Perceived value[1] The project includes a national survey of 17-year olds The timing for the survey is important given that schools are now faced with the question of whether they will

• ffer Core Maths qualifications from 2015

Data linking: NPD agreement on minimum requirements for data linking Match to 2016 2017 A level NPD Timeline: Stage Time Plan Development/piloting Jan-July 14 Makes use of some TIMSS items; plan for linking to NPD Recruitment Sept-Nov 14 Random samples contacted; 116 institutions recruited with possible sample of 14,000 Survey January 15 Returns to date suggest just over 8,000 Data entry April-July 15 Sub-contracted by Seymour Research Analysis Feb/March 15

Table: School survey timeline

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 38 / 41

SLIDE 39

In Progress

Work Package 5: Perceived value[2]

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 39 / 41

SLIDE 40

In Progress

Work Package 5: Perceived value[3]

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 40 / 41

SLIDE 41

Where to next?

Future research avenues

Linked data - Analysis of linked data during 2016/17 Preliminary work has been carried out on estimating the impact of mathematics and computing A level on earnings over the life course and this will be presented at the Centre for Longitudinal Studies conference in March this year (and expanded substantially by the proposed PRiSM project). Predicting A level outcomes - potentially via an ordinal logistic/proportional odds multilevel model, but this may be better served by a different data source. PRiSM - Pipeline and Return to Science and Mathematics Quantitative literacy

Andy Noyes & Mike Adkins (UoN) REVAMP February 11, 2015 41 / 41