School and teacher characteristics vs. student progress Sandra - - PowerPoint PPT Presentation

▶

Mar 15, 2024 269 likes •649 views

School and teacher characteristics vs. student progress Sandra Sousa, Miguel Portela and Carla S September 2017 Stata UGM 2017, Portugal Universidade do Porto 1 / 37 School characteristics Motivation The decision to invest in education

SLIDE 1

School and teacher characteristics vs. student progress

Sandra Sousa, Miguel Portela and Carla Sá September 2017 Stata UGM 2017, Portugal

Universidade do Porto

1 / 37

SLIDE 2

School characteristics Motivation

The decision to invest in education is taken jointly by the family and the student, who

compare the benefits and costs of such investment

Educational externalities
Education is one of the main services provided by governments
School assessment has been an instrument of a guarantee of productivity and efficiency of

the education systems, and used to improve the quality of education

2 / 37

SLIDE 3

School characteristics Motivation

School performance can be evaluated by its value-added
Using data at the student level, for the period 2010–2012, we analyse possible factors that

influence students’ achievement gains on Portuguese and Mathematics national exams

One concludes that achievement gains, both in Portuguese and in Mathematics, are mainly

determined by student’s characteristics

While peers’ characteristics seem not to influence students’ performance, class size does

play a role Research question: What are the factors that determine students’ performance?

3 / 37

SLIDE 4

School characteristics Literature

Internacional

In the economics of education literature, one of the most common conceptual frameworks

employed takes the form of a production function, also referred to as “input–output” analysis:

◮ The output corresponds to the results achieved by the students at the end of a cycle of studies ◮ Education outcomes: test scores, in particular, the maths, reading and science scores, students’

success rates, attendance rates, repetition rates and dropout rates

◮ 3 groups of inputs: student, family e school characteristics

Parametric estimators: OLS, multilevel, fixed-effects
Non-parametric: DEA

4 / 37

SLIDE 5

School characteristics Literature

Internacional

Leading results in the literature:

Lee e Barro (2001) – family background and socio–economic factors are the most important

determinants of student performance as compared to school resources

Hanushek et al. (2003) and Kirjavainen (2012) – the higher the prior achievement scores,

the higher the final achievement scores

Hanushek (1986) and Lee e Barro (2001) – growing up in a low–income family has a

negative impact on educational outcomes

5 / 37

SLIDE 6

School characteristics Literature

Internacional

Hanushek (1986), Lee and Barro (2001), Woessman (2003) and Kirjavainen (2012) –

parents’ education level influences positively student’s performance

Hanushek (1997), Krueger (2003) and Lee and Barro (2001) – the results suggest only

weak relationships between school expenditures and student performance, once one controls for family characteristics

Lee e Barro (2001) and Akerhielm (1995), for example - smaller classes have a positive

effect on student achievement

6 / 37

SLIDE 7

School characteristics Literature

Internacional

Empirical evidence on peer effects is rather mixed
However, the average peer group achievement (Hanushek, 2003), the average education of

mothers of other students in the same class (McEwan, 2003), and a high proportion of girls, (Kirjavainen2012, have a highly significant effect on student performance

Brunello and Rocco (2013) – the higher the share of immigrant pupils in schools, the lower

the performance of native students, especially those with a disadvantaged parental background

7 / 37

SLIDE 8

School characteristics Literature

Portugal

Carneiro (2008) –

◮ The observable factor that contributes the most to the inequality in student performance is the

family background

◮ The school resources have a limited role on student results

Pereira (2010) –

◮ Male students perform better in mathematics and female students have better reading

performance;

◮ Socio–economic background has a strong effect on test scores ◮ Parents’ education (secondary and university education) has a positive impact on student

achievement;

8 / 37

SLIDE 9

School characteristics Literature

Portugal

Ferrão (2012): the relationship between prior achievement and student performance is

stronger than the relationship between socio–economic status and student performance

Oliveira and Santos (2005): school environment characteristics (e.g., unemployment rate,

access to health care services, adult education and living infrastructures) are determinants

f school efficiency
Several authors, e.g., Oliveira and Santos (2005), Pereira e Reis (2012) and Portela et al.

argue that coastline schools have better performance when compared too the inland ones

9 / 37

SLIDE 10

School characteristics Methodology

Education production function

log Aijk = λ log Ai9 + βXijk + δCjk + θSk + ǫijk (1) Aijk: student’s achievement in Portuguese or Mathematics, measured by the 12th grade national exam score, for student i in class j in school k; A9

i : student’s achievement in the 9th grade exam in the same subject;

Xijk : observable student and family characteristics; Cjk: measurable class j, in school k, characteristics; Sk: measurable school characteristics; ǫijk: error term

10 / 37

SLIDE 11

School characteristics Methodology

Estimation methods

OLS
Fixed – effects (school level)
A multilevel model with 3 levels – student, class and school
Non–parametric approach: Data Envelopment Analysis (DEA)

11 / 37

SLIDE 12

School characteristics Data

The dataset was built from two distinct databases managed by the Portuguese Ministry of Education:

MISI (Sistema de Informação do Ministério da Educação): it is a very detailed

administrative database that contains information on pre–school education, as well as basic and secondary education, in public schools, overseen by the Ministry of Education. It contains information at student–level, such as gender, nationality, academic outcome, grade, social support eligibility, type of student, type of education, residence, availability of computer and internet at home, kinship of legal–guardians, legal–guardians’/parents’ employment situation and legal–guardians’/parents’ education, class and school. Information at school–level includes location, school resources

12 / 37

SLIDE 13

School characteristics Data

JNE (Statistics published by Júri Nacional de Exames – Direção Geral de

Educação) : information on scores in national exams on all disciplines of basic and secondary education subjected to examination.

Time interval: 2010 – 2012
Estimations are performed, separately, for national exams on Mathematics and Portuguese
About 36,000 students performed the exam of Mathematics type A and/or Portuguese at

upper secondary education (“12o ano”)

Only internal students and students who enrolled in the national exam but who attended

the discipline throughout the school year are included

13 / 37

SLIDE 14

School characteristics Data

≈ 25% – benefits from social support
≈ 26% – without internet at home
≈ 71% – the mother is the legal–guardian
≈ 53% – with parents/legal–guardians with at least the high school diploma
351 public schools (Portugal mainland)
4,817 teachers in Mathematics and Portuguese

◮ ≈ 35% of teachers work outside their county ◮ ≈ 92% have an undergraduate degree ◮ ≈ 75% are women 14 / 37

SLIDE 15

School characteristics Descriptive statistics

Table 1: Descriptive statistics on students

Variable Mean

Std. Dev.

Mean

Std. Dev.

Mean

Std. Dev.

2010 2011 2012 Mathematics 125.34 46.164 108.33 47.256 106.833 44.743 Portuguese 116.95 29.499 105.93 31.347 112.418 30.261 Maths 9th 95.00 43.401 124.28 47.051 133.712 38.589 Portuguese 9th 138.37 24.046 138.60 25.401 129.029 26.927 Female 0.596 0.588 0.579 Age 18.11 0.401 18.280 0.551 18.27 0.528 Portuguese student 0.991 0.995 0.995 Internet 0.587 0.806 0.832 Beneficiary s.s. 0.267 0.249 0.221 Parent/legal–guardian Father 0.219 0.203 0.195 Mother 0.712 0.700 0.709 Own 0.045 0.076 0.073 Other 0.025 0.021 0.023 Parent/legal–guardian education Tertiary 0.249 0.227 0.241 Secondary 0.221 0.237 0.244 3rd cycle 0.209 0.239 0.235 2nd cycle 0.174 0.165 0.157 1st cycle or less 0.147 0.133 0.122 Parent/guardian employment status Worker for others 0.640 0.636 0.633 Self–employed 0.114 0.101 0.104 Unemployed 0.049 0.053 0.061 Student 0.048 0.079 0.075 Domestic/retired 0.140 0.125 0.119 Other 0.009 0.006 0.008 Source: Computations of the author based on MISI and JNE Statistics, 2010–2012. Note: The number of observations in all variables, except Mathematics and Portuguese variables, is 12,782, 9,984 and 13,131 in 2010, 2011 and 2012, respectively. The corresponding values for Mathe- matics variable are 7,800, 6,109 and 8,454, and for Portuguese variable are 12,773, 9,550 and 12,583, respectively.

15 / 37

SLIDE 16

School characteristics Descriptive statistics

Table 2: Descriptive statistics on classes and schools

Variable Mean

Std. Dev.

Mean

Std. Dev.

Mean

Std. Dev.

2010 2011 2012 Class level variables Class size 23.80 4.961 25.11 5.194 25.57 5.453 % economically disadvantaged 24.46 17.31 23.96 16.74 21.67 15.45 % of girls 57.05 17.91 56.55 17.55 55.98 17.74 % of immigrants 3.608 5.876 3.667 5.666 3.335 5.166 % more educated parent/guardian 83.06 14.88 82.94 14.50 82.49 14.25 School level variables School size 969.7 315.9 1,056 325.3 1,030 342.0 % economically disadvantaged 31.87 14.67 29.67 14.24 28.90 14.03 % of girls 51.51 3.755 50.59 3.878 50.05 3.655 % of immigrants 4.496 4.844 4.890 5.413 4.590 4.941 % more educated parent/guardian 86.40 9.389 86.53 9.869 86.37 9.460 Expenditure–student ratio 567.1 229.1 518.1 204.5 477.6 165.2 Source: Computations of the author based on MISI and JNE Statistics, 2010–2012. Note: The number of classes observed is 1,817, 1,928 and 2,115 in 2010, 2011 and 2012, respectively; the number of schools is 284, 283 and 322 in 2010, 2011 and 2012, respectively.

16 / 37

SLIDE 17

School characteristics Descriptive statistics 5 10 15 20 boy girl

excludes outside values

Mathematics Portuguese

Source: Created by the author based on MISI and JNE Statistics, 2010–2012

Figure 1: Mean national exams scores by discipline and by gender

17 / 37

SLIDE 18

School characteristics Empirical analysis: Multilevel

Table 3: Multilevel model results (Year of examination: 2012)

(1) (2) (3) (4) (5) (6) (7) (8) VARIABLES Dependent variable: Log of Mathematics scores Dependent variable: Log of Portuguese scores Fixed–effects Parameters Log Maths 9th 1.0110*** 1.0000*** 1.0006*** (0.0246) (0.0246) (0.0246) Log Portuguese 9th 0.5869*** 0.5873*** 0.5874*** (0.0098) (0.0099) (0.0099) Female 0.0742*** 0.0773*** 0.0785*** 0.0488*** 0.0484*** 0.0483*** (0.0106) (0.0106) (0.0106) (0.0043) (0.0044) (0.0044) Age

0.1037***
0.1039***
0.1026***
0.1025***
0.1018***
0.1019***

(0.0128) (0.0128) (0.0128) (0.0045) (0.0045) (0.0045) Beneficiary S.S.

0.0571***
0.0533***
0.0534***
0.0062
0.0080
0.0080

(0.0142) (0.0144) (0.0144) (0.0054) (0.0055) (0.0055) Internet 0.0217 0.0166 0.0190

0.0009
0.0003
0.0004

(0.0158) (0.0158) (0.0158) (0.0063) (0.0063) (0.0064) Portuguese student

0.0120
0.0161
0.0152
0.0418
0.0482
0.0483

(0.0726) (0.0727) (0.0726) (0.0283) (0.0284) (0.0284) Class size 0.0281** 0.0303** 0.0123*** 0.0122*** (0.0106) (0.0106) (0.0034) (0.0034) Class size sq

0.0006**
0.0006**
0.0002***
0.0002***

(0.0002) (0.0002) (0.0001) (0.0001) % of economically disadvantaged (class)

0.0004
0.0007

0.0003 0.0003 (0.0006) (0.0006) (0.0002) (0.0002) % of female (class)

0.0009
0.0008
0.0015*
0.0015*

(0.0017) (0.0017) (0.0006) (0.0006) % more educated parents/guardians (class)

0.0029***
0.0040***

0.0004 0.0004 (0.0006) (0.0007) (0.0002) (0.0003) % more educated parents/guardians (school) 0.0031**

0.0002

(0.0011) (0.0005) Continued on next page

18 / 37

SLIDE 19

School characteristics Empirical analysis: Multilevel

Table 5.4.3 – continued from previous page Random–effects Parameters Level–three variance: σ2

r0 Intercept variance

0.0130*** 0.0067*** 0.0075*** 0.0073*** 0.0036*** 0.0021*** 0.0020*** 0.0020*** (0.0028) (0.0017) (0.0018) (0.0017) (0.0006) (0.0004) (0.0003) (0.0003) Level–two variance: σ2

u0 Intercept variance

0.0361*** 0.0174*** 0.0148*** 0.0147*** 0.0079*** 0.0034*** 0.0033*** 0.0033*** (0.0040) (0.0025) (0.0023) (0.0023) (0.0007) (0.0004) (0.0004) (0.0004) Level–one variance: σ2

e Residual variance

0.2714*** 0.2101*** 0.2101*** 0.2101*** 0.0727*** 0.0506*** 0.0507*** 0.0507*** (0.0046) (0.0035) (0.0035) (0.0035) (0.0010) (0.0007) (0.0007) (0.0007) Deviance 13877.12 11463.36 11414.94 11406.04 3910.46

935.15
959.05
959.18

Observations 8454 8454 8454 8454 12583 12583 12583 12583 LR test (χ2) 430.80 217.47 196.72 192.84 598.56 384.73 351.22 349.85 Prob > chi2 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Source: Computations of the author based on MISI and JNE Statistics, 2010–2012. Note: Standard errors in parentheses. Significance levels: *** p<0.01, ** p<0.05, * p<0.1. All regressions include a set of dummies for the parents’/legal– guardians’ education, except in the null model.

19 / 37

SLIDE 20

School characteristics Empirical analysis: Multilevel

14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 7 9 1 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 2 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 4 5 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 6

−.06 .06 Prediction of random effects 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 Rank

Mathematics: 2012

11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 7 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 8 5 2 4 9 1 6 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 3 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18

−.06 .06 Prediction of random effects 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 Rank

Portuguese: 2012

Source: Created by the author based on MISI and JNE Statistics, 2010–2012. Note: Each number represents a district: 1 – Aveiro; 2 – Beja; 3 – Braga; 4 – Bragança; 5 – Castelo Branco; 6 – Coimbra; 7 – Évora; 8 – Faro; 9 – Guarda; 10 – Leiria; 11 – Lisboa; 12 – Portalegre; 13 – Porto; 14 – Santarém; 15 – Setúbal; 16 – Viana do Castelo; 17 – Vila Real; 18 – Viseu.

Figure 2: District effects and approximate 95% confidence intervals versus ranking of districts in Mathematics and Portuguese achievement gains (district identifiers are shown on the top of the error bar)

20 / 37

SLIDE 21

School characteristics Empirical analysis: Multilevel

241 241 241 241 241 241 241 241 241 241 241 53 53 53 53 53 53 53 53 53 53 53 53 53 64 64 64 64 64 64 64 64 64 64 64 64 64 64 64 64 64 64 64 64 183 183 183 183 183 183 183 183 183 183 183 183 183 183 183 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 273 273 273 273 273 273 273 273 273 273 273 273 273 273 273 273 273 273 273 273 273 273 273 76 76 76 76 76 76 76 76 76 76 76 76 76 76 76 76 76 76 76 76 76 76 76 76 76 76 76 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 105 105 105 105 105 105 105 105 105 105 57 57 57 57 57 57 57 57 57 57 57 57 71 71 71 71 71 71 71 71 71 71 71 71 71 71 71 71 71 71 71 71 71 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 220 220 220 220 220 220 220 220 220 220 220 220 220 220 220 220 220 220 220 220 220 220 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 85 168 168 168 168 168 168 168 168 168 168 168 168 168 168 168 253 253 253 253 253 253 253 253 253 253 253 253 253 253 253 253 253 253 253 253 253 253 267 267 267 267 267 267 267 267 267 267 267 267 267 267 267 267 267 267 267 267 267 267 267 267 275 275 275 275 275 275 275 275 275 275 275 275 275 275 275 275 275 275 275 275 275 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 305 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 237 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 270 328 328 328 328 328 328 328 328 328 328 328 328 328 328 328 328 328 328 328 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 306 306 306 306 306 306 306 306 306 306 306 306 306 306 306 306 306 306 306 306 306 83 83 83 83 83 83 83 83 83 83 83 83 83 83 83 83 7 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 3 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 146 146 146 146 146 146 146 146 146 146 146 146 146 146 146 146 146 146 146 146 146 146 146 146 146 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 180 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 234 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 54 54 54 54 54 54 54 54 54 54 54 54 54 54 54 54 54 54 54 54 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 263 8 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228 228

−.1 .25 Prediction of random effects 50 45 40 35 30 25 20 15 10 5 1 Rank

Mathematics: 2012

40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 201 139 139 139 139 139 139 139 139 139 139 139 139 139 139 139 139 139 139 156 156 156 156 156 156 156 156 156 156 156 156 156 156 156 156 156 156 156 244 244 244 244 244 244 244 244 244 244 244 244 244 244 244 244 244 244 244 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 203 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 312 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 247 209 209 209 209 209 209 209 209 209 209 209 209 209 209 209 209 209 209 209 209 209 209 209 209 209 209 209 168 168 168 168 168 168 168 168 168 168 168 168 168 168 168 168 168 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 271 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 145 125 125 125 125 125 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 225 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 138 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 332 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 122 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 173 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 26 26 26 26 26 26 26 26 26 26 26 26 26 26 26 26 26 26 26 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 195 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 177 2 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 193 193 193 193 193 193 193 193 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 261 261 261 261 261 261 261 261 261 261 261 261 261 261 278 278 278 278 278 278 278 278 278 278 278 278 278 278 278 278 278 278 278 278 278 278 278 278 278 278 278 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 155 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 328 328 328 328 328 328 328 328 328 328 328 328 328 328 328 328 328 328 328 328 328 328 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 294 224 224 224 224 224 224 224 224 224 224 224 224 224 224 224 224 224 224 224 224 224 224 224 224 224 224 224 224 4 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 321 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 50 50 50 50 50 50 50 50 50 50 50 50 50 50 50 50 50 50 50 50 50 50 50 50 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236 236

−.1 .25 Prediction of random effects 50 45 40 35 30 25 20 15 10 5 1 Rank

Portuguese: 2012

Source: Created by the author based on MISI and JNE Statistics, 2010–2012

Figure 3: School effects and approximate 95% confidence intervals versus ranking of 50 best schools in Mathematics and Portuguese achievement gains (school identifiers are shown on the top of the error bar)

21 / 37

SLIDE 22

School characteristics Empirical analysis: Multilevel

114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 114 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 338 338 338 338 338 338 338 338 338 338 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 298 266 266 266 266 266 266 266 266 266 266 266 266 266 266 266 266 266 35 35 35 35 35 35 35 35 35 35 35 35 35 35 35 35 35 35 35 35 35 35 35 35 35 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 252 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 319 152 152 152 152 152 152 152 152 152 152 152 152 152 152 152 152 152 152 152 152 152 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 143 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 235 235 235 235 235 235 235 235 235 235 235 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 56 188 188 188 188 303 303 303 303 303 303 303 303 303 303 303 303 303 303 303 303 303 303 303 303 303 303 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 49 123 123 123 123 123 123 123 123 123 123 123 123 123 123 123 123 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 329 329 329 329 329 329 329 329 329 329 329 329 329 329 329 329 329 329 329 329 329 329 329 329 329 329 329 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 230 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 215 215 215 215 215 215 215 215 215 215 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 142 142 142 142 142 142 142 142 142 142 142 142 142 142 142 142 142 142 142 142 179 179 179 179 179 179 179 179 179 179 179 179 179 179 179 282 282 282 282 282 282 282 282 282 282 282 282 282 282 282 282 282 282 282 282 282 149 149 149 149 149 149 149 149 149 149 149 149 149 149 149 149 149 149 149 149 149 149 149 149 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 337 98 98 98 98 98 98 98 98 98 98 98 98 98 98 98 98 98 98 98 98 98 98 98 98 98 98 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 178 206 206 206 206 206 206 206 206 206 206 206 206 206 69 69 69 69 69 69 69 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 112 112 112 112 112 112 112 112 112 112 112 112 112 112 112 112 112 112 112 112 112 112 112 112 112 112 112 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 265 352 352 352 352 352 352 352 352 352 352 352 352 352 352 352 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 347 347 347 347 347 327 327 327 327 327 327 327 327 327 327 327 327 327 327 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 277 281 281 281 281 281 281 281 281 281 281 281 281 281 281 281 281 281 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 197 197 197 197 197 197 197 197 197 197 197 197 197 197 197 197 197 197 197 197 197 197

−.3 .15 Prediction of random effects 270 275 280 285 290 295 300 305 310 315 320 Rank

Mathematics: 2012

226 226 226 226 226 226 226 226 226 226 226 226 226 226 226 226 226 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 60 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 182 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 211 8 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 245 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 233 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 348 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 210 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 171 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 285 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 107 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 350 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 154 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 186 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 293 214 214 214 214 214 214 214 214 214 214 214 214 214 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 103 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 248 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 116 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 82 222 222 222 222 222 222 222 222 222 222 222 222 222 222 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 205 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 292 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 352 352 352 352 352 352 352 352 352 352 352 352 352 352 352 352 352 352 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 120 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 148 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 316 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 147 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 196 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 133 343 343 343 343 343 343 343 343 343 343 343 343 343 343 343 343 343 343 343 343 343 343 1 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 132 132 132 132 132 132 132 132 132 132 132 132 132 132 132 132 132 132 132 132 132 132 132 132 132 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 242 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 246 113 113 113 113 113 113 113 113 113 113 113 113 113 113 113 113 113 113 113 113 81 81 81 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 254 280 280 280 280 280 280 158 158 158 158 158 158 158 158 158 158 158

−.3 .15 Prediction of random effects 272 277 282 287 292 297 302 307 312 317 322 Rank

Portuguese: 2012

Source: Created by the author based on MISI and JNE Statistics, 2010–2012.

Figure 4: School effects and approximate 95% confidence intervals versus ranking of 50 worst schools in Mathematics and Portuguese achievement gains (school identifiers are shown on the top of the error bar)

22 / 37

SLIDE 23

School characteristics Discussion

Student prior achievement is the strongest predictor of current performance in both

subjects, Mathematics and Portuguese (the effect is stronger for Math)

Female students perform better than males students in both fields; results for 2012 show

that on average girls perform better than boys in about 8% for Mathematics and 5% for Portuguese

Age has a negative and significant influence on student achievement gains, which probably

reflects a grade repetition effect

Low income students perform worse than the higher income ones, mainly in Mathematics

(-5% in 2012)

23 / 37

SLIDE 24

School characteristics Discussion

Student background characteristics, in particular, the parent/guardian education has an

important, positive influence on student achievement gains

School resources, measured by the teaching expenditure–student ratio, have no effect on

student achievement gains

At the school level, the class size gets the highest weight in students’ performance, mainly

in Mathematics: by adding 5 students to a class of size 20 increases student’s performance by about 1,7%; an increase of 10% reduces the effect to 0,3% (estimates for 2012))

In most cases the school–effect is null

24 / 37

SLIDE 25

Teacher characteristics

How do teacher characteristics impact on student’s performance?

25 / 37

SLIDE 26

Teacher characteristics Methodology

Production function

log Aijt = λ log Ai9 + βXit + δTjt + γt + εijt (2) Aijt: is the student outcome in the 12th grade national exams Portuguese or Mathematics A, measured by the national exam score; Ai9: score of the ith student in the national exam of 9th grade; Xit : vector of student and family background characteristics; Tjt: measurable teacher characteristics and includes the class size; γt: time fixed-effects; εijt: error term.

26 / 37

SLIDE 27

Teacher characteristics Methodology

Estimation methods

Ordinary Least Squares (OLS);
Fixed-effects (FE)

log Aijt = λ log Ai9 + βXit + δTjt + τj + εijt (3) where τj is a teacher fixed–effect.

27 / 37

SLIDE 28

Teacher characteristics Data

A matched student–teacher dataset including the national test scores of students in

Mathematics and Portuguese, student background information, and teacher information;

Information at the student–level: gender, date of birth, nationality, academic outcomes, year
f schooling, social support eligibility, residence, availability of computer and internet at

home, parents’ employment situation and parents’ education, class and school, among

thers;
Information at the teacher–level: gender, date of birth, education, teaching experience,

disciplinary group, salary, county and district of residence, among others;

Period: 2010 – 2012;

28 / 37

SLIDE 29

Teacher characteristics Data

The sample contains:

21,549 student observations:

◮ attended the scientific–humanistic courses of secondary education; ◮ performed the 12th grade national exams of Mathematics and Portuguese; ◮ aged between 17 and 20 years (91% of students are 18 years old); ◮ 23% of the students benefit from social support; ◮ 73% of the students have internet access at home.

4,817 unique teachers:

◮ working in 446 Portuguese public secondary schools; ◮ about 35.1% are working outside their county of residence; ◮ 91.8% of the teachers have a bacharelato or bachelor degree as the highest level of education; ◮ 75% are female; ◮ 50% of the teachers have at least 25 years of experience; 29 / 37

SLIDE 30

Teacher characteristics Empirical results: OLS

Table 4: OLS estimates

(1) (2) (3) (4) (5) (6) (7) (8) (9) All students Mathematics Portuguese Log of 9th exam scores 0.7110*** 0.7101*** 0.7101*** 0.9534*** 0.9503*** 0.9505*** 0.6972*** 0.6959*** 0.6959*** (0.0129) (0.0129) (0.0129) (0.0242) (0.0241) (0.0241) (0.0133) (0.0133) (0.0133) Female student 0.0598*** 0.0597*** 0.0598*** 0.0594*** 0.0589*** 0.0589*** 0.0602*** 0.0604*** 0.0604*** (0.0050) (0.0050) (0.0050) (0.0103) (0.0103) (0.0103) (0.0044) (0.0044) (0.0044) Age

0.1211***
0.1210***
0.1210***
0.1865***
0.1845***
0.1841***
0.0974***
0.0968***
0.0968***

(0.0077) (0.0077) (0.0077) (0.0234) (0.0232) (0.0232) (0.0066) (0.0066) (0.0066) Beneficiary S.S.

0.0494***
0.0489***
0.0488***
0.0853***
0.0842***
0.0844***
0.0301***
0.0295***
0.0296***

(0.0062) (0.0062) (0.0062) (0.0141) (0.0141) (0.0141) (0.0053) (0.0053) (0.0053) Internet 0.0196*** 0.0195*** 0.0196*** 0.0175 0.0177 0.0179 0.0157*** 0.0157*** 0.0157*** (0.0057) (0.0057) (0.0057) (0.0126) (0.0126) (0.0126) (0.0049) (0.0049) (0.0049) Continued on next page 30 / 37

SLIDE 31

Teacher characteristics Empirical results: OLS Continued from previous page Female teacher 0.0219*** 0.0219*** 0.0222*** 0.0288** 0.0278** 0.0281** 0.0120** 0.0120** 0.0120** (0.0060) (0.0060) (0.0060) (0.0117) (0.0117) (0.0117) (0.0053) (0.0053) (0.0053) Advanced degree 0.0157* 0.0161* 0.0165*

0.0082
0.0089
0.0080

0.0254*** 0.0253*** 0.0252*** (0.0089) (0.0089) (0.0089) (0.0198) (0.0198) (0.0197) (0.0071) (0.0071) (0.0071) Experience 0.0013*** 0.0013***

0.0010

0.0012 0.0012

0.0061

0.0007** 0.0007** 0.0011 (0.0004) (0.0004) (0.0022) (0.0008) (0.0008) (0.0046) (0.0003) (0.0003) (0.0018) Experience sq 0.0000 0.0002

0.0000

(0.0000) (0.0001) (0.0000) Commuting

0.0081
0.0078
0.0081
0.0162
0.0152
0.0163
0.0084*
0.0080*
0.0079*

(0.0053) (0.0053) (0.0053) (0.0114) (0.0113) (0.0114) (0.0046) (0.0046) (0.0046) Class size 0.0156*** 0.0156*** 0.0355*** 0.0356*** 0.0064** 0.0065** (0.0035) (0.0035) (0.0087) (0.0087) (0.0028) (0.0028) Class size sq

0.0003***
0.0003***
0.0007***
0.0007***
0.0001**
0.0001**

(0.0001) (0.0001) (0.0002) (0.0002) (0.0001) (0.0001) Observations 21,549 21,549 21,549 8,100 8,100 8,100 13,449 13,449 13,449 R-squared 0.251 0.252 0.252 0.298 0.300 0.300 0.329 0.329 0.329 RMSE 0.245 0.245 0.245 0.245 0.245 0.245 0.245 0.245 0.245 Source: Computations of the author based on MISI and JNE Statistics, 2010–2012. Note: Robust standard errors in parentheses. Significance levels: *** p<0.01, ** p<0.05, * p<0.1. The dependent variable is log 12th grade national exam score. All regressions include a set of dummies to control for district/region and year. 31 / 37

SLIDE 32

Teacher characteristics Empirical results: FE

Table 5: FE estimates

(1) (2) (3) (4) (5) (6) All students Mathematics Portuguese Log of 9th exam scores 0.7067* 0.7066* 0.8347* 0.8341* 0.6466* 0.6463* (0.0149) (0.0148) (0.0297) (0.0296) (0.0151) (0.0151) Female student 0.0544* 0.0544* 0.0661* 0.0661* 0.0607* 0.0607* (0.0055) (0.0055) (0.0126) (0.0126) (0.0054) (0.0054) Age

0.1112***
0.1105***
0.1983***
0.1963***
0.0932***
0.0925***

(0.0087) (0.0088) (0.0289) (0.0289) (0.0078) (0.0078) Beneficiary S.S.

0.0443***
0.0442***
0.0706***
0.0714***
0.0327***
0.0325***

(0.0072) (0.0072) (0.0188) (0.0187) (0.0064) (0.0064) Internet 0.0141* 0.0142* 0.0017 0.0024 0.0226* 0.0225* (0.0080) (0.0080) (0.0191) (0.0190) (0.0073) (0.0073) Continued on next page

32 / 37

SLIDE 33

Teacher characteristics Empirical results: FE

Continued from previous page Advanced degree

0.0223
0.0218
0.0957
0.0913

0.0505 0.0501 (0.0507) (0.0505) (0.1136) (0.1107) (0.0411) (0.0411) Experience 0.0565* 0.0567* 0.2238* 0.2195* 0.0076 0.0084 (0.0211) (0.0209) (0.0492) (0.0486) (0.0179) (0.0179) Experience sq

0.0018***
0.0018***
0.0065***
0.0063***
0.0003
0.0003

(0.0004) (0.0004) (0.0010) (0.0010) (0.0004) (0.0004) Commuting

0.0216**
0.0215**
0.0136
0.0136
0.0214**
0.0212**

(0.0100) (0.0100) (0.0255) (0.0254) (0.0090) (0.0090) Class size 0.0142** 0.0315* 0.0103** (0.0060) (0.0169) (0.0051) Class size sq

0.0003**
0.0007**
0.0002*

(0.0001) (0.0003) (0.0001) Observations 21,549 21,549 8,100 8,100 13,449 13,449

No. of teachers

4,817 4,817 2,868 2,868 3,828 3,828

Source: Computations of the author based on MISI and JNE Statistics. Note: Robust standard errors in parentheses. Significance levels: *** p<0.01, ** p<0.05, * p<0.1. The dependent variable is log 12th grade national exam score. All regressions include a set of dummies to control for district/region. 33 / 37

SLIDE 34

Teacher characteristics Notes

Results show that female teachers have a positive effect on students’ achievement gains
Teachers working away from home have negative and significant effects on student results
Teachers with more qualifications (postgraduate, masters or PhDs) do not show better

performance than those with a bacharelato or licentiate diploma

Teachers with more experience are more effective in increasing students’ performance than

those with less experience

34 / 37

SLIDE 35

Teacher characteristics Multiple fixed-effects

Table 6: Teacher & School fixed-effects: mathematics

No FE Teacher FE School FE

T. and S. FE

Math9 0.6551* 0.6279* 0.6324* 0.6284* (0.0129) (0.0142) (0.0131) (0.0143) Female 0.0868* 0.0845* 0.0853* 0.0847* (0.0086) (0.0092) (0.0086) (0.0092) R2-adjusted 0.30 0.33 0.32 0.33 RMSE 0.42 0.41 0.42 0.41 Source: Computations of the author based on MISI and JNE Statistics, 2010–2012. Note: The number of observations 11,407. 2018 teachers & 286 schools. 52% girls. Math9: mean = 135, median = 140, 10th percentile = 74, sd = 40.

35 / 37

SLIDE 36

Teacher characteristics Multiple fixed-effects 4 4.5 5 5.5 Linear Prediction 50 100 150 School ID

Predictive Margins of eeid with 95% CIs

Source: Created by the author based on MISI and JNE Statistics, 2010–2012

Figure 5: Marginal effects by School

36 / 37

SLIDE 37

Teacher characteristics Multiple fixed-effects 3.5 4 4.5 5 5.5 6 Linear Prediction 200 400 600 800 1000 Teacher ID

Predictive Margins of eeid with 95% CIs

Source: Created by the author based on MISI and JNE Statistics, 2010–2012

Figure 6: Marginal effects by Teacher

37 / 37

School and teacher characteristics vs. student progress

Sandra Sousa, Miguel Portela and Carla Sá September 2017 Stata UGM 2017, Portugal

Universidade do Porto

compare the benefits and costs of such investment

the education systems, and used to improve the quality of education

influence students’ achievement gains on Portuguese and Mathematics national exams

determined by student’s characteristics

play a role Research question: What are the factors that determine students’ performance?

Internacional

employed takes the form of a production function, also referred to as “input–output” analysis:

success rates, attendance rates, repetition rates and dropout rates

Internacional

Leading results in the literature:

determinants of student performance as compared to school resources

the higher the final achievement scores

negative impact on educational outcomes

Internacional

parents’ education level influences positively student’s performance

weak relationships between school expenditures and student performance, once one controls for family characteristics

effect on student achievement

Internacional

mothers of other students in the same class (McEwan, 2003), and a high proportion of girls, (Kirjavainen2012, have a highly significant effect on student performance

the performance of native students, especially those with a disadvantaged parental background

Portugal

family background

performance;

achievement;

Portugal

stronger than the relationship between socio–economic status and student performance

access to health care services, adult education and living infrastructures) are determinants

argue that coastline schools have better performance when compared too the inland ones

Education production function

log Aijk = λ log Ai9 + βXijk + δCjk + θSk + ǫijk (1) Aijk: student’s achievement in Portuguese or Mathematics, measured by the 12th grade national exam score, for student i in class j in school k; A9

i : student’s achievement in the 9th grade exam in the same subject;

Xijk : observable student and family characteristics; Cjk: measurable class j, in school k, characteristics; Sk: measurable school characteristics; ǫijk: error term

Estimation methods

The dataset was built from two distinct databases managed by the Portuguese Ministry of Education:

Educação) : information on scores in national exams on all disciplines of basic and secondary education subjected to examination.

upper secondary education (“12o ano”)

the discipline throughout the school year are included

Table 1: Descriptive statistics on students

Table 2: Descriptive statistics on classes and schools

Source: Created by the author based on MISI and JNE Statistics, 2010–2012

Figure 1: Mean national exams scores by discipline and by gender

Table 3: Multilevel model results (Year of examination: 2012)

Figure 2: District effects and approximate 95% confidence intervals versus ranking of districts in Mathematics and Portuguese achievement gains (district identifiers are shown on the top of the error bar)

Source: Created by the author based on MISI and JNE Statistics, 2010–2012

Figure 3: School effects and approximate 95% confidence intervals versus ranking of 50 best schools in Mathematics and Portuguese achievement gains (school identifiers are shown on the top of the error bar)

Source: Created by the author based on MISI and JNE Statistics, 2010–2012.

Figure 4: School effects and approximate 95% confidence intervals versus ranking of 50 worst schools in Mathematics and Portuguese achievement gains (school identifiers are shown on the top of the error bar)

subjects, Mathematics and Portuguese (the effect is stronger for Math)

that on average girls perform better than boys in about 8% for Mathematics and 5% for Portuguese

reflects a grade repetition effect

(-5% in 2012)

important, positive influence on student achievement gains

student achievement gains

in Mathematics: by adding 5 students to a class of size 20 increases student’s performance by about 1,7%; an increase of 10% reduces the effect to 0,3% (estimates for 2012))

How do teacher characteristics impact on student’s performance?

Production function

Estimation methods

log Aijt = λ log Ai9 + βXit + δTjt + τj + εijt (3) where τj is a teacher fixed–effect.

Mathematics and Portuguese, student background information, and teacher information;

home, parents’ employment situation and parents’ education, class and school, among

disciplinary group, salary, county and district of residence, among others;

The sample contains:

Table 4: OLS estimates

Table 5: FE estimates

(0.0087) (0.0088) (0.0289) (0.0289) (0.0078) (0.0078) Beneficiary S.S.

(0.0072) (0.0072) (0.0188) (0.0187) (0.0064) (0.0064) Internet 0.0141* 0.0142* 0.0017 0.0024 0.0226*** 0.0225*** (0.0080) (0.0080) (0.0191) (0.0190) (0.0073) (0.0073) Continued on next page

Continued from previous page Advanced degree

0.0505 0.0501 (0.0507) (0.0505) (0.1136) (0.1107) (0.0411) (0.0411) Experience 0.0565*** 0.0567*** 0.2238*** 0.2195*** 0.0076 0.0084 (0.0211) (0.0209) (0.0492) (0.0486) (0.0179) (0.0179) Experience sq

(0.0004) (0.0004) (0.0010) (0.0010) (0.0004) (0.0004) Commuting

(0.0100) (0.0100) (0.0255) (0.0254) (0.0090) (0.0090) Class size 0.0142** 0.0315* 0.0103** (0.0060) (0.0169) (0.0051) Class size sq

(0.0001) (0.0003) (0.0001) Observations 21,549 21,549 8,100 8,100 13,449 13,449

4,817 4,817 2,868 2,868 3,828 3,828

performance than those with a bacharelato or licentiate diploma

those with less experience

Table 6: Teacher & School fixed-effects: mathematics

No FE Teacher FE School FE

Source: Created by the author based on MISI and JNE Statistics, 2010–2012

(0.0072) (0.0072) (0.0188) (0.0187) (0.0064) (0.0064) Internet 0.0141* 0.0142* 0.0017 0.0024 0.0226* 0.0225* (0.0080) (0.0080) (0.0191) (0.0190) (0.0073) (0.0073) Continued on next page

0.0505 0.0501 (0.0507) (0.0505) (0.1136) (0.1107) (0.0411) (0.0411) Experience 0.0565* 0.0567* 0.2238* 0.2195* 0.0076 0.0084 (0.0211) (0.0209) (0.0492) (0.0486) (0.0179) (0.0179) Experience sq