Comparative Effectiveness Evaluation and Monitoring, Austrian - - PowerPoint PPT Presentation

Comparative Effectiveness Evaluation and Monitoring, Austrian perspectives

Workshop: More Effective Public Workforce Programs through Comparative Performance Monitoring

Helmut Mahringer November 13, 2018

1

Impact evaluation on active labor market policies in Austria

 Improvement of evaluation culture in Austrian Labor Market Policy

 Since Austria joined the EU in 1995

Initially: Obligation for systematic evaluation of European Social Fund (ESF) defined by European Commission (EC) First large project commissioned by the ML: Evaluation of Austrian ESF-Program compared treatment effects of training between nationally financed programs and ESF (OP 2000-2006)

 Application of monitoring tools and counterfactual impact analysis

Mutual trust between Public Employment Service (PES) and its supervisory authority (the Federal Ministry of Labor (ML)) that evaluation results are used in a constructive way PES increasingly engaged in monitoring and impact analysis (e.g. BSC , randomized controlled trial on case loads of counselling officers)

 Development of data-background by PES and ML: European best

practice example 2003

Based on administrative employer-employee microdata joined with PES data on unemployment, UI-benefits, active LMP measures (EU best practice example)

 Evidence from counterfactual impact evaluation available for

most workforce development programs in Austria

2

Evidence from counterfactual impact analysis of LMP in Austria

Skills and training measures

 Job-search training

 Causal evaluation strategy: control group design (propensity score matching)  No significant effect on outcome variables: (unsubsidized) employment

 Occupational orientation and guidance

 Causal evaluation strategy: control group design (ps matching)  No significant effect on (unsubsidized) employment

 Basic skills development and further training

 Causal evaluation strategy: control group design (ps matching)  Positive effects on employment (after lock-in)  Higher impact for women, middle age and higher qualified

3

Evidence from counterfactual impact analysis of LMP in Austria

Employment measures

 Temporary wage subsidies:

 Causal evaluation strategy: exploiting variation in regional treatment intensities

combined with control group design (matching unemployed and jobs)

 High deadweight effects (~50% of the cases), positive effects on employment  Higher effects for older and long-term unemployed

 Transitional employment in non-profit projects:

 Causal evaluation strategy: control group design (matching)  positive effects on employment (after lock-in)  Higher effects for women, long-term unemployed, unemployed with health

issues

4

Evidence from counterfactual impact analysis of LMP in Austria

Instruments handled by PES-officers

 Placement counseling

 Causal evaluation strategy: random controlled trial  Result: shorter unemployment spells due to transition into employment as well

as into inactivity

 Heterogeneity: more effective for the long-term unemployed, less for

unemployed with health issues and the young

 Longer period of UB receipt

 Causal evaluation strategy: regression discontinuity design on age limits (52

instead of 39 weeks of entitlement with age 50)

 No significant effects on transition into employment but less transition into

inactivity

 Higher effect on transition into inactivity for women due to means-testing of

Unemployment Assistance (following UB)

 More sanctions (withdrawal of UB)

 Causal evaluation strategy: regional variation in the frequency (strictness) of

the application of sanctions

 No significant effects on transition into employment but more transition into

inactivity

5

Concerns about effectiveness of workforce development programs

 Large heterogeneity of treatment effects for different types of training

measures

 Both, with respect to target groups and types of measures  Low effectiveness in particular for the low skilled unemployed

 Unobserved heterogeneity of training measures

 A more differentiated view of the type, content and quality aspects of the

training measures is needed to assess (possible reasons for) their effectiveness

 More informative data required  Heterogeneity in regional strategies (regional PES branches)

 Training measures cover a large share of labor market policy

expenditures

 High share of low skilled unemployed, while skill requirements increase  Public debate about effectiveness of PES training measures

=> Training effectiveness strategy within the PES

6

Research agenda to support an

• ngoing effectiveness strategy

 Improving available information

 A more precise coding of PES training measures is required to obtain insight on

the effects of certain types of training:

Learning content of measures (e.g. occupational skills, language, social skills, …), costs and duration, target group, modular combination with other measures, structure of participants in courses, competition of courses, …

 Exploiting not encoded data records, combining available information

 Identification of causal impacts of training measures

 Effect heterogeneity with respect to types of training, target groups, …  (Fiscal) efficiency

 Comparison of performance (impacts) between regions

 To take into account differences in regional composition of unemployed …  … and the regional labor market situation  Comparability is required to identify successful regional strategies

7

Impact analysis for individual types of measures

 Counterfactual analysis of treatment effects

 Control group design: Nearest Neighbor Propensity Score Matching  Applied separately for a regional breakdown based on a suitable aggregation

regional PES-branches

 Indicators for labor market outcomes

 Employment, unemployment, out-of-labor force

measured after x months after entry into treatment,

• r as days within a period after entry into treatment,

 time-to-job duration, retention rate in employment, entry-wage, …

 Problem of comparability of outcomes and treatment effects between

regions (or different periods)

8

Comparison of program performance between regions

 Regional outcomes of training programs may differ for several reasons

 program effectiveness  participants characteristics  labor market conditions

 Objective: Fair comparison of regional program effectiveness

 Personal characteristics and local labor market conditions affect labor market

• utcomes of (potential) participants in programs

 Raw (unadjusted) comparison would judge regional program performance

unfairly …

 … and would set unintended incentives (cream-skimming)

=> Adjustment of regional outcomes/effects for composition of participants and labor market conditions is required

9

Comparison of program performance between regions

 Well-established method for adjusted performance measurement (see

Eberts et al., 2018)

 Statistical estimation: how much of the variation in individual outcomes is due

to individual characteristics and local economic conditions …

 … and how much is due to the program performance (“value added”)

𝒋𝒌 𝒋𝒌 𝒌 𝒋𝒌 𝑍

… measured outcome/effect for participant i in region j

𝑌 … is a vector of personal characteristics or labor market conditions for participant i in region j and B … is a vector of estimated coefficients for 𝑌 𝑋

… is the estimated performance of region j

𝜻 … is an error term

 The regional program performance

𝒌 equals the average outcome/effect measured in the region j, minus the outcome that would be expected given

the average characteristics of participants and labor market conditions in region j

𝒌 𝒌 𝒌

10

Comparison of program performance between regions

 Estimation on individual participant data

 makes full use of heterogeneity across individuals (to include composition

effects adequately)

 Comprehensive information on participants (and other unemployed) available

 Regional control group design would also allow for comparison of

(causal) effects between regions

 Average treatment effect for the participants

 Choice of estimation model

 OLS, logistic regression, spatial models

11

Data background

 Merged administrative individual datasets

 Administrative PES-data:

Individual characteristics (date of birth, gender, level of education, existence of health restrictions, care responsibilities ...) Unemployment episodes on daily basis, including information on participation in active measures and other temporary reasons for interruption of UB receipt Interventions by PES (appointments, placement suggestions, assignment to training, sanctions, ...)

 Austrian social security records: matched employer-employee

dataset regularly processed at WIFO

Employment relationships (duration on daily basis, wages on yearly basis) Employee characteristics (age, gender, full labor market career back to 1972, ...) Employer characteristics (industry, number of employees, ...)

12

Potential problems

 Relevant personal characteristics or regional labor market

conditions may not be fully observable

 Omitted variables may lead to based results

 Program performance may not be separable from other policies

 Training programs may interact with placement service of the PES  Adequate specification of the programs considered is necessary for

performance measurement

Homogeneous measure vs. policy system?

 Correspondence between policy(-level) and performance indicator

Is the outcome substantially influenced by the policy?

13

Conclusions

 Performance measurement systems could improve labor market

policies in Austria

 Excellent data background  Experience in learning from indicator based (regional) comparison  Consistent management system

 Monitoring tools are in many cases not adjusted for differences in

composition of clients and starting conditions

 Outcome vs. impact

 Ongoing processes to improve policy effectiveness and efficiency

 Training effectiveness  Application of profiling  Policies for the long-term unemployed  …

 Austria could serve as a good example for Europe