Experimental Evidence from Ghanas Civil Service Experimental - - PowerPoint PPT Presentation

Experimental Evidence from Ghana’s Civil Service

Experimental Evidence from Ghana’s Civil Service Imran Rasul (UCL and IFS), Daniel Rogger (World Bank and IFS), Martin J. Williams (Oxford) [AEA registered with pre-analysis plan] December 2019

State capability as a driver of development

◮ From a macro perspective: ◮ Large share of aggregate economics activity [20% of GDP in LICs] ◮ Key driver of growth, welfare and equality (Acemoglu and Robinson; Besley and Persson; etc.) ◮ Affects productivity of international assistance

State capability as a driver of development

◮ From a macro perspective: ◮ Large share of aggregate economics activity [20% of GDP in LICs] ◮ Key driver of growth, welfare and equality (Acemoglu and Robinson; Besley and Persson; etc.) ◮ Affects productivity of international assistance ◮ From a micro perspective: ◮ Presumption behind many program evaluations is that successful ones can be scaled-up and delivered by government (Bold et al., 2018; Romero et al., 2017; Vivalt, 2017, Meager 2019)

State capability as a driver of development

◮ From a macro perspective: ◮ Large share of aggregate economics activity [20% of GDP in LICs] ◮ Key driver of growth, welfare and equality (Acemoglu and Robinson; Besley and Persson; etc.) ◮ Affects productivity of international assistance ◮ From a micro perspective: ◮ Presumption behind many program evaluations is that successful ones can be scaled-up and delivered by government (Bold et al., 2018; Romero et al., 2017; Vivalt, 2017, Meager 2019) ◮ As such, this paper investigates alternative approaches to designing capacity building programmes in the public sector

Bureaucrat’s human capital matters

◮ Tasks of public administration characterized by ambiguity and uncertainty (Dixit, 2002; Rasul et al., 2019) ◮ Many public goods have elements of incomplete contracting ◮ Public officials are ‘residual claimant’ on shocks in society

Bureaucrat’s human capital matters

◮ Tasks of public administration characterized by ambiguity and uncertainty (Dixit, 2002; Rasul et al., 2019) ◮ Many public goods have elements of incomplete contracting ◮ Public officials are ‘residual claimant’ on shocks in society ◮ Implies human capital is fundamental in the production of public services ◮ Public officials must ‘fill in the gaps’

Bureaucrat’s human capital matters

◮ Tasks of public administration characterized by ambiguity and uncertainty (Dixit, 2002; Rasul et al., 2019) ◮ Many public goods have elements of incomplete contracting ◮ Public officials are ‘residual claimant’ on shocks in society ◮ Implies human capital is fundamental in the production of public services ◮ Public officials must ‘fill in the gaps’ ◮ Public officials constitutional protections imply that they must be able to learn and update their skills ◮ Often seen as a non-pecuniary benefit of public service

Substantial investments in public sector capacity building

◮ Most frequent response is training

Substantial investments in public sector capacity building

◮ Most frequent response is training ◮ Public sector agencies invest in compulsory skills upgrading – Service-wide training programmes – Senior leadership courses

Substantial investments in public sector capacity building

◮ Most frequent response is training ◮ Public sector agencies invest in compulsory skills upgrading – Service-wide training programmes – Senior leadership courses ◮ International development community complements with external investments – Default component of many donor programmes – World Bank in Africa: USD7.5 billion over past decade

This paper

◮ Does civil service training improve bureaucratic

productivity?

◮ Tracks learning outcomes in standard service training ◮ Implements an improved, ‘productivity-focused’ curriculum, based in realities of service and focused on specific reforms ◮ Trials ‘embedded’ capacity building, within division-level classes

This paper

◮ Does civil service training improve bureaucratic

productivity?

◮ Tracks learning outcomes in standard service training ◮ Officials are learning: 0.7 s.d. increase in core curriculum ◮ Implements an improved, ‘productivity-focused’ curriculum, based in realities of service and focused on specific reforms ◮ Trials ‘embedded’ capacity building, within division-level classes

This paper

◮ Does civil service training improve bureaucratic

productivity?

◮ Tracks learning outcomes in standard service training ◮ Officials are learning: 0.7 s.d. increase in core curriculum ◮ Implements an improved, ‘productivity-focused’ curriculum, based in realities of service and focused on specific reforms ◮ Officials learn more in this setting: additional 0.4 s.d. increase in expanded curriculum ◮ Trials ‘embedded’ capacity building, within division-level classes

This paper

◮ Does civil service training improve bureaucratic

productivity?

◮ Tracks learning outcomes in standard service training ◮ Officials are learning: 0.7 s.d. increase in core curriculum ◮ Implements an improved, ‘productivity-focused’ curriculum, based in realities of service and focused on specific reforms ◮ Officials learn more in this setting: additional 0.4 s.d. increase in expanded curriculum ◮ Trials ‘embedded’ capacity building, within division-level classes ◮ Dynamics of hierarchy tend to mediate effects

Where does this fit in the literature?

◮ Little evidence on whether training programs effectively increase state capacity (World Bank, 2016) External training content not closely aligned with workplace realities Rajasthani Police Reform (Banerjee et al., 2018) Training can improve ‘soft skills’ of police ... but not bureaucratic processes Danish change-oriented training amongst teachers (Jakobsen et al., 2019) Training can impact individual classroom teacher practices

Where does this fit in the literature?

◮ Little evidence on whether training programs effectively increase state capacity (World Bank, 2016) External training content not closely aligned with workplace realities Rajasthani Police Reform (Banerjee et al., 2018) Training can improve ‘soft skills’ of police ... but not bureaucratic processes Danish change-oriented training amongst teachers (Jakobsen et al., 2019) Training can impact individual classroom teacher practices ◮ Training should be ‘embedded’ in work programs (HBR, 2016; OECD, 2018) Constrained by existing incentives of bureaucratic hierarchy

Welcome to Ghana

Productivity differences across and within

• rganisations

Management differences across and within

• rganisations

Management differences across and within

• rganisations

Civil Service Training Centre (CSTC)

Training in Ghana’s Civil Service

Focus on mid-level professionals in central government policymaking/oversight roles, not frontline implementers

◮ 56 orgs, ∼ 400 divisions, ∼ 3000 individuals

CSTC staff conduct routine 10-day trainings (“Scheme of Service”, SOS) throughout the year

◮ All staff below Director required to complete as prerequisite for promotion ◮ Cycle through once every 3-4 years ◮ Cohort: 10-40 other staff of same rank from across the Service ◮ We developed roughly a day’s worth of material focusing on generating productivity gains

New Training for Productivity (TFP) curriculum

Designed by international consultant with research team input

◮ Training materials based on numerous pilots across client countries ◮ Adapted and delivered by CSTC’s regular trainers ◮ Replaces standard SOS productivity curriculum

Content more applied, tangible, context-specific. Bundle of:

◮ Examples and data from Ghana’s Civil Service ◮ Diagnostic tools ◮ Motivational video ◮ Action Plan development ◮ Role-playing

TFP curriculum

• 0. STEPS DELIVERING OUTPUTS

OUTPUT ??? ??? ??? What outcomes do we want?

• ???
• ???
• ???

Which officers are involved?

• ???
• ???
• ???
• What are the steps in delivering one of your key outputs?

When finished, complete Action Plan Q1

TFP curriculum

• 11. Identifying Problems: Fishbone diagram
• Helps to structure and

then prioritise the multiple possible causes of a problem

• Make each category of

causes a “bone” of the fish, and list the possible causes in that category along it

• Use a standard set of

categories like that shown,

• r generate your own set

specific to the problem and context

13

TFP curriculum

T1: Enhanced Scheme-of-Service training (TfP)

◮ T1: 33% of civil servants trained in 2017 were randomly sorted into TfP training ◮ Designed action plan to capitalise on wider training ◮ For those in treatment, promotion interviews included discussion of action plan and follow up activities ◮ Encouraged to return to division, implement reform in action plan ◮ Potential spillovers to colleagues in terms of training and reform implementation

Action plan characteristics

.1 .2 .3 .4 Routine work process Occasional work process Resource generation Financial management Procurement Communication within org. Communication outside org. Data collection Personnel management Monitoring and oversight ICT Administrative/clerical Other

a) Main area

.1 .2 .3 .4 New innovation (minor) New innovation (major) Implement existing practice Expand existing practice Get add'l resources Meeting / discussions Other

b) Nature of change

Note: Bars show proportion of Action Plans falling in each category. See Appendix C for details of coding.

T2: Team-based training

Binding constraint may be anti-innovation team cultures, not lack of individual ideas/skills/motivation

◮ Changing team culture equilibrium may require simultaneously shifting actions of whole team

Perceived obstacles to action plan implementation

.1 .2 .3 Proportion of Action Plans

Note: Bars show proportion of Action Plans falling in each category. Proportions sum to more than one because multiple obstacles could be cited in each Action Plan. See Appendix C for details of coding.

• 1. Individual skills
• 2. Cooperation within division
• 3. Cooperation w/ other divisions in org.
• 4. Cooperation w/ other gov't orgs
• 5. Cooperation w/ non-gov't actors
• 6. Resources/logistics
• 7. Time
• 8. Other

T2: Team-based training

Binding constraint may be anti-innovation team cultures, not lack of individual ideas/skills/motivation

◮ Changing team culture equilibrium may require simultaneously shifting actions of whole team

T2: (follow-up) team-based trainings with an entire division

◮ Same content and duration as TFP T1 ◮ Divisions of 40% of individual training participants randomly selected (2-3 months after the individual training)

Project timeline

Attend 10-day individual training in 2017?

All civil servants eligible for individual 10-day training (estimated n: ~ 2500) Attend individual 10-day training in 2017 (25%) No training in 2017 Non- random

Randomized into TFP or SOS 1-day training?

All civil servants eligible for individual 10-day training (estimated n: ~ 2500) Attend individual 10-day training in 2017 (25%) No training in 2017 T1: New indiv. prod. training (50%) SOS: Standard indiv.

• prod. training (50%)

Non- random Random

Randomized into T2 team-level follow-up?

All civil servants eligible for individual 10-day training (estimated n: ~ 2500) Attend individual 10-day training in 2017 (25%) No training in 2017 T1: New indiv. prod. training (50%) SOS: Standard indiv.

• prod. training (50%)

T2: Team training (40%) No team training (60%) T2: Team training No team training T2: Team training (40%) No team training (60%) Non- random Random Random

This gives us 5 treatment cells plus 1 control

All civil servants eligible for individual 10-day training (estimated n: ~ 2500) Attend individual 10-day training in 2017 (25%) No training in 2017 T1: New indiv. prod. training (50%) SOS: Standard indiv.

• prod. training (50%)

T2: Team training (40%) No team training (60%) T2: Team training No team training T2: Team training (40%) No team training (60%) T1 + T2 T1 SOS + T2 SOS T2 Control Non- random Random Random 6 cells:

Treatment summary

Table A: Treatment Distribution - Intervention Sample No indiv. training Standard

• prod. content

T1: new

• prod. content

Sub-totals Division Level Treatment (T2) No

• 189

83 272 Yes 561 112 63 736 Sub-totals 561 301 146 1,008 Individual Level Treatment (T1)

Empirical approach

Main specification for division-level impacts: yj = α0 + α1SOSj + β1T1j + β2T2j+ β3(SOSj ∗ T2j) + β4(T1j ∗ T2j)+ δZj + ǫj

◮ yj is an outcome measure for division j (in levels/ differences) ◮ SOSj: % of division members who attended 2-week individual training (or corresponding binary) ◮ T1j: % of division members attended new indiv. productivity training (or corresponding binary) ◮ T2j: division-level follow-up training ◮ Zj are division-level characteristics

Empirical approach

Main specification for division-level impacts: yj = α0 + α1SOSj + β1T1j + β2T2j+ β3(SOSj ∗ T2j) + β4(T1j ∗ T2j)+ δZj + ǫj Comparisons of interest:

◮ β1 : impact of training content (TFP vs SOS) ◮ β2 vs β1 : impact of training cohort (T1 vs T2) ◮ β3 & β4 : complementarity/ substitutability of individual and divisional trainings

Data collection

Individual-level

◮ Pre- and post-training tests ◮ Hand-coded action plans ◮ Baseline and endline surveys of universe of relevant civil servants (n ≈ 3, 000 each) ◮ Promotion interview scores

Division-level

◮ Management quality (aggreg. from individual surveys) ◮ Work process productivity ◮ Quality of bureaucratic process (random file audits) ◮ Speed/quality of replies to internal letter (audit experiment)

Project/task-level

◮ Division-level output completion

Data collection

Individual-level

◮ Pre- and post-training tests ◮ Hand-coded action plans ◮ Baseline and endline surveys of universe of relevant civil servants (n ≈ 3, 000 each) ◮ Promotion interview scores

Division-level

◮ Management quality (aggreg. from individual surveys) ◮ Work process productivity ◮ Quality of bureaucratic process (random file audits) ◮ Speed/quality of replies to internal letter (audit experiment)

Project/task-level

◮ Division-level output completion

What are the relative learning gains in

• ur treatments?

Raw learning gains (SOS and T1)

Raw learning gains (SOS and T1)

Relative learning gains (SOS vs T1)

Heterogeneity in learning gains (T1)

◮ Among the worker characteristics we evaluate: ◮ Gender has no impact on the impact of our intervention ◮ High pay workers/managers have significantly worse learning gains ◮ Workers with higher entry grades have significantly higher learning gains ◮ We cannot reject the null that these learning effects are homogeneous across organisations with different attributes

Potential interpretation

◮ In external trainings, organisational features do not mediate learning gains for individuals, but individual characteristics are critical ◮ Learning lower cost for individuals with higher baseline aptitude

Raw learning gains (T2)

Heterogeneity in learning gains (T2)

◮ None of the worker characteristics we evaluate exhibit differential learning gains ◮ There is substantial heterogeneity across divisions in learning from team training ◮ We cannot reject the null that officials who attended TFP do not learn from team training ◮ Workers attending SOS have similar learning gains to workers not attending SOS ◮ Workers whose co-workers attended TFP have similar learning gains to those without coworkers attending TFP

Potential interpretation

◮ Learning gains embedded inside organisations are mediated by

• rganisational characteristics

◮ This does not seem to be due to spillover effects from previous training episodes

Heterogeneity in learning gains (time b/w T1 and T2)

◮ In general, TFP attendees score better than their colleagues on T2 entry scores ◮ The longer the time between T1 and T2, the higher the test scores of TFP attendees ◮ This is not true for SOS attendees ◮ At the same time, their co-workers who did not attend TFP do increasingly worse over time ◮ This is particularly true when the TFP attendee is a manager

Potential interpretation

◮ This suggests that TFP is exactly teaching officials to capitalise

• n their work experience

◮ Co-workers are then free-riding on their colleagues efforts and limiting their acquisition of costly information (Rogger and Somani, 2019)

How did these impact the quality of bureaucratic processes?

Quality of procedure

Quality of procedure

Quality of procedure

Quality of content

Quality of content

Training for productivity

◮ Evidence that external TFP training improved: ◮ Quality of bureaucratic procedure ◮ Quality of content of activities ◮ Much weaker evidence that team training improved these margins ◮ Some evidence that team training retarded the impacts of TFP training

Potential interpretation

◮ Team training creates free-rider problems for development of productivity enhancing reforms ◮ If you are the only official who is driving reform agenda, you cannot free-ride, feel empowered, and are a focal point

Did these lead to wider changes in management quality?

Quality of management

Training for more productivity

◮ The TFP trainings were not only aimed at improving individual processes ◮ They aimed at improving the way divisions functioned and were managed ◮ No evidence that trainings were able to increase the quality of management anywhere

Potential interpretation

◮ Trainings are underpowered to shift hierarchical dynamics ◮ Individual efforts to develop improved quality of management face significant resistance from bureaucratic structures ◮ Consistent with psychological attachment to hierarchy (Fast et al 2014), not rent-seeking, structural rigidity, or cultural opposition

Conclusion

• 1. Training can make public officials more productive
• 2. This paper provides evidence that ‘tweaked’ external

training is best

• 3. Dynamics of hierarchy mediate impacts of any

training exercise

Thank you

Funding: IGC, EDI, i2i Research team: Jane Adjabeng, Mohammed Abubakari, Julius Adu-Ntim, Temilola Akinrinade, Sandra Boatemaa, Eugene Ekyem, Paula Fiorini, Margherita Fornasari, Jacob Hagan-Mensah, Allan Kasapa, Kpadam Opuni, Owura Simprii-Duncan, Liah Yecalo-Tecle Advice, guidance, other inputs: Nana Agyekum-Dwamena, Dora Dei-Tumi, members of our project steering committee, Andrew Wyatt, Stefan Dercon Training, data collection, coding, facilitation: 80+ civil servants from across the service

Project timeline

T2