APPLICATION OF NEW APPROACHES TO BUDGET SUPPORT EVALUATION EC & - - PowerPoint PPT Presentation

APPLICATION OF NEW APPROACHES TO BUDGET SUPPORT EVALUATION – EC & DAC SUPPORTED EVALUATIONS IN TUNISIA, MALI AND ZAMBIA THE TUNISIA CASE

Enzo Caputo (DRN) Meeting Series on Budget Support ODI Meeting 1 - September 22, 2010 Not for quote. For Internal Use during the meeting.

Remind of the 3 Step approach

2

• The approach builds on the previous experience and aims at overcoming some of

its limits, namely the attempt to find the development results at the end of a linear chain that starts with donors’ inputs.

• It aims at ensuring an adequate consideration of the historical, political and

economic context in the determination of the results,

• While identifying the specific Budget Support contribution.

Comprehensive Evaluation Framework

3

The 3 Steps

4

• Step 1 covers levels 1 to 3 of the IL in their relations between each other and the
• context. It includes: (i) a description of the inputs and their assessment in relation to

the context; (ii) an assessment of the outputs generated by the implementation of the programme; and (iii) an assessment of the Government policy changes supported and possibly induced by Budget Support (level 3).

• Step 2 covers levels 4 and 5 in relation with level 3 and the context. This is the

main innovation compared to the previous approaches. It allows an investigation of the actual processes that have occurred and factors that have played toward the achievement (or non achievement) of the intended outcomes and impacts. This section of the evaluation allows a political economy assessment and various kinds of quantitative analyses, according to the data available. It highlights the relation between the development results and the policy changes supported by the programme.

• Step 3 compares the results of the previous Steps and identifies the contribution
• f Budget Support to the outcomes and impacts, via the government policies.

The approach allows an emphasis on the results, but avoids the limits that other methods, like the RCT, have shown when addressing complex programmes.

BS 3 STEPS EVALUATION

Figure 2 - GBS/ SBS Comprehensive Evaluation Framework: the Three-step approach

GOVERNMENT POLICY & SPENDING ACTIONS (STRATEGY)

Inputs to Government Policy & Spending Actions 1.1. GBS / SBS inputs 2.1. Direct Outputs. Improvement in the relationship between external assistance and the national budget & policy process

• 3. Induced Outputs. Positive

changes in the financing and institutional framework for public spending and public policy, and in public policy management and service delivery

• 4. Outcomes. Positive

responses by beneficiaries – service users and economic actors – to Government policy management and service delivery.

• 5. Impacts.

Sustainable Growth & Poverty Reduction  Transfer of Funds to Budget  Policy dialogue and related conditionality  TA/capacity building  GBS/ SBS aligned to government policies and systems and harmonised across donors  Increased size and share of external assistance funds made available through the national budget  Increased size and share of budget available for discretionary spending  Increased predictability of external funds  Policy dialogue and conditionalities, coordinated, consistent with and conducive for government strategy  TA/capacity building coordinated, consistent with and conducive for government strategy  External assistance as a whole more harmonised & aligned 1.2. Various Government inputs 2.2. Other effects by various Government inputs  xxxx  Domestic Revenue Funding and Domestic Policy Inputs 1.3. Other external assistance programmes 2.3. Other effects by other external assistance

• xxxx

 xxxx  Improved fiscal discipline and macroeconomic management.  Strengthened PFM and procurement systems.  Improved public policies‟ design and public policy processes.  Increased funding for discretionary spending resulting in increased quantity and quality of goods and services provided by the public sector.  Enhanced allocative and

• perational efficiency of

public expenditure.  Improved budget process, including better links between government and parliament  Increased use and appreciation by the beneficiaries of the goods and services provided by the public sector.  Positive response by the general economy to the improvements in government initiatives targeted by BS arrangements.  Increased business confidence deriving from more effective macroeconomic and regulatory policies.  Improved general confidence of people and enhanced democratic accountability, particularly over the budget process.  Enhanced & sustainable economic growth  Reductions in income & non- income poverty  Empowerment & social inclusion of poor people and disadvantaged groups (including women)  Other areas, according to specific partnership frameworks and priorities (e.g. improvements in democracy, human rights, environment protection, ….)  Various features of the “entry conditions”  Overall aid framework  Existing learning processes and tools  Government capacity to implement reforms;  Extent of political commitment to reform processes  Capacity of public sector  Nature of demand for Govt services  Strength of domestic accountability  Global economic development  Asset endowments  Responses to changing incentives

EXTERNAL FACTORS, CONTEXT FEATURES AND FEED BACK PROCESSES

5

The application to Tunisia

The reference to the Tunisian case supports the present methodological discussion and should not be intended as an anticipation of the actual results, since the final evaluation reports are still being considered by the partners and will be discussed only in the next weeks.

6

Programme Year of decision Amount (millions €) EC actual Payment (millions €) Appui budgétaire Global - ABG FAS I (with the WB) 1996 100 + 75M$ 100 FAS II (with the WB and the AfDB) 1999 80 + 300M$ 80 FAS III (with the WB and the AfDB) 2001 80 + 468M$ 67,1 FAS IV (with the WB and the AfDB) 2005 78 + 267M$ 77,8 PAC (programme d'appui à la compétitivité) 2006 41 9 PAGBO (programme d'appui à la gestion budgétaire par objectifs) 2007 30 Appui budgétaire Sectoriel - ABS Programme d'appui à la réforme de l'éducation de base 2000 40 39,8 Assurance maladie 2000 40 19,7 Appui à la modernisation du secteur portuaire 2002 20 17,8 Appui à la modernisation de l'enseignement supérieur 2003 48 46,8 Appui à la réforme de l'enseignement Secondaire 2005 30 17,5 MANFORM II (appui au secteur de la formation professionnelle) 2006 30 10 TOTAL 587 485,5

The total GBS (including the contribution of the Banks) ranges from 0.4 to 1% of the GDP and from 1 to > 2 % of the public expenditure.

7

Facilités d’Ajustement Structurel (FAS) - Appui Budgétaire Global

Objectif Objectifs spécifiques / Secteurs d’intervention FAS I Stabilité macroéconomique FAS II Bâtir la zone de libre échange avec l’UE

• 1. Stabilité macro-économique
• 2. Secteurs bancaires
• 3. Désengagement de l’Etat.

FAS III Améliorer la compétitivité

• 1. Gestion de la dette extérieure et élargissement assiette fiscale
• 2. Secteur Privé, Investissement, Concurrence, Privatisation
• 3. Technologies info et communication (TIC)
• 4. Intermédiation financière et assurances

FAS IV Consolider l’amélioration de la compétitivité

• 1. Maintien d’un cadre macro-économique solide
• 2. Investissement privé
• 3. Renforcement du secteur financier : banques et assurances

Appui Budgétaire Sectoriel

Modernisation de l'enseignement supérieur - PAMES Contribuer à la croissance de l’emploi et des revenus ainsi qu’à la réalisation de la société du savoir (X Plan). Améliorer la pertinence de l’offre de l’école supérieure. Acquisition de la méthode CDMT Appui à la réforme de l'enseignement Secondaire Contribuer à l’amélioration de l’efficacité interne et externe et de l’équité du système scolaire Améliorer l’offre éducative notamment dans les domaines professionnels. Synergie public-privé Appui au secteur de la formation professionnelle - MANFORM II Contribuer à la réforme de la FP engagée par le Gouvernement Promotion de la formation professionnelle (FP) pour en faire une voie de réussite Poursuite de la mise à niveau du système de FP Développement de l’efficience du dispositif de FP

Government and Partners - Reference Group in Tunis

8

• The evaluation has started before a full agreement had been reached at DAC

level on the methodology tests. Therefore, it has regarded only the EC funded programmes.

• The strong harmonisation of EC, WB and AfDB, in the management of

GBS (including joint formulation and monitoring) -however- has required that such partners be completely on board and participate in the RG.

• The local RG has met four times in Tunis.
• The Tunisian Government has established a task force –headed by the

present Minister of Development and International Cooperation (then vice Minister) of not less than 20 heads of department in the different

• areas. The task force has met the team three times in plenary and various
• ther times in restricted meetings to discuss the outputs of the work.

Application of the methodology: Identification of EQs

9

• identification of the EQs has been one of the most delicate issues. According to

the methodology, these should be adapted to the context and should identify a limited and manageable number of areas.

• the initial proposal in the first version of the inception report identified 9 EQs as

shown in the figure below, which displays the EQs on the IL. According to this proposal, any detail or sub-question should have been addressed through the formulation of appropriate Judgement Criteria

• indeed, the steering Committee agreed on the need to have the same questions in

each test-evaluation, proposing 23 EQ.

• this large number of EQs, with many EQs insisting in the same areas and the

consequent need to remind some of the general features of the area to respond to the specific questions, made the exercise relatively heavy and redundant. We will remedy such redundancy in the synthesis phase.

10

Application of the methodology: Addressing the Step 1

11

• Though the STEP 1 does not present specific methodological

innovations, the approach -with its focus on the non linear and synergic relations- has allowed to put a particular emphasis on the relations between budget support and the global partnership between the country and the EU.

• In particular, GBS has shown to be a key instrument to strengthen

the credibility and the practical dimension of the Association Agreement and the related political dialogue, with respect to the economic and social reform.

• The AA and the political dialogue ensure a strong focus on the final

results, so that GBS may focus on the creation of a specific space for dialogue on the priorities and implementation of the reforms.

Application of the methodology: Addressing the Step 2

12

Two methods have been adopted:

• an overview of the economic policy history of the country,

including economic literature review, data analysis, interviews with high level resource persons. This method has allowed an identification of the main policy choices that have supported and driven the development of the country since the sixties, and the historical features to be associated with the present performance in terms of growth and development results.

• three combined types of statistical analyses

Application of the methodology: STEP 2 / statistical analyses

Statistical analyses have included:

• Cluster analysis on 6 countries of the region, to identify

the specific factors linked to economic and social growth in each country.

• Statistical regression on Tunisia, to identify the relation

between two dependent variables (GNP per capita growth and the Human Development index) and a number of variables related to policy and non policy factors.

• Dynamic Panel Data analysis to better catch the relation

between specific polcies over time and the “intended” results.

13

Application of the methodology: STEP 2 / statistical analyses – Cluster analysis

• A cluster analysis on six countries of the region (Tunisia, Morocco, Algeria, Egypt, Jordan,

and Syria). In each country it has been studied the correlation between 54 macroeconomic, agricultural, energy and demographic variables and two dependent variables (the rate of growth of GDP per capita and the Human Development Index of United Nations).

14

14 macro-economic variables, one related to agricolture development et one related to demography have shown a strong correlation to the two dependent variables (GNP pc and HDI), in all countries, but with significant differences between countries.

Application of the methodology: STEP 2 / statistical analyses – Regression

15

• The analysis of the determinants of the growth dynamics in Tunisia, based on a

regression of the two dependent variables ( Δ GDP pc and HDI) on six variables, including: Gross fixed capital formation, Foreign Direct Investment, Quasi Money, Export , Savings, and Chinn Index (a financial openness indicator used as a proxy of the level of liberalization).

• In the first model (Growth), the net foreign direct investment, exports and

financial development appear to contribute most to economic growth in

• Tunisia. Gross fixed capital and Savings, however, contribute only

marginally to growth.

• In the second model (HDI), the significant factors are the same, with different
• weights. The importance of capital flows from outside and export

income remains crucial.

• In both models, the variable of macroeconomic reform is clearly important and

statistically significant.

Application of the methodology: STEP 2 / statistical analyses – Dynamic Panel Data Analysis

16

• A dynamic panel data analysis to measure the impact of reforms and

structural changes at the regional level, in five countries (the same sample as in the cluster, with the exception of Syria ruled out for a problem data).

• The vectors of the GDP per capita growth rates in the 5 countries have

been regressed (as dependent variables) on economic and institutional

• reforms. The latter have been formalised in two vectors, one quantitative and
• ne qualitative, including trade openness, inflation, exchange rate, current

balance, and political, economic, and financial country risk, quality of institutions, political stability, and again the Chinn Index.

STEP 2 Synthesis

17

• .

Determinants of the economic growth and social change, according to the different analyses and the correspondences between them Historic analysis Statistical analysis 1. Innovative role of governments (women role, non confessional state) 1. Trade openness and partnership policies 2. Voluntaristic choice in favour of exports, since ’72 2. Export led strategies 3. International dialogue and partnerships having influenced key strategic choices (stabilisation and integration) 3. Economic stabilisation (cooperation with Bretton Woods Org) 4. Association Agreement with the EU and the related reforms 4. Inefficiencies dues to the pervasive presence of the state in the economy 5. Inefficiencies related to the domestic savings and investments

STEP 3

18

• On one side the STEP 2 has shown the factors that have influenced the

country performance (positive and negative) on the “intended” results. Among these factors there are some key government policies, which have been put in place during the last 15 years and support through the partnership with the EU.

• On the other side, the STEP 1 has shown where and how the Budget

support has contributed to the implementation of such government policies, as well as where and how its contribution has been weak or has failed.

• The STEP 3 is not a single evaluation question, but a final synthetic

assessment, which combines the conclusions of STEP 1 and STEP 2, with a focus on the government policies (the INDUCED OUTPUTS) of the Intervention Logic.

Advantages of the methodology: preliminary highlights

• n the new areas addressed

19

The method has allowed:

• a comprehensive and in depth appreciation of the results;
• an understanding of the factors that have contributed to the
• performance. Especially this last point is very much overlooked

in the previous evaluations (various evaluations available in the same region);

• an understanding of the relations between budget support and

the other factors in place

20

Advantages of the methodology: preliminary highlights

• n the relations between aid and the political framework

The method has allowed to catch the following:

• some key political choices, made by different governments, which -over

the years- have been at the root of the performance (STEP 2);

• the continuity and development of such policies wouldn’t have been

possible without the Association Agreement and the opportunities of economic, social and cultural integration it has provided (STEP 2).

OPPORTUNITY FRAMEWORK

• the AA needed an intensification and acceleration of the reform process

that the country wouldn’t have been able to perform without an appropriate support programme (Budget Support - STEP 1). The understandings in STEP 2 have “informed” the assessments in STEP 1. Previous assessments were partly misleading for the lack of the comprehensive view provided by this method.

21

Advantages of the methodology: overall preliminary highlights

Our idea is that the method is a good tool to understand what happens in the combination between government strategies, partnerships and aid:

• the understanding that the government policies and the opportunities provided by

the AA have determined most of the performance sheds new light on the understanding of aid and the linkages between economic and political agreements and aid programmes.

• it highlights also the importance and irreplaceable role of global agreements and

partnerships --comprehensive economic and political frameworks-- that can offer to the countries opportunities for jumping in development.

• it shows that, once the process is strongly owned by the governments, then

GBS/SBS are the best instrument to seat beside the government in the steering room of the reform processes and develop a day by day dialogue, with the possibility of providing adequate technical inputs and help the government

• vercoming the internal pressures against change.

Summary of the driving factors, the role of the AA and the support programmes

Factors that had a specific impact on the outcomes and impacts expected:

22

Government Policies Partenariat Euro- Tunisien Coopération Européenne et d’autres partenaires

• Macro-economic stabilisation and

first trade liberalisation (since 2nd half of the „80s)

• Off Shore industrialisation (idem)
• Integration in the EU Common

market and competitiveness policies (since 2nd half of „90s)

• Social inclusion policies

ASSOCIA- TION AGGRE- MENT

• Support to the consolidation of

macroeconomic stabilisation

• Support to the sectoral policies to

enhance competitiveness (trade, finances, business dev., education)

• Support to upgrading industrial

enterprises

• Support to investment in social sectors
• Support to investment in infrastructures

(BEI, others)

The position of Budget Support in relation to the political dialogue

23 UE

GOUVER- NEMENT

AUTRES PARTENAIRES

AA

Dialogue politique et sur les politiques COMITES

Appui budgétaire Appui budgétaire Projets Divers Projets Divers

Objectifs et Résultats Stratégiques

Mise en oeuvre réformes macro et sectorielles

Réalisations spécifiques processus de réforme Dialogue spécifique sur mise en oeuvre réformes

POSITION DE L’APPUI BUDGETAIRE DANS LE CONTEXTE DE L’ACCORD D’ASSOCIATION

Budget Support

Last slide

24

THANK YOU