Beyond Test-Scores: Measuring quality of education in Colombia and using it for policy making Camila Gomez Afanador (Ministerio de Educación Nacional de Colombia) Pedro Cerdán-Infantes (World Bank) April 6, 2016
Agenda This is an agenda slide 1 Why are we doing this? 2 What are we doing? 3 How are we doing it? – “ Colegios 10” (Top Schools) 3.1 Phase I – Developing a monitoring system 3.2 Phase II – How to use the information 1
1 Why are we doing it? Contextual relevance
1 Ambitious policy for a complex quality education issue This is an agenda slide Colombian National Ministry of Education Main Objective "Colombia will be the best educated country in Latin America in 2025" Increasing coverage Improving quality The enrolment rate has been increasing in recent years, however Colombia continues to fall behind in quality scores Gross enrolment rate 2014 PISA 2012 - Position 62 out of 65 countries 106.5% Primary % of students scored in the lowest level 101.4% Secondary Colombia 74% 77.3% Brazil Media Chile OECD Average Private, 23% 16.30% 0% 20% 40% 60% 80% Science Reading Math Public, 83.70 % 3 Footer Info
1 Many different efforts towards improving quality This is an agenda slide SABER Tests Synthetic Index of Measures of (Standardized Education Quality Quality state national (ISCE) exams) Initiatives and Full time schools Programa todos a (institutional) Aprender(PTA) programs (classroom) Colombia Colombia Teaching Curriculum Free of Integration Excellence Bilingual Illiteracy . . . 4 Footer Info
1 Many efforts but not many results This is an agenda slide A large number of Existing The documented initiatives, often not information in the information directed towards systems is not presents problems, the real needs of always relevant to and the actions for schools. users. It is not schools to follow Lack of coordination easy to navigate. are unclear. between stakeholders. Leading to Weak incentives for improving • Low spending efficiency • Low community engagement • 5 Footer Info
2 What are we doing? Addressing the limitations
2 What are we working on? This is an agenda slide Is a results-based monitoring system of different dimensions of quality and its systemic use for decision making Seeks to address the current limitations Better targeting Transparency in Provide relevant and better reporting and greater information and management of the access channels for analytical tools, for stakeholders community all decision makers. initiatives: participation. Clear Promote information - Ministry of Education recommendations flows between - Local Departments of Education for all stakeholders. stakeholders. - Schools - NGO and private sector 7 Footer Info
3 How are we doing it? Organizing the schedule
3 Information flows and stakeholder use it This is an agenda slide PHASE I PHASE II Developing a Monitoring System How to use the information 1. Define the 2. Pilot the 4. Pilot the Matrix of Monitoring 3. Guidelines complete Dimensions System system and Indicators - Review of existing - Getting dimensions and stakeholders, from indicators - Creating a - Pilot its selected comprehensive menu - Definition of new implementation in with clear municipalities, to indicators recommendations for 1000 schools effectively use the - Definition of schools. tools provided in the information platform. recollection 9 Footer Info
3.1 Phase I Developing a Monitoring System A systematic and comprehensive Monitoring System to improve information and transparency
3.1 Building a robust Matrix of Dimensions and Indicators This is an agenda slide Improving the coverage and quality of Education Objective PISA scores Synthetic Index SABER Tests The 6 dimensions must encompass the areas impacting the quality measures How to do it? Dimensions Concrete / Indicators actions Based on evidence of their impact on outcomes (at a domestic and ✓ Selecting imensions and international level) dimensions ✓ Through empirical association with outcomes in Colombia, if possible and indicators Pilot Testing the Measuring Ensuring collection of costs and reliability indicators impact 11
3.1 Building a robust Matrix of Dimensions and Indicators This is an agenda slide Dimensions selected by the Ministry Teacher Profile Pedagogical and Variables Continuous development of Academic teachers and school leadership team Learning communities Evaluation Family, schools Teachers and community Infrastructure Administrative Criteria for the selection of the indicators: ✓ Focus the indicators of each dimension on outcomes School environment and ✓ Process and results indicators well being ✓ Reliable and cost-effective 12
Selecting dimensions, areas and indicators based on 3.1 evidence This is an agenda slide • Look for rigorous evidence in Colombia • If it worked in Colombia, it should be included Colombian Evidence • Evidence in other countries in similar contexts • Explore empirical association in Colombia when possible International Evidence • Programs / areas that are inherently good (no dirt floors, toilets, etc.) • Programs / interventions with a coherent logical framework, even if there is no evidence “Desirable • In these cases, try to generate evidence in Colombia Interventions” • Collecting information in different ways to test cost and reliability • Being creative about data collection (using existing mechanisms) Piloting data collection
The actual system will be in an open portal, and will 3.1 include diagnostic, references and interactive futures This is an agenda slide 2. COMPARATIVE ANALYSIS* 1. LEVEL OF ANALYSIS COUNTRY Colombia School X Pedagogical al Academic DEPARTMENT 7 6 5 4 Administrative Teachers MUNICIPALITY 3 2 1 0 SCHOOL Family, Schools Infraestracture and Community 3. REPORT School Envirnment and Well Being For any stakeholder selected By Dimension Score Analysis of Recommendations and actions to be each dimension undertaken 14
3.2 Phase II How to use the information More than just information to improve coordination, information and transparency
3.2 Creating a comprehensive menu with options This is an agenda slide Relevant information Helpful tool for decision making available for user Ministry Better targeting for programs and resources Access as Local Department of Better decision making for development plans Education School Better decision making for improvement plans Active participation and accountability Community Materials available Description of Complete analysis by Clear and concise News and reports Programs school, municipality, guidelines department and nationwide
3.2 Pilot the complete system This is an agenda slide Where? What? Transparency and Financing Management 10 municipalities with accountability significant differences in the Better targeting Performance synthetic index and the Engaging the for national based transfers socioeconomic conditions community programs (at both school (mapping of the and regional existing programs level) to the selected dimensions) Improve Improve Improve coordination information transparency Better decision making 17
3.2 Management – To improve coordination This is an agenda slide Aligning government programs to the needs identified in the different dimensions Map of the association of needs and programs Program Dimension A 1 2 3 4 5 6 ✔︐ ✔︐ A ✔︐ B Program Program ✔︐ ✔︐ ✔︐ C Program C ✔︐ D B E ✔︐ ✔︐ ✔︐ ✔︐ ✔︐ Program F E ✔︐ ✔︐ G Program H Program D Possible actions after diagnosis: Adjusting the targeting of existing programs o Changing or redesigning existing programs o P rogram Program F Designing new programs in areas with limited o G existing support Positive impacts on quality Duplicity of efforts ▪ Not responding to the real needs ▪ Low efficiency of government spending ▪
3.2 Management - To improve coordination This is an agenda slide Analyzing existing programs Each existing program will be assessed using the 6 selected dimensions ➢ For each dimension the actions and results are identified ➢ Decision makers within each dimension are also identified ➢ Dimension 1 2 3 4 5 6 Actions Analysis of all Results existing programs Decision Makers
• 3.2 Management – Mapping of processes in each dimension This is an agenda slide Analyzing existing processes Map processes and stakeholder actions for each area ➢ and dimension: what needs to happen for an indicator to change? Action1 Identify good practices in processes, for each variable ➢ in the different dimensions is being considered Provide user friendly information (references, ➢ manuals) for all stakeholders to make effective decisions Action 2 ADVANTAGES DISADVANTAGES - It may not contemplate - Facilitates decision making for Action 3 differences in contexts (socio- action plans economic, cultural) - Ensures a positive impact on - Any error can cause a chain of the improvement of variables errors - Saves time and work
Recommend
More recommend
