Learning Performance and good practice from RCEs Robert J. Didham, - - PowerPoint PPT Presentation

ESD Monitoring and Evaluation: Identifying the key factors of ESD Learning Performance and good practice from RCEs

Robert J. Didham, Paul Ofei-Manu & Akira Ogihara

1

Agenda

1) Brief review of main points for conceptual framework from ISAP expert group meeting. 2) Structure of Evaluation Framework 3) Detailed Target Areas of Evaluation Framework 4) Country Selection Criteria & Selected Countries 5) Research Plan/Schedule and expected outputs (update) 6) Survey & Questions for first round of research 7) Good Practice Reporting Framework for use with RCE representatives 8) Next Steps

2

Examples (General) Source of Information Ease of Collecting Quality of Information

INPUTS

 Funding amount  Disciplines integrating ESD curriculum  Available teaching materials  Number of ESD Trained Teachers National Government; available from Ministries of Educations’ statistics Easiest Least Beneficial; limited ability to evaluate quality of ESD

THROUGH PUTS

 Number of students receiving ESD  Variety of ESD programs  Hours of ESD teaching School-Level or local/school board- level; likely reported by principals and teachers Medium Medium; still mainly quantity assessment

• f ESD, but some

quality factors can be implied

OUTPUTS

 ESD Knowledge Gain  Student Learning  Behaviour Change Performance testing of students Hardest Most Beneficial for quality assessment of ESD

FIGURE 1: Common Division of Measuring Approaches for Educational Evaluation

3

Baseline Indicators

(Institutional Frameworks & Resources)

Knowledge & Leadership Indicators Learning Indicators

Input Indicators looking at if appropriate policy and curriculum mandates exist. Is ESD linked with other mandates for SD and SCP? Are appropriate resources directed towards ESD implementation? Throughput Indicators looking at the knowledge framing and structuring ESD implementation. Does the appropriate knowledge, expertise and leadership go into the system? Is the use of this knowledge done in a holistic and systemic manner? Output Indicators looking at the learning achievements from ESD and its quality. What is the overall quality and performance of the ESD being implemented? What impact is ESD having on the learners? Do mandates for ESD clearly exist? Is education based on good knowledge & training? Are learning outcomes being achieved? Are the necessary resources made available? How well are teachers trained in ESD? Are learners gaining new learning methodologies? Are SD principles applied to whole school management? Are good teaching materials available? Achieving Five Pillars of Learning? Is the education system sustainable and resilient? Are core ESD subjects addressed;

• ie. climate change, indigenous

knowledge, DRR & SCP? Are learners shifting behaviours to be contributors in achieving sustainable societies?

FIGURE 4: Types of Indicators and relevant information/topics

4

Monitoring and Evaluation of Education for Sustainable Development ESD Indicators: Comparability & Replicability Target User on M&E Findings: National Governments and Policy Makers

• Especially Ministries of Education & Ministries of Environment

National Curriculums Formal Education Non-Formal Education Teacher Training Private Sector & Civil Society Capacity Assessment Targets

• Institutional

Capacities/Frameworks

• Knowledge & Leadership
• Resource Capacities
• Accountability

Thematic Topics:

• Climate Change Education
• Disaster Risk Reduction
• Sustainable Consumption &

Production / Education for Sustainable Consumption

• Indigenous Knowledge

FIGURE 3: Systems Map of M&E of ESD Focal Areas

Partnerships

5

Goals in Research Process towards ESD Indicators

• Demonstrate conceptual framework for developing and structuring indicators so

process could be replicated to create other sets of indicators.

• Identify common leverage points (or capacities) and barriers in ESD

implementation which can be used in M&E process to show substantial movement.

• Develop system that provides for visual comparability between countries’ ESD

implementation status.

• Develop indicators for an M&E process that allows for easy replication in data

collection and analysis.

• Develop M&E process that can identify clear policy opportunities and

recommendations for improving ESD implementation. (consider what policy makers want to know)

• Identify indicators relevant for different sources of information, ie. national

government, school-level, and performance testing of students.

• Attempt to demonstrate causality in between systems inputs and outputs, ideally

provided in a cost-benefit format.

6

Structure of Evaluation Framework

Coverage based on 6 sectors and 3 levels of reporting (indicators) Sectors: National Curriculum (main agent: national government), Formal Education (main agent: school boards, school administration & teachers), Teacher Training (main agent: teacher education institutes), Non-Formal Education (main agent: national and local governments, continuing education systems), Community & Civil Society (main agent: NGOs and civic participation, also role of media), Private Sectors (main agent: businesses and corporations, professional organisations) Breakdown of Indicator Levels Input Capacities (for Status Indicators):

• Institutional Arrangements (including Streamlined Process, Clear definition of Roles/Responsibilities, Merit-

based Appraisal mechanism, Coordination mechanism)

• Policy Mandates
• Resource Capacities (include financial, material, infrastructure and human resources)

Throughput Capacities (for Facilitative Indicators):

• Leadership (including Vision, Communication Standards, Management Tools, Outreach Mechanisms)
• Knowledge (including Research Supply & Demand, Brain Gain and Retention, Knowledge Sharing)
• Pedagogies and Methodologies

Output Capacities (for Effect Indicators):

• Accountability (include audit systems and practice standards, participatory planning mechanism, stakeholder

feedback mechanism, monitoring & evaluation process, and systems learning cycles)

• Learning Outcomes/Performance
• Value and Behaviour Change

7

Evaluation Framework – base format

8

Developing Target Areas of the Evaluation Framework

• Based on information collected in the literature review,

leverage points and barriers were reflected on for each area. However, it is expected that these will be clarified during the research process.

• Important Target Areas that need to be addressed by the M&E

process were detailed.

• A general guideline was prepared for the amount of target

areas under each category. For the primary sectors (NC, FE, and TT): 5 x status indicators, 3 x facilitative indicators, and 2 x effect indicators. For the secondary sectors (NF, CS, PS): 3 x status indicators, 2 x facilitative indicators, and no effect indicators.

See attached A3 table: Evaluation Framework and Target Areas for full details

9

Country Selection Criteria & Selected Countries

Country Selection Criteria

• Geographic Location: from Northeast and Southeast Asia
• Recent Active Involvement with UNESCO-BKK’s M&E process

(including cooperation with clear M&E focal point or National Commission)

• Has at least one Regional Centre of Expertise (RCE) on ESD

Northeast Asia:

 Japan  China  Republic of Korea

Southeast Asia:

 Indonesia  Malaysia  Philippines  Viet Nam  Thailand  Cambodia?

(depends on consideration of recent participation in criteria 2)

10

Country Statistics on Education Systems

SCHOOL AGE POPULATION1

(thousands)

SCHOOL LIFE EXPECTANCY2

(years)

Girls Boys

NET ENROLMENT RATION3

(% of respective school-aged children)

Primary Secondary

GENDER PARITY INDEX4

Primary Secondary

PUPIL-TEACHER RATIO5

Primary Secondary

ADULT LITERACY RATE6

(% of popolutation age 15 and above)

PUBLIC EXPENDITURE ON EDUCATION7

% of GDP of Government Expenditure

PUBLIC EXPENDITURE PER PUPIL8

(2005 PPP Dollars)

Primary Secondary

Northeast (NE) Asia Countries China 274,169 11.4 11.4 98.0% n/a n/a n/a 17.7 16.4 93.3% n/a n/a n/a 633.77 Japan 17,057 14.8 15.1 99.8% 98.2% n/a 1.00 18.5 12.2 99.0%* 3.5% 9.5% 6,989.45 7,052.42 Republic of Korea 8,379 15.7 18.0 97.6% 96.9% 0.94 0.94 25.6 18.1 97.9%** 4.4% 15.3% 4,793.62 5,966.53 Southeast (SE) Asia Countries Cambodia 4,966 9.2 10.4 89.4% 34.1% 0.96 0.88 50.9 28.9 76.3% 1.6% 12.4% 98.56 112.64 Indonesia 62,293 12.2 12.5 94.8% 67.5% 0.96 1.01 18.8 13.0 92.0% 2.5% 17.5% n/a n/a Malaysia 8,104 13.1 12.4 99.9% 68.7% 1.00 1.10 16.9 17.0 91.9% 4.6% 25.2% 1,839.46 2,667.22 Philippines 30,990 12.1 11.5 91.3% 61.3% 1.02 1.20 33.7 35.1 93.4% 2.5% 15.2% 278.98 295.20 Thailand 14,825 14.5 13.4 93.9% 76.1% n/a 1.12 17.7 21.0 94.1% 3.9% 20.9% 982.56 1,082.24 Viet Nam 23,080 9.9 10.7 94.5% 61.0% n/a n/a 20.4 21.8 90.3% 5.3%*** n/a n/a n/a Notes:

General: All figues in this table are based information from UN ESCAP (2010) Statistical Yearbook for Asia and the Pacific 2009. 1 Calculated from ESCAP (2010) Statistical Yearbook 2009 figures [Total Population] x [Proportion of Children as percentage of total poulation]; figures come from 2008; However, children are calculated as those aged 0-14, so there is some minor error in this as the ideal calculation would be from 4-16. 2 Figures come from 2007; except for Malaysia from 2005, Philippines from 2006, and Viet Nam from 2000. 3 Figures come from 2007; except for China from 1991, Korea for primary from 2005, Malaysia from 2005, and Viet Nam from 2000. 4 Figures come from 2007; except for Korea for primary from 2000, Malaysia for primary from 2006 and for secondary from 2005. 5 Figures come from 2007; except for Malaysia from 2006. 6 Figures come from 2007; except for Indonesia from 2006, and Viet Nam from 1999. 7 Figures come from 2007; except for Japan from 2006, Malaysia from 2006 and 2004, and Philippines from 2005. 8 Calculated from ESCAP (2010) Statistical Yearbook 2009 figures [GDP per capita (2005 PPP dollars)] x [Public Expenditure per Pupil as percentage of GDP per capita]; there is a minor error in this calculation as GDP per capita is calculated from 2008 statistic, while public expenditure per pupil is calculated from earlier statistics. Public expenditure per pupil for China is from 1999, for Japan from 2005, for Korea from 2005, for Cambodia from 2004 and 2001, for Malaysia from 2004, for Philippines from 2005, and for Thailand from 2004. * Japanese Literacy rate from the CIA (2011) The World Factbook; https://www.cia.gov/library/publications/the-world-factbook/ ** Korean Literacy rate from the CIA (2011) The World Factbook; ibid. *** VietNam Expenditure on Education from from the CIA (2011) The World Factbook; ibid.

11

Country Statistics on Development & Society

Human Development Index1

Score Global Rank

HDI Development Status2 HDI Education Index3

Score Global Rank

Political Engagement4

(% of people who voiced

• pinion to public officials)

Satisfaction with Freedom of Choice5

(total % satisfied)

Democracy Index6

Score Global Rank

Political Participation category of Democracy Index7

Score

Ecological Footprint8

(hectares per capita)

Northeast (NE) Asia Countries China 0.663 89 Medium 0.851 97 n/a 70% 3.14 136 3.89 2.2 Japan 0.884 11 Very High 0.949 34 22% 70% 8.08 22 6.11 4.7 Republic of Korea 0.877 12 Very High 0.988 8 22% 55% 8.11 20 7.22 4.9 Southeast (SE) Asia Countries Cambodia 0.494 124 Medium 0.704 132 14% 93% 4.87 100 2.78 1.0 Indonesia 0.600 108 Medium 0.840 102 11% 75% 6.53 60 5.56 1.2 Malaysia 0.744 57 High 0.851 96 11% 83% 6.19 71 5.56 4.9 Philippines 0.638 97 Medium 0.888 73 24% 87% 6.12 74 5.00 1.3 Thailand 0.654 92 Medium 0.888 72 29% 84% 6.55 57 5.56 2.4 Viet Nam 0.572 113 Medium 0.810 113 16% 73% 2.94 140 3.33 1.4 Notes:

1 Total possible score out of 1.0; Data from 2010, published in UNDP's Human Development Report 2010; pp.143-7 2 Data from 2010, published in UNDP's Human Development Report 2010; pp.143-7 3 Total possible score out of 1.0; Data from 2007, published in UNDP's Human Development Report 2009; http://hdrstats.undp.org/en/indicators/93.html 4 Data from 2008, published in UNDP's Human Development Report 2010; pp.164-7 5 Data from 2009, published in UNDP's Human Development Report 2010; pp.164-7 6 Total possible score out of 10.0; classifications of political system follow scoring criteria 10-8 = full democracy, 7.99-6 = flawed democracies, 5.99-4 = hybrid regimes, 3.99-0 = authoritarian regimes; Data from 2010, published in the Economist Intelligence Unit's Democracy Index 2010 7 Total possible score out of 10.0; Data from 2010, published in the Economist Intelligence Unit's Democracy Index 2010 8 Data from 2010, published in the Global Footprint Network's Ecological Footprint Atlas 2010

12

Research Plan/Schedule

and expected outputs (update)

September 2011 October 2011

• M&E research framework (round 1)
• Meeting with UNU-IAS on research framework

November 2011

• Survey (ver. 1)
• Initiate contacts with NE Asia Partners

December 2011

• NE Asia workshop (possible location:

Yokohama)

• Proceedings of workshop (short version)

January 2012

• M&E research framework (round 2)

February 2012

• Survey (ver. 2)
• Initiate contacts with SE Asia Partners

March 2012

• SE Asia workshop (possible location: Bangkok)
• Proceedings of workshop (short version)

April 2012

• Research Report: Country ESD Status Reports;

based on findings from workshops

May 2012

• Consultant Meeting (potential) and Feedback

with NE Asia partners

June 2012

• Policy Report on M&E of ESD in East Asia and

seminar presentation

July 2012

• Regional monitoring and evaluation framework

August 2012

13

Survey & Questions on M&E of ESD

• To have National Commission member or M&E focal

person complete.

• To be utilised in the first round of research process with

NE Asian countries.

• Following this, the survey is to be refined (removing

questions that seem insignificant or impossible to answer), then second round survey to be utilised with SE Asian countries.

• A second round of revision should narrow us down to a

core indicator set.

• Currently though, there are 57 questions in total –

many of them two part questions, and the whole survey is just over 10 pages. We are slightly concerned about this length. * See attached survey form for following discussion

14

Review Criteria for Survey Questions

1) Is it clearly defined, reportable and replicable? 2) Can the data be easily obtained without professional/scientific measurements? 3) Is the indicator measurable and will the data actually express a value of some type? 4) Does it measure something useful and relevant and will people (i.e. government officers and educators) care about this? 5) Will it lead for comparability between countries? 6) Does a change in this indicator suggest a course of action?

*Note: This criteria will be more important when we are developing our final indicators, and in some cases may not fully apply to the first round of survey questions as this is a wide scoping process.

15

Good Practice Reporting Framework

(for first and second research round)

• This framework is to be used with the RCE members

to collect a good practice case on ESD from each RCE (or each country)

• The purpose of this two-fold:
• 1. To identify success factor, strengths and weaknesses in

regards to implementing effective ESD initiatives

• 2. To be able to provide some story/qualitative material

to the country ESD status report.

* See attached good practice reporting framework for following discussion

16

Next Steps

1) Identify contacts for each country to be M&E Focal point with this research project (in cooperation with UNESCO). 2) Identify contacts for RCEs in each country (in cooperation with UNU-IAS). 3) Initiate contact with NE Asian contacts and disseminate surveys. 4) Preparations for NE Asia workshop. 5) Collect survey and analyse findings. 6) Prepare workshop proceedings. 7) Assess and refine evaluation framework and survey, then initiate process with SE Asian countries.

17

Thank You

for you time!

Robert J. Didham, Paul Ofei-Manu & Akira Ogihara

18