P I S A 1 8 2 0 Joy Pangilinan Bureau of Education Assessment - - PowerPoint PPT Presentation

Capacity development workshop “Conceptualization, Measurement and Use of Contextual Data”

10-13 September, 2018, Penang, Malaysia

Philippines’ Participation in the Programme for International Student Assessment (PISA) 2018

Joy Pangilinan Bureau of Education Assessment Department of Education Philippines

P

2

I S A

1 8

2

SEA-PLM 2019

To measure the learning

• utcomes of primary

school children (Grade 5 learners in particular, regardless of age)

TIMSS 2019

To provide information that will assist countries to monitor and evaluate the success of their mathematics and science education across time and across grades

PISA 2018

To evaluate education systems worldwide by testing the skills and knowledge of 15-year

• ld students, who are

approaching the end of their compulsory education

Philippines’ Participation in ILSA P

2

I S A

1 8

Philippines Participation in ILSAs: Legal Bases

• 1. Republic Act

10533 or Enhanced Basic Education Act of 2013 (DepEd Order

• No. 43, s. 2013)

3

RA 10533, Section 15

• PH’s

Commitment to International Benchmarks

4

Philippines Participation in ILSAs: Legal Bases

• 2. DepEd Order No. 29,
• s. 2017 (Policy

Guidelines on System Assessment in the K to 12 Basic Education Program)

5

Objectives - DO 29, s. 2017

To establish baseline given the implementation of the new curriculum; To measure effectiveness of curriculum implementation (internal, national assessments) through international benchmarking and comparability; To provide evidence-based program management at systems level (planning, budgeting, monitoring and evaluation) with assessment data as critical inputs; and To establish variability within the system both across regions, across curriculum area, across year levels to be used as critical input to overall K to 12 monitoring and evaluation system.

6

Philippines

Name of the assessment programme

PISA 2018

Scope International Coverage: Sample-based Grade level: Secondary (Grade 7 or higher) Students aged 15.3 to 16.2 years old (age- specific) Domains

• Reading Literacy
• Mathematical Literacy
• Scientific Literacy
• Global Competence

Contextual/background questionnaire Student Questionnaire School Questionnaire

7

• 81 participating countries
• 11 new countries including Philippines
• Mode of Administration:
• 1. Paper-Based Assessment (PBA) – 9 countries
• 2. Computer-Based Assessment (CBA) – 72 countries
• Student Delivery System (SDS) - the software that

delivers the PISA assessments and questionnaires to students in schools (CBA-scheme); saved in USB flash drives.

8

PISA 2018

PISA Focus

Assessment Year

2000 2003 2006 2009 2012 2015 2018

Subject Assessed Reading Math Science Reading Math Science Problem Solving Reading Math Science Reading Math Science Reading Math Science Creative Problem Solving Financial Literacy Reading Math Science Collabora- tive Problem Solving Financial Literacy Reading Math Science Global Competence Subject focus on: Reading Math Science Reading Math Science Reading

Source: “Large-Scale Assessments for use in the Philippines”, Assessment Research Centre, University of Melbourne, 2015. 9

PISA Coverage- Main Survey

10

Vestibulum congue tempus

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor.

*Reading Literacy- major domain in PISA 2018

10

Technical assistance from the Assessment, Curriculum, Technology and Research Centre (ACTRC)

Field Trial

• Adaptation of cognitive instruments and questionnaires (student

and school)

• Monitoring of the field trial
• Coding (cognitive and occupation)
• Analysis of country data

Main Survey

• Analysis of final sampling data
• Advocacy campaign
• Monitoring of the main survey
• Coding (cognitive and occupation)
• Analysis of country data
• Report writing

11

PISA 2018 Field Trial

March 6-20, 2017

• Computer lab as testing room
• AVR as testing room
• Library as testing room

12

Sampling

• 28 sample schools
• 81 students per school
• 8 Administrative Regions
• 27 Schools Divisions
• No. of Examinees (Target): 2,268
• No. of Examinees (Actual): 2,162
• Assessment Rate (Actual): 95%

PISA 2018 Advocacy Campaign

January 2018

13

PISA 2018 Main Survey Training

February 2018

14

15

Sample Size

• 16 Admin.

Regions

• 108 Divisions
• 187 schools
• 7,269 students

Mode of Assessment

• CBA
• PHL

procured 5,000 pieces

• f USB flash

drives

PISA 2018 Main Survey

March 5-April 5, 2018

PISA 2018 Main Survey:

First Computer-Based Assessment in Basic Education

16

PISA 2018 Main Survey:

First Computer-Based Assessment in Basic Education

17

PISA 2018 Main Survey:

First Computer-Based Assessment in Basic Education

18

PISA 2018 Main Survey Coding and Data Management

April-May 2018

19

PISA Contractor’s Initial Analysis of PHL’s MS Sampling Data

• 95.29% assessment rate

(exceeded the 85% national target)

• 100% unweighted school response

rate

20

Main Survey Sample Schedule

Time Activity Room Availability 7:00 – 8:00 AM Computer room set-up 60 mins. (approx.) 8:00 – 8:15 AM Student log-ons, passwords, and introducing the assessment 15 mins. (approx.) 8:15 – 8:20 AM General Introduction 5 mins. (approx.) 8:20 – 9:20 AM COGNITIVE ASSESSMENT 60 mins. (EXACTLY) 9:20 – 9:25 AM Short break Generally, no more than 5 mins. 9:25 – 9:30 AM Introducing Section 2 of test 5 mins. (approx.) 9:30 – 10:30 AM COGNITIVE ASSESSMENT 60 mins. (EXACTLY) 10:30 – 10:45 AM Break 15 mins. 10:45 – 11:20 AM The Student Questionnaire 35 mins. (approx.) 11:20 – 11:25 AM Ending the session, collection, and transfer of data 5 mins. (approx.) 11:25 AM – 12:05 NN Packing up and resetting room 30-40 mins. (approx.) Total 5 to 6 hours (approx.)

Section 1 Section 2 Session 2 Session 1

21

Student Questionnaire (STQ)

• The Student Questionnaire

examines students’ attitudes toward learning and their life experiences.

22

PISA process in developing a valid and reliable background questionnaire

• Adaptation, Verification, Reconciliation and Finalization
• Applies to:
• Cognitive Instruments
• Reading Literacy | Mathematical Literacy| Scientific

Literacy

• Test Adaptation Spreadsheet
• Open Language Tool (for editing XLIFF files)
• Questionnaires
• School and Student
• Questionnaire Adaptation Spreadsheet
• Questionnaire Authoring Tool (QAT)

23

Workflow: Field Trial to Main Survey

Adaptation Verification and Verification Review Implementation and Verification Feedback Final Check and Harmonization Check Final Check Feedback Questionnaire Update and Review

24

Adaptation

• National adaptations are intentional

deviation from the international source version made for cultural reasons or to conform to local usage or to the local situations.

• An adaptation is needed if it is necessary

to ask the same question and collect comparable data.

25

26

On Contextual Questionnaires

• Choose any one or two constructs available in

your contextual/background questionnaire (for example, SES) and explain any issues or difficulty in developing a measure of the chosen construct. In constructing a measure for a construct, how do you ensure that the indicators/items suitable in your local context?

27

PHL Constructs

UNIQUE IN THE COUNTRY

• Language often used at home

CONFUSING TO SOME STUDENTS DURING THE FIELD TRIAL

• Highest educational attainment of

the mother/father

28

What language do you speak at home most of the time?

1 English 11 Maguindanaon 2 Akeanon/Akianon 12 Maranao 3 Bikol 13 Pangasinan/Pangasinense 4 Chavacano 14 Sambal 5 Hiligaynon 15 Sinugbuanong Binisaya/Cebuano 6 Ibanag/Ybanag 16 Surigaonon 7 Ilokano/Iloko 17 Tagalog 8 Ivatan 18 Tausug 9 Kapampangan 19 Waray 10 Kinaray-a 20 Yakan

29

PHL’s Current Educational System

30

ISCED Equivalent of 10 Years of Basic Education in PHL (Parents’ Education)

ISCED Approved by the PISA Contractor ISCED 3A

Programmes at Level 3 designed to provide direct access to ISCED 5A (Master’s Degree) – N/A to PHL Merged High School graduate

ISCED 3B

Programmes at Level 3 designed to provide direct access to ISCED 5B (Bachelor’s Degree)

ISCED 3C

Programmes at Level 3 designed to prepare students for direct entry into the labour market, although they also provide access to ISCED 4 programmes or other ISCED 3

• programmes. Upper secondary apprenticeship

programmes would fall into this category unless the programme was primarily designed to prepare students to enter ISCED 5. College undergraduate (did not finish Bachelor’s degree course, High school with technical-vocational courses)

ISCED 2

Junior High School High school undergraduate

ISCED 1

Elementary Elementary graduate

31

ISCED LEVEL DESCRIPTION (as agreed in STQ) REMARKS 1 Elementary Composed of 6 years compulsory (Elementary) education, Grades 1-6 2 Junior High School (JHS) Composed of 4 years (Junior High School) compulsory education, Grades 7-10 3B College Undergraduate Could be 1 to 3 years of Tertiary Education, Not completing or receiving diploma 3C High School with Tech Voc Courses Technical Vocational courses have varying hours of completion depending on the course taken except when taken in Senior High School where students who opted to take the Technical Vocational Track are expected to complete the course for 2 years 3A High School Graduate Composed of 6 years (High School) compulsory education, includes 4 years of JHS and 2 years of Senior High School 4 Technical Vocational Degree High School Graduates or any individual with Bachelor's Degree and higher may take Technical Vocational Diploma courses which usually takes 3 years to complete 5B Bachelor's Degree 4 years of Tertiary Education ( with additional year for some degrees like Engineering) 5A Master's Degree 2 Years of Post- Baccalaureate studies 6 Doctor's Degree 2-3 years of Doctoral studies

32

PISA STQ ST005 and ST007 Highest Educational Attainment

33

National Center

• As observed in the FT, students had confusion

with ST005/ST007 because there were students whose parents were college graduates.

• There was no applicable option for these

students to choose from.

• Some students were observed to have skipped

this item while others ticked High School Graduate as a more plausible response.

• PHL suggested to add College Graduate. This

additional option is needed to validate their response in the succeeding item, ST006/ST008.

34

International Contractor

• We noticed that for ST005 and ST007 you propose to

enter a new category “College Graduate“.

• We would need to know how to harmonize this

category into the international ISCED levels used in these questions.

• Usually, college would probably be assigned a

tertiary level of education, e.g. ISCED 5B. This is being assessed in the questions ST006 and ST008.

• Thus we would propose to keep ST005 and ST007 as

they are focusing on secondary education.

35

Additional Instruction for Students

36

ST006 and ST008 Educational Qualification

37

ST006 and ST008 ISCED Equivalent

PhD/EdD ISCED 6 Master’s Degree ISCED 5A Bachelor’s Degree ISCED 5B Technical-Vocational Degree ISCED 4

38

Declined

• The request for an additional option

(tertiary level) was declined because it is not within the scope of the construct that is measured in the item.

39

PHL PISA MS Country Data

• Review the adaptation /

international comparability of PHL’s

• ld educational system (parents)
• Check the student responses in

ST005 and ST007

• Recognize the limitation in adding

another national option in an international item

40

danilyn.pangilinan@deped.gov.ph Bureau of Education Assessment Department of Education Philippines

41