Integrating national items into questionnaires of international large - - PowerPoint PPT Presentation

Capacity development workshop “Conceptualization, Measurement and Use of Contextual Data”

10‐13 September, 2018, Penang, Malaysia

Integrating national items into questionnaires of international large‐scale studies: The case of Iran

• Dr. Masoud Kabiri

1

WHAT ARE TIMSS AND PIRLS ASK IN THEIR BACKGROUND QUESTIONNAIRES?

2

Objectives of designing background questionnaires

• Gathering information about major factors

affecting on students achievement

• Accessing to priori information to better

estimate the students achievement: estimation Plausible Values in Bayes theorem

3

The leading Questions

• What should students learn?
• Who provides the instruction?
• How is the instruction organized?
• Where does instruction take place?
• When does instruction take place?
• What have students learned?

4

Model of Potential Educational Experiences (Chen & Hsieh, 2008)

What are students expected to learn? Who delivers the instructions? How is instruction

• rganized?

What have students learned? System School Classroom Student

National/Regional curriculum goals

(intended curriculum) School goals Teachers’ content goals (implemented curriculum) Official Teacher Certification qualifications Teacher professional

• rganization &

environment Teacher characteristics a.Background b.Subject matter

• rientation

c.Pedagogical beliefs System characteristics

1.Tracking 2.Grade level 3.Content decisions 4.Related characteristics

School course

• ffering &

instructional support functions Instructional activities Implemented curriculum

Student characteristics 1.background 2.Household economic capital 3.Household cultural capital 4.Attitude 5.Activities 6.expectations

TIMSS test

• utcomes

Attained curriculum

5

Conceptual framework of determinants of student outcomes (Nilsen, Gustafsson, and Blömeke)

6

Various aspects of items in BQ in TIMSS

• Types of background questionnaires in TIMSS

and PIRLS:

– Student questionnaire – Home questionnaire (grade 4 only) – Teacher questionnaire (for grade 8 math and science questionnaires, seperatly) – School questionnaire

7

8

Student Background questionnaire

• Contains the responses of the students

– Attitude towards the subject measured – Personal/academic backgrounds – School experiences – Career/educational goals – Social‐economic statues (SES)

9

Teacher Background questionnaire

• Contains responses of the teachers

– Background and education – Instructional practices – Academic climate of school – Resources for Teaching – Homework

10

School Background questionnaire

• Contains the responses of the school

principals

– School Enrollment and Characteristics – School climate – Availability of resources

11

Home Background questionnaire

• Contains the responses of the parents of

tested students to home background

– Early literacy activities – Home educational resources – Language in the home – Out‐of‐school (literacy) activities

WHAT DID WE WANT TO ADD INTO TIMSS/PIRLS QUESTIONNAIRES?

12

Problems in TIMSS and PIRLS questionnaires

• International questionnaires collect

comprehensive, but:

– It collected to explain leaning of students, but not necessarily policy making about learning – Amount of asked information is limited relatively to other questionnaires (especially in school questionnaire) – Some of useful information were not covered

13

Reason for importance of adding national items

• Gathering information in nationally

representative samples

• Linking to students achievement
• Obtaining baseline to record change (in the

case od repeating the items)

• No more expense to gather data

14

Challenges of adding items (Limitations)

• Should not change the context of questionnaires
• Should not be so many that respondents being

tired and refuse to response either whole or a part of questionnaire

• Should not enter meanwhile of items, they

should add at the end of questionnaire

• Should be presented like the international items

(in content and layout)

• Adapting the codebooks and NAFs in order to

integrate new questions

15

How selecting the items

• Analyzing the TIMSS/PIRLS background

questionnaire to find gap informations

• Interview with some policymakers
• Review background questionnaire of major

large‐scale assessment: PISA, TALIS, NAEP, TEDS, NALS, ALL, ICCS, ICILS, PIAAC

16

Sample items

• School:

– Size of school – Enrolment policy – Administrative activities – School authorities

• Teacher:

– Recruitment mode – Professional development – Assessment and evaluation in school – Attitudes to textbooks

• Parent:

– Educational costs for children – Cooperation with school – Spent time with children

• Student

– Working with calculator

17