PARCCs Use of Technology in Performance Level Setting CCSSO - - PowerPoint PPT Presentation

PARCC’s Use of Technology in Performance Level Setting

CCSSO National Conference on Student Assessment June 21, 2016 Eric L. Moyer, Ph.D. Laurie Laughlin Davis, Ph.D.

PARCC Online Testing

• The PARCC tests are administered online, although

paper-and-pencil versions are available for schools that are not ready for the computer-based version.

• PARCC’s ultimate goal is the digital delivery of the PARCC

ELA and mathematics assessments on the widest variety

• f devices that will support interchangeable scores
• Technology enhanced items and online delivery of tests

require a new approach to managing standard setting processes, materials, and logistics

2

Example Items from PARCC Practice Sets

3

Example Items from PARCC Practice Sets

4

Example Items from PARCC Practice Sets

5

Example Items from PARCC Practice Sets

6

Content Research Students Policy

Evidence Based Standard Setting (EBSS)

• Integrates empirical data from systematic

research with content expert judgment and policy goals in setting performance standards for students

7

Steps in the EBSS Approach

• 1. Define outcomes of interest and policy goals
• 2. Develop research, data collection, and analysis plans
• 3. Synthesize the research results
• 4. Conduct pre-policy meeting
• 5. Conduct PLS meetings with panels
• 6. Conduct reasonableness review with post-policy panel
• 7. Continue to gather evidence in support of standards

(longitudinal research planned for 2015-2017)

8

Extended Yes/No Judgment Task (Modified Angoff)

• Panelists made judgments for each item and

performance level

– Dichotomous: “Would a borderline Level 4 student be likely* to get this item correct?” Response: ‘Yes’ or ‘No’, translated to 1 point/0 point – Polytomous: “How many points would a borderline Level 4 student likely earn on this item (or set of items)?” Response: 0 points to maximum possible points

• Points are summed to get total score

9

PARCC Uses of Technology in Performance Level Setting

• 1. Post-secondary Educators’ Judgment Study

– An entirely virtual process resembling a standard setting but with nearly 200 participants in locations across the country

• 2. Meeting Facilitator Training

– An entirely virtual process used for training 3 dozen staff members across multiple locations

• 3. Dry-Run of Performance Level Setting

– A hybrid virtual (for pre-work) and in person process conducted with approximately 20 educators

• 4. Performance Level Setting Meetings

– A hybrid virtual (for pre-work) and in person process conducted with approximately 200 educators

10

What is Moodle?

11

https://moodle.org/

What is Moodle?

• Secure access using unique login id and password
• Access to external links; view external resources and

access the secure TestNav 8 site for viewing PARCC

• nline test forms
• Data collection using online surveys for item judgments,

evaluation, panelist information, etc.

• Document storage for panelist and facilitator access
• Discussion forum for panelists discussion and

communication with facilitators for answering questions

12

Postsecondary Educators’ Judgment Study (PEJ)

Overview of PEJ Study

• Purpose:

– Minimum level of performance that would indicate students are academically ready to succeed in entry-level, college- credit bearing postsecondary courses – Results used to inform next steps in performance level setting process

• Online standard setting process – All virtual process

– Only 1 round – Extended Yes/No Modified Angoff Judgment

• Nearly 200 faculty from various institutes of higher

education

14

Use of Moodle for … PEJ Study

• Accessed training and reference materials
• Completed demographic questionnaires
• Reviewed PARCC items through link to TestNav 8
• Made judgments about PARCC items
• Posted questions to study facilitators and other

participants

• Completed an evaluation survey at the end of the

process

15

Survey Results for PEJ Study— ELA/L Grade 11

16

Statement Not Successful Partially Successful Successful Very Successful Use of the website for the study 3% 12% 54% 30% Use of online training for the study process 1% 17% 50% 32% Use of TestNav to review test items 4% 19% 48% 29% Use of online survey to collect Item judgments 3% 19% 44% 33%

Survey Results for PEJ Study— Algebra II

17

Statement Not Successful Partially Successful Successful Very Successful Use of the website for the study 1% 7% 57% 35% Use of online training for the study process 2% 11% 52% 35% Use of TestNav to review test items 0% 8% 56% 36% Use of online survey to collect Item judgments 0% 12% 51% 37%

Training of Meeting Facilitators

Training of Meeting Facilitators

• Provide in-depth training

– Live and “on-demand” training

• 10 training sessions
• General and specific topics

– Track facilitator completion and retention of training – Build facilitator familiarity with Moodle

• Participants:

– 36 content and psychometric facilitators and analysts – Multiple locations and two vendors

19

Use of Moodle for … Facilitator Training

• Training in the Moodle Site

– Access to training materials – Moodle Bootcamp

• Modelling of Moodle Site

– Demonstration of tasks – “Switch role to…”

• Practice with Moodle Site

– Facilitators “own” the site – Built into the presentation slides

20

Dry-Run of Performance Level Setting

Dry Run Meeting: Purpose and Goals

• Purpose:

– Evaluation of the process due to the innovative features of PARCC assessments – Collect panelist feedback related to strengths and weaknesses of process and methodology – Identify areas in which updates in process should be made (timing, process, materials, etc.)

• Participants:

– 2 Meetings: Grade 11 ELA/L and Algebra II – About 10 panelists per meeting

22

Use of Moodle for … Dry Run Meetings

• Access to pre-work; completed before dry run

meeting (tutorials, practice tests, demographic questionnaire)

• “Experienced the Test” through link to test items
• Collaborated with team to create borderline PLDs

through links to documents on Google drive

• Reviewed items and recorded item judgments

(practice round and 3 judgment rounds)

• Completed process evaluation surveys throughout

the process

23

Dry-Run “Lessons Learned”

Positive “Lessons”

• Moodle provided control over secure materials while also

providing a single point of sign-on for all meeting activities

• Panelist data entry of their item judgments resulted in

faster turnaround of results

• Facilitators could monitor panelists access and completion
• f tasks
• Panelists reported that they liked the use of Moodle
• Quick resolution of issues that arose during meeting

24

Technology Take-Aways From the Dry-Run

Learned “Lessons”

• Use of “conditional” logic, which hides content

within the site until prior activities are completed, resulted in some panelist frustration

– Result: conditional logic was eliminated for all tasks except for the non-disclosure agreement

• Data entry of item judgments was considered

inefficient and laborious

– Result: Item judgment process and survey were revised.

25

Original Data Entry Form

26

Revised Data Entry Form

• After you have recorded your judgments on paper

you will then transfer them to Moodle

27

Technology Take-Aways From the Dry-Run

Learned “Lessons”

• Technology requirements must be met

– Internet connectivity at meeting site is critical – Have onsite technical support from both the hotel as well as from your vendor – Moodle server capacity needs to be sufficient for the number of simultaneous users

• Not a completely online process, panelists

needed to manage multiple resources

28

Performance Level Setting Meetings

Use of Moodle for… Performance Level Setting

Participants

• Access to pre-work; completed before PLS meeting

(tutorials, practice tests, demographic questionnaire)

• “Experienced the Test” through link to test items
• Collaborated with team to create borderline PLDs through

links to documents on Google drive

• Reviewed items and recorded item judgments (practice

round and 3 judgment rounds)

• Completed process evaluation surveys at end of process

30

Use of Moodle for … Performance Level Setting

Project managers/project leads

• Monitored panelist completion of pre-work activity

prior to the onsite meeting

Facilitators

• “Modelled the Moodle” at the introduction of each

task to orient panelists to tasks and materials

• Received feedback data following each judgment

round in the facilitator section of each Moodle site

Data Analysts

• Extracted data for analysis; posted feedback data

31

32

Development and Review of Borderline PLDs

• Original PLDs stored and accessed through Moodle
• Specific Google document was assigned and accessed by

each group to review and revise borderline PLDs – Google documents allowed real-time access and editing of the same document by multiple members

• f a group

– Facilitators had access to all table group documents and created a master version for full group review

• Improved efficiency, accuracy, and access to

borderline PLDs.

Use of Google Documents

Evaluation Survey Results from PLS Meetings—Math (n=105)

33

Statement Not Adequate Somewhat Adequate Adequate More than Adequate Moodle site for accessing materials and making judgments 0% 3% 21% 76% Computers used during meeting 1% 3% 24% 72%

Evaluation Survey Results from PLS Meetings—ELA/L (n=93)

34

Statement Not Adequate Somewhat Adequate Adequate More than Adequate Moodle site for accessing materials and making judgments 0% 3% 40% 57% Computers used during meeting 0% 1% 40% 59%

35

Questions?