SLIDE 1
Use of Technology in Achievement Levels Setting for NAEP Technology - - PowerPoint PPT Presentation
Use of Technology in Achievement Levels Setting for NAEP Technology - - PowerPoint PPT Presentation
Use of Technology in Achievement Levels Setting for NAEP Technology and Engineering Literacy at Grade 8 CCSSO National Conference on Student Assessment June 21, 2016 Steve Fitzpatrick, Ph.D. Definition of Technology and Engineering Literacy
SLIDE 2
SLIDE 3
TEL Assessment Areas and Practices
SLIDE 4
TEL Assessment Design
The TEL assessment is wholly computer-administered and includes scenario-based and stand-alone items There are three types of scenario-based assessment tasks: long (30 minutes), medium (20 minutes), and short (10 minutes). These scenarios incorporate animations, audio, and video components as part of the TEL items. There are interactive and static stand-alone items.
SLIDE 5
Technology and Society in TEL
SLIDE 6
Information and Communication Technology in TEL
SLIDE 7
Design and Systems in TEL
SLIDE 8
Interactive Scenario-Based Tasks
Iguana Home http://www.nationsreportcard.gov/tel_2014/#tasks/iguana Bike Lanes http://www.nationsreportcard.gov/tel_2014/#tasks/bikelanes
SLIDE 9
Use of Computers in the ALS Process
The interactive nature of the scenarios required that panelists have access to the assessment items in the “live” format in
- rder to make judgments about the items during the item
review process.
Dual Computer Arrangement
- One computer was used to present TEL assessment items to
panelists so they could accurately judge the cognitive demand imposed on the student by the items
- A second computer was used to complete the ALS activities
including presentation of the OIB and collection of panelists’ judgments
SLIDE 10
Dual Computer Arrangement
ALS NAEP NAEP
- Take the NAEP TEL
- Interact with the items
and scenario-based tasks for purposes of reviewing items
SLIDE 11
Dual Computer Arrangement
NAEP ALS
- Descriptions of knowledge and skills
needed to correctly answer each item
- Cut score recommendations for each
level at each round
- Reviewing feedback data
- Responding to questionnaires and
evaluations
- Selecting exemplar items
SLIDE 12
TEL ALS Methodology
- Item mapping methodology
- “Modified Bookmark” approach
- Adaptation of procedures used for previous NAEP ALS in 2005
Mathematics (grade 12), 2006 Economics (grade 12), 2009 Science (grades 4, 8, and 12)
- Conducted using computers with digital materials
SLIDE 13
Studies during the ALS Process
Meeting Primary Purpose Dates Venue Dual Computer Usability Study To test the logistics involved in using two laptop computers December 2-4, 2014 Chandler, AZ Initial Pilot Study To implement the process designed for the operational meeting and evaluate the need for change(s) March 16-19, 2015 San Antonio, TX Second Pilot Study To test implementation of modifications based on initial pilot study findings June 1-5, 2015 San Antonio, TX Operational ALS Meeting To implement achievement levels-setting procedures to develop recommendations for consideration of the Governing Board September 28 – October 2, 2015 San Antonio, TX
SLIDE 14
Dual Computer Usability Study - Procedures
Five eighth grade science teachers from the Phoenix area participated in the study December 2-5, 2014 All had previous experience in setting performance standards
- n Arizona tests
Each teacher competed a four-hour session
– Took a form of the NAEP-TEL assessment – Reviewed the knowledge and skills required by the items – Reviewed the Achievement Level Descriptions and developed
borderline performance descriptions
– Completed two rounds of standard setting ratings
Participants were video taped working through the activities
SLIDE 15
Dual Computer Usability Study - Findings
- Participants were comfortable using the ALS and NAEP
computers together and understood the purpose of each
- Two participants noted that wheels on the chairs facilitated
moving between the two computers
- Two participants said that mastering the navigation tabs and
terminology was difficult
- Directed practice in using the computers should be included
during the Pilot Study and ALS meetings
SLIDE 16
Initial Pilot Study - Procedures
Used an Item Mapping Procedure based on the Bookmark Procedure 14 panelists participated Activities took place on two computers
- NAEP computer containing test forms, scenarios, and items
- ALS computer using web-based standard setting software
including digital ordered item books
The meeting occurred over a period of three and a half days
SLIDE 17
Initial Pilot Study - Procedures
- Took a form of the TEL assessment on the NAEP computer as
a student would
- Received training on the NAEP TEL Frameworks and
Achievement Level Descriptions
- Reviewed TEL scenarios and items using the NAEP computer
- Recorded the knowledge and skills assessed by each item
using the ALS computer
- Performed three Rounds of standard setting ratings using the
ALS computer
- Recommended Exemplar items for each of the performance
levels
- Provided feedback throughout the process by responding to
questionnaires
SLIDE 18
Initial Pilot Study – Modifications to Procedures
Shorter days – 8:30 to 4:30 Scheduled breaks – 15 min. mid-morning, 30 min. in afternoon More time allotted for the Item Review activity The meeting extended to 4.5 days Re-conceptualized the use of digital technology
- Digital Ordered Item List in Excel
- Web based bookmark submission and questionnaires
- Digital transfer of feedback using secure FTP
SLIDE 19
Second Pilot Study - Procedures
Used an Item Mapping Procedure based on the Bookmark Procedure 29 panelists participated Activities took place on two computers
- NAEP computer containing test forms, scenarios, and items
- ALS computer using Excel tools, online questionnaires, and FTP
site
The meeting occurred over a period of four and a half days
SLIDE 20
Second Pilot Study - Outcomes
Participants successfully completed all planned activities The use of EXCEL as the tool for presenting the ordered item list (OIL) worked well Bookmarks and survey responses were collected electronically Feedback to panelists was provided electronically
SLIDE 21
Operational ALS Meeting
September 28 - October 2, 2015 (San Antonio, TX) Item Mapping Procedure based on the Bookmark Method 31 panelists participated Activities took place on two computers
- NAEP computer containing test forms, scenarios, and items
- ALS computer containing digital ordered item lists
The meeting occurred over a period of four and a half days
SLIDE 22
Digital Tools Used During the ALS Meeting
Excel Workbook – A shortcut on the desktop
- Review the test
- Item Review
- Ordered Item List
- Interactive Consequences Chart
- Exemplar item ratings
Survey Monkey – Links in a PDF file
- Entering bookmark selections
- Questionnaires throughout the process
FTP site – A shortcut on the desktop
- Round feedback
SLIDE 23
The ALS Computer Desktop
SLIDE 24
Excel Workbook
- Review the test
- Item Review
- Ordered Item List
- Interactive Consequences Chart
- Exemplar tem ratings
Excel Workbook Interactive Consequences Chart Exemplar Item Rating
SLIDE 25
Bookmarks and Questionnaires
Survey Monkey
- Questionnaires throughout the process
- Entering bookmark selections
Links to Questionnaires
SLIDE 26
Questionnaires
SLIDE 27
Questionnaires
SLIDE 28
Entering Bookmark Selections
SLIDE 29
FTP site to transfer feedback
FTP Feedback Site
SLIDE 30
Selected Questionnaire Results
Question # Orientation Topic ALS Average Rating End of Day 1 16 How to use the NAEP computer to review items 4.55 End of Day 1 17 How to use the ALS computer to review items and record comments 4.42 End of Day 1 19 Comfortable using the two computers together 4.65 End of Day 2 3 Working through the Item Review tab in the Excel file during the Item Review 4.35 End of Day 2 4 Successfully navigate through the scenarios on the two computers. 4.65 Post Round 1 4 How to select and record bookmarks 3.90 Post Round 2 4 How to select bookmarks 4.58 Post Round 3 4 How to select bookmarks 4.77
SLIDE 31