Fo Formative Eva valuation of New Ham Hampshir hire' e's Per - - PowerPoint PPT Presentation

Fo Formative Eva valuation of New Ham Hampshir hire' e's Per erformanc ance e Asses essmen ent t

• f
• f C

Com

• mpetency E

Education

• n (

(PACE)

Gene Wilhoit, Center for Innovation in Education Arthur Thacker, HumRRO Paul Leather, New Hampshire Dept. of Education Susan Lyons, Center for Assessment (Access the Evaluation Here)

Formative Evaluation of New Hampshire’s Performance Assessment of Competency Education (PACE)

Paul Leather, Deputy Commissioner New Hampshire Department of Education

2

NH PACE -- Performance Assessment of Competency Education

• PACE is a learning system designed to

capitalize on the latest advances in understanding how people learn. The goal is to structure learning opportunities that allow students to grapple with gaining meaningful knowledge and skills at a depth of understanding that can they can transfer to new real-world situations. Current Status:

• 3 years of waiver implementation, and
• ESSA planning to result in Waiver request,

subject to Sec. 1204 implementation by Secretary DeVos

• Our goal is to scale PACE across our state
• f New Hampshire and to inspire other

states to accept a similar challenge.

6/28/17 3

PACE – What are the basics?

• First-in-the-nation USED approved alternative assessment and

accountability system pilot

• Reduced level of standardized testing along with locally developed

common performance assessments

• Smarter Balanced/SAT given once in elementary school, once in

middle school and once in high school (three grades instead of seven)

• In all other years, administration of common and local performance

assessments developed by the districts and validated at the state level

Engaging the US Department of Education

March 27, 2012

1st Meeting with Arne Duncan

• Sept. 23, 2014

2nd Meeting with Arne

March 3, 2015

Approval!

5

PD Design

Theory

• f

Action

Implementation NH Process

What has changed with PACE?

NCLB/Waiver

• All districts held accountable at once to

state system

• State level assessment – last two in NH

are nationally developed

• Annual Determinations based on State

Assessments solely

• There often are two systems – state and

local

• Educator development tied to school

improvement after assessment results are released

PACE

• Districts must meet guard rails before

joining PACE

• PACE Assessments developed locally –

System is State AND Local

• Annual determinations based on multiple

measures

• Local communities and educators OWN

the single system of accountability

• Intensive Educator Development occurs

PRIOR to joining PACE, and continues as PACE involvement evolves

6

Assessment Continuum

Extended Performance Tasks (SCALE, EPIC, ILN) Narrow Assessment Assessments of Deeper Learning CCSS Assessments (SBAC & PARCC) Performance Based Items & Tasks (MARS, BAM) Student- Designed Projects

(Envision, NY Performance Standards Consortium, Singapore, IB)

Traditional Tests Descriptions

Standardized, multiple- choice tests of routine skills Systems of standardized performance items and tasks (1 day to 1 week) that measure key concepts in

thought- provoking items that require extended problem solving

Examples

Standardized tests with m-c & open-ended items + short (1-2 day) performance tasks of some applied skills Performance tasks that require students to formulate and carry out their

• wn inquiries,

analyze & present findings, and (sometimes) revise in response to feedback Longer, deeper investigations,(2-3 months) & exhibitions, including graduation portfolios, requiring students to initiate, design, conduct, analyze, revise, and present their work in multiple modalities

Linda Darling-Hammond

What is PACE? – Water Tower Proposal!

• The Problem: Your town’s population is predicted to

increase over the next 3 years. As one of the town planners, you are asked to address this issue in terms

• f the town’s water supply. In order to meet the

future needs of the town, you need to make a proposal to add a water tower somewhere on town property that will be capable of holding 45,000 ± 2,000 cubic feet of water. The town is looking for a water tower to contain the most amount of water while using the least amount of construction material.

• Student Task: Your job is to prepare a proposal that

can be submitted to the town planning committee. Using your calculations of surface area and volume for the two designs, describe and analyze the characteristics that lead you to a final recommendation.

6/28/17 8

Geometry PACE Common Task

PACE: Local, State and Common Assessments

Grade ELA MATH SCIENCE K-2 Local PBA Local PBA Local PBA 3 Smarter Balanced Common PACE PBA Local PBA 4 Common PACE PBA Smarter Balanced Common PACE PBA 5 Common PACE PBA Common PACE PBA Local PBA 6 Common PACE PBA Common PACE PBA Local PBA 7 Common PACE PBA Common PACE PBA Local PBA 8 Smarter Balanced Smarter Balanced Common PACE PBA 9 Common PACE PBA Common PACE PBA Common PACE PBA 10 Common PACE PBA Common PACE PBA Common PACE PBA 11 SAT SAT Common PACE PBA 12 Local PBA/Capstone Exhibitions Local PBA/Capstone Exhibitions Local PBA/Capstone Exhibitions

Who is PACE -- Which Districts/Schools?

6/28/17 10

Tier 1 -- Year 1 (2014-15) Additional Tier 1 -- Year 2 (2015-16) Tiers 2 and 3 – Year 3 (2016-17)

• Rochester
• Sanborn

Regional

• Epping
• Souhegan HS
• Concord
• Monroe
• Pittsfield
• Seacoast

Charter

• Allenstown
• Fall Mountain
• Plymouth
• SAU 23 No

Haverhill

• SAU 58 Groveton
• Manchester
• Rollinsford
• Ashland
• SAU 39 Amherst

and Mont Vernon

• VLACS

Additional Tier 1 -- Year 3 (2016-17)

• SAU 35 White

Mountains (as negotiated with USED)

Organization to PACE Scaling

Adopted from NH DOE graphic

Tier1 Implementing Districts

• Data Collection
• Calibrating and Scoring
• Body of Work Protocols

Tier 2 Preparing Districts

• Performance

Assessment Development

• Implementation Process

Tier 3 Planning Districts

• Develop CBE
• Competencies,

Instruction, Assessment and Grading

PACE System Design is based on High Performing Professional Learning Communities

• Where educators work together to continuously:
• Review competency expectations, aligned with college and

career outcomes

• Evolve instructional practice to support student learning of

competencies

• Develop strategies to enhance personalized learning where

student agency is a primary student outcome

• Improving the Assessment system to measure student

achievement and growth related to competencies

• Educators review student work on performance tasks regularly

6/28/17 12

Headquarters: 66 Canal Center Plaza, Suite 700, Alexandria, VA 22314-1578 | Phone: 703.549.3611 | www.humrro.org

Formative Evaluation of New Hampshire’s Performance Assessment of Competency Education (PACE)

Presentation for NCSA

Presenters : Arthur Thacker, HumRRO Sunny Becker, HumRRO

June 29, 2017

Presentation Overview

• What is HumRRO?
• Brief Reminder of Select PACE Characteristics
• Evaluation Design
• Key Evaluation Findings
• Evaluation Recommendations
• The Story of PACE

14

Human Resources Research Organization (HumRRO)

• Independent, non-profit U.S. based organization
• Approximately 100 employees
• Professional staff mostly education researchers, I-O psychologists,

statisticians, and measurement experts

• Expertise:
• Program evaluations at local, state, and national levels

Evaluations of school improvement initiatives Large scale, long range evaluations of state and national education programs Quality assurance for National Assessment of Educational Progress (NAEP) Independent, real-time replication of assessment results as a quality control check for several states

15

Select PACE Characteristics

• Competency education
• Performance assessments integrated into students’ day-to-day work
• Tier 1, 2, and 3 districts
• Common and local tasks
• Task bank

16

Evaluation Design ( 1 of 2)

• April 2016 – February 2017
• Formative evaluation
• Theory of Action
• Data sources
• observations of major PACE events (e.g., task development and scoring sessions);
• interviews with students, parents, teachers, principals, and district leaders;
• classroom observations;
• surveys of teachers;
• review of materials provided to and by Tier 1 districts; and
• analyses of score data.
• Accumulated evidence regarding each claim in the Theory of Action over

the course of the evaluation.

17

PA

18

PACE Theory of Action

Key Evaluation Findings (1 of 3)

• Report includes detailed findings on each claim within the Theory of

Action

• Buy-in
• Educators determine what is assessed, how it is assessed, and how the tasks

are scored

• Assessments integrated into instruction
• Replaces Smarter Balanced assessments and requires deep knowledge
• Collaboration
• Cross-district activities (task development sessions, professional

development, scoring sessions, standard-setting, etc.)

• LibGuide (online repository)

19

Key Evaluation Findings (2 of 3)

• Teaching & Learning
• Increased depth of knowledge
• “Testing to what is taught” vs. “teaching to the test”
• Learning while testing
• Context
• Previous experience with competency-based education
• District size
• Substantial work for teachers

20

Key Evaluation Findings (3 of 3)

• Continuous process improvement
• Curriculum Coordinator
• Content leads
• Buddy district
• Saturday task development session
• Providing consultants to support PACE coordinator

21

Data Analyses

22

47.3 53.5 55.5 44.2 63.3 57.2 48.9 44.6 52.0 44.0 46.0 51.0 57.0 48.0 47.0 52.0 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 Grade 3 Grade 5 Grade 6 Grade 7

Percentage of Students Meeting or Exceeding Achievement Level Benchmarks (3 or 4)

Mathematics

PACE 2014-2015 PACE 2015-2016 SB 2014-2015 SB 2015-2016

Data Analyses

23

43.6 49.7 40.1 48.6 51.3 55.3 43.1 42.4 56.0 63.0 57.0 63.0 57.0 63.0 59.0 62.0 10 20 30 40 50 60 70 Grade 4 Grade 5 Grade 6 Grade 7

Percentage of Students Meeting or Exceeding Achievement Level Benchmarks (3 or 4)

ELA

PACE 2014-2015 PACE 2015-2016 SB 2014-2015 SB 2015-2016

Data Analyses

Math 2015 Science 2016 Science 2016 .487 ELA 2016 .317 .637

24

Table 1. Correlation Table Grade 3-4

Science 2015 ELA 2015 ELA 2016 ELA 2015 .555 ELA 2016 .459 .630 Math 2016 .440 .603 .735

Table 2. Correlation Table Grade 4-5

Data Analyses

Math 2015 ELA 2015 ELA 2016 ELA 2015 .635 ELA 2016 .520 .619 Math 2016 .625 .590 .648

25

Table 3. Correlation Table Grade 5-6

Math 2015 ELA 2015 ELA 2016 ELA 2015 .470 ELA 2016 .482 .586 Math 2016 .558 .513 .531

Table 4. Correlation Table Grade 6-7

Data Analyses

Math 2015 ELA 2015 Science 2016 ELA 2016 ELA 2015 .517 Science 2016 .523 .448 ELA 2016 .557 .483 .574 Math 2016 .488 .477 .590 .541

26

Table 5. Correlation Table Grade 7-8

Recommendations (1 of 2)

Ongoing Monitoring

1. Monitor and Support District Engagement 2. Evaluate Effectiveness of Collaboration Methods

Additional training/supports

3. Consider Additional Training/Supports for Teachers Not Directly Involved in Common Task Development 4. Infuse Equity and Accommodations Training into PACE Activities

27

Recommendations (2 of 2)

Routinize program features

6. Routinize Timely Reviews of Local Performance Tasks 9. Consider Systematically Recycling Tasks

• 10. Begin Tracking Performance from Year to Year

Targeted Studies

5. Investigate the Impact of Reading/Writing Requirements on Accessibility 7. Plan for Future Research on the Impact of PACE on Teaching and Learning 8. Evaluate the Benefit of Time in Program on Outcomes

28

The Story of PACE

• Integrated competency-based education system with performance

events are a dramatic shift from traditional schooling

• Substantial progress toward this goal
• Sustainability
• Scalability
• Careful expansion, district by district

29

Programmatic Responses to PACE Formative Evaluation

Susan Lyons, Ph.D. Center for Assessment

CCSSO’s National Conference on Student Assessment

June 29, 2017

PACE is about classroom impact

Lyons • Page 31

Complex Performance Assessments

• Building educator

capacity

• Enhancing

student engagement

Transforming Instruction

• Increasing rigor
• Developing

student agency

College and Career Readiness

• Preparing students

to engage meaningfully in their post- secondary plans

Increasing College and Career Readiness

Supporting All Teachers

vConsider Additional Training/Supports for Teachers Not Directly Involved in Common Task Development

Teachers routinely reported that the process of developing the common tasks greatly improved their own task development process and their approach to

• assessment. As the program

expands, it will be important to maintain that benefit for all participants.

ü 16-17: Hiring of PACE Teacher Leaders within districts to better transmit institutional knowledge to all teachers. ü 16-17: Investment in online intranet for all PACE teachers to share key documents and resources. ü 17-18: Expanding high quality performance assessment development training within Tiers 2. ü 17-18: Developing robust set

• f common resources for

competency-based education and assessment across all three tiers of PACE.

Quality Local Assessment

vRoutinize Timely Reviews of Local Performance Tasks

It is important to ensure that the tasks and rubrics are of sufficient quality to be used to generate student scores and annual

• determinations. Teachers

report that their skill level in developing these tasks improves with each year

• f PACE participation….

Instituting a system of regular task review will help ensure that happens.

ü 16-17: The NHDOE is now reviewing one major assessment per competency for each PACE course in all of the participating districts. ü 16-17: Contract with Stanford to review all local performance assessments. ü Summer ’17: NH DOE will provide feedback to districts related to their assessments systems and targeted supports for those districts in need of additional guidance.

Evaluating Impact

vPlan for Future Research on the Impact of PACE on Teaching and Learning

The positive impacts of PACE on teaching and learning should continue to be externally verified beyond this evaluation. This may be part of a future research agenda when it becomes possible to evaluate the predictive strength of PACE results on college and career performance.

ü On-going: Annual evaluation

• f student performance on

standardized assessments. 2016 yielded some early indications of success. ü 16-17: Providing access to PACE data to internal and external researchers ü 17-18: Seeking funding from Hewlett to more deeply understand the connection between learning and engagement in complex performance assessments. ü 17-18: Begin to longitudinally track trends in career and college readiness (e.g., persistence in college).

For more information: Center for Assessment www.nciea.org

Susan Lyons slyons@nciea.org