C HAIRPERSON G OVERNOR S TATE OF H AWAII S TATE P UBLIC C HARTER S - - PDF document

1

DAVID Y. IGE GOVERNOR JOHN S. S. KIM CHAIRPERSON

STATE OF HAWAII STATE PUBLIC CHARTER SCHOOL COMMISSION (ʻAHA KULA HOʻĀMANA)

1111 Bishop Street, Suite 516, Honolulu, Hawaii 96813 Tel: 808-586-3775 Fax: 808-586-3776

PRESENTATION SUBMITTAL

DATE OF SUBMITTAL: August 8, 2018 DATE OF MEETING: August 9, 2018 TO: John Kim, Chairperson State Public Charter School Commission FROM: Sione Thompson, Executive Director State Public Charter School Commission AGENDA ITEM:

• V. Presentation on Value Added Measures Pilot and Guidance Materials

I. DESCRIPTION Presentation on the State Public Charter School Commission’s (Commission’s) pilot of its recently developed Value Added Measure guidance materials. II. AUTHORITY Pursuant to Hawaii Revised Statutes (“HRS”) §302D-16, “The performance provisions within the charter contract shall be based on a performance framework that clearly sets forth the academic, financial, organizational, and operational performance indicators, measures, and metrics that will guide the authorizer's evaluations of each public charter school.” Section 5.1 of the State Public School Charter Contract (Charter Contract) provides that “[t]he School's academic, organizational, and financial performance under this Charter Contract shall be evaluated using the Academic, Organizational, and Financial Performance Frameworks, respectively, attached as Exhibit B to this Charter Contract. The specific terms, forms, and requirements of the Performance Frameworks, including any required indicators, measures, metrics, and targets, are maintained and disseminated by the Commission and shall be binding

2

• n the School. Material changes to the Performance Frameworks shall require approval by the

Commission.” III. BACKGROUND The Academic Performance Framework As described in the current Charter Contract, the Commission’s Academic Performance Framework (APF) contains two sections:

• 1. Student Academic Outcomes
• 2. Value Added

The first section consists of a variety of measures that focus on student outcomes — both those that are required by Hawaii Revised Statutes and drawn from the Hawaii Department of Education’s Strive HI Performance System (Strive HI) and other optional measures of a school’s choosing — and corresponding performance targets for each year of the contract, which are developed by charter schools in consultation with Commission staff. The required measures include student academic proficiency and growth, achievement gaps in proficiency between high needs and non-high needs students, and various college and career readiness measures. The second section contains Value Added Measures (VAMs), whose goal is to capture the work that a charter school is doing to impact student academic performance through mission-aligned

• initiatives. They are used to assess a school’s effectiveness in fulfilling its mission and achieving

the desired results of its educational program, and they focus on the unique aspects of a school’s model that may not be captured by the Student Academic Outcomes portion of the APF

• r Strive HI.

History of Value Added Measures Last month, VAMs celebrated their first birthday. They were introduced in the current version

• f the APF and went into effect in July 2017 as a part of the Charter Contracts that the

Commission executed with nearly all of its portfolio that summer. Although VAMs are a new feature, the APF has included school-selected, mission-aligned measures since it was first implemented in school year 2013-2014. The earlier iteration of VAMs were called “School-Specific Measures,” or “SSMs,” and were very similar to VAMs. The primary difference between the two is that SSMs were an optional component of the APF and only appeared in the APFs of two charter schools; VAMs, on the other hand, are a required component of the APF and, therefore, appear in the APFs of all charter schools. Guidance on Value Added Measures During Summer 2017, Commission staff met with charter school leaders to introduce the concept of VAMs and to engage in “guided discussions” regarding potential topics; however, no

3 formal guidance was provided to schools regarding the Commission’s expectations for VAM content, proposals, or reporting, save the brief description of VAMs that appears in the Charter Contract: “The second section of the Academic Performance Framework captures the work that schools are doing to add value to their school community or the education system at large. Each school must create a minimum of one Value Added goal. This section can measure the implementation

• f systems designed to increase program effectiveness, innovative practices and those that are

aligned to the school’s mission and vision. These goals will be specifically articulated and measurable and will include implementation timelines.”1 In an effort to fill this gap and to augment the information presented in the Charter Contract about VAMs, as well as to respond to prior feedback from charter schools regarding the challenges that they experienced due to a lack of clarity around the Commission’s expectations for School-Specific Measures, Commission staff engaged with external consultants in Spring and Summer 2018 to develop guidance materials for schools about VAMs, specifically:

• 1. Value Added Measure General Guidance
• 2. Value Added Measure Proposal Template
• 3. Instructions for Value Added Measure Proposal Template

Drafts of these documents are included with this submittal as Exhibits 1 through 3. IV. INFORMATION FOR CONSIDERATION About the Pilot Because the goal of these guidance materials is to provide charter schools with support structures for the development of VAMs and a clearer understanding of the Commission’s expectations regarding VAM content, proposals, and reporting, the only way to truly assess their efficacy is to ask charter schools to try out the materials and to share their feedback. Thus, a small group of charter schools has been invited to pilot test the guidance materials this school year. This pilot is designed to serve multiple purposes:

• To assess the efficacy of the newly developed VAM guidance materials and whether

they meet the needs of charter schools.

• To provide the Commission with feedback that will inform the refinement and

improvement of the VAM guidance materials and support structures, specifically:

1 See Exhibit B: Performance Frameworks (Academic, Financial, and Organizational) of the State Public Charter

School Contract.

4

• VAM proposal and contract amendment review and approval process
• VAM implementation process
• VAM reporting to the Commission
• Rollout of VAM support structures to all Hawaii charter schools
• To act in accordance with the Strategic Authorizing Vision articulated in the

Commission’s Strategic Plan, in particular to “Actualize a Learning Organization and System” by acquiring new knowledge (through the pilot) and then making adjustments to practices and processes to reflect what has been learned. Eleven charter schools have been invited to participate in the pilot. These schools were selected because they all have one-year VAMs that end this summer, which means that they must develop new VAMs for school year 2018-2019 and beyond and request the Commission’s approval to add these new measures to their Charter Contracts. Timeline and Next Steps August & September 2018: Charter schools develop VAM proposals. October 2018: Commission staff and charter schools present VAM proposals and contract amendment requests to the Commission’s Performance & Accountability Committee and the full Commission for review and decision-making. Throughout school year 2018-2019:

• Charter schools collect VAM data.
• Commission staff gathers feedback about the VAM guidance materials and support

structures. June 2019:

• Charter schools submit school year 2018-2019 VAM data and reports to the Commission.
• Commission staff revise the VAM guidance materials and support structures per the

feedback gathered during the pilot. For the purposes of the pilot, the draft document “Value Added Measure General Guidance” (Exhibit 1) will serve as an agreement between the Commission and schools regarding what is a high-quality VAM proposal. Charter schools will use this document to guide the development of their proposals and the Commission will use it as touchstone for their review of the proposals.

5 Exhibit 1 Draft VAM General Guidance

Value Added Measures

General Guidance - DRAFT FOR PILOT

Hawaii State Public Charter School Commission

1

What are Value Added Measures?

Value-added measures (VAMs) are specific, measurable goals that capture the work that schools are doing to impact student academic performance through mission-aligned initiatives. They should:

• focus on the unique aspects of a school’s educational program that may not be captured by the

Student Academic Outcomes portion of the Commission’s Academic Performance Framework and

• assess a school’s effectiveness in achieving the desired results of its educational program.

Progress on a VAM will be assessed based on evidence submitted by the school and in accordance with the performance targets developed by the school and articulated in the VAM section of the charter school contract. As a component of the Commission’s Academic Performance Framework, VAMs will be part of the Commission’s annual performance reports and will contribute to contract renewal decision- making.

How do VAMs highlight my school’s unique focus?

VAMs specifically center around student performance in a school’s unique focus area. For example, if a school’s focus is Hawaiian language immersion, the VAM may be articulated to address whether students are successfully acquiring and applying Hawaiian language as a result of the immersion model. Student performance on a language assessment can then be tracked and submitted to demonstrate progress on the VAM. Here are some other examples: School Focus Possible Question for the VAM to Answer Possible Evidence Increasing students’ love of learning Are students with low motivation improving their attitudes toward learning? AIMSWEB Motivation Behavior Scale or Tripod Student Survey Civic engagement Do students who participate in service learning projects exhibit increased civic responsibility? Civic Responsibility Level 2 Survey Tool Serving economically disadvantaged students Are our economically disadvantaged students achieving adequate growth in mathematics (or reading, or language usage)? NWEA / SBA

Value Added Measures

General Guidance - DRAFT FOR PILOT

Hawaii State Public Charter School Commission

2

How should a VAM be articulated?

Before you can write strong, measurable Value Added Measures, it is important to understand the

difference between inputs, outputs, and outcomes. Both outputs and outcomes can be measured, but

• nly outcomes are focused on the results of the activity. The level of completion of an activity, including

participation, is an output and can be quantified. However, participation alone is not a result, since it is not a measure that provides information about the performance of a student. Definition Example Inputs The resources needed to complete the activity. Videos, books, training resources, consultants/trainers, The number of professional development days or trainings given Outputs The completed activity and level of completion. The number or % of teachers who complete the training, observable changes in teacher practice Outcomes The measurable results of the activity. Measurable changes in student performance, the number or % of students whose performance changes on a specific tool or assessment.

How can I develop strong VAMs?

• 1. Consider your school goals, Student Academic Outcomes targets, mission, vision, and essential

elements of your educational program. Then, think about the following questions to help guide you in choosing what to measure:

• If you had to turn your mission into a measurable objective, what would that look like?
• Does your vision statement include any big picture student outcomes that could be

broken down into measurable goals?

• What aspects of your educational program are the most critical for maintaining your

school’s mission and vision?

• What goals have you already articulated for your school (e.g., governing board strategic

plan, CSI plan, Title I plan) that you could also leverage as your VAM?

• 2. With each VAM, strive to answer the following question:

HOW MANY of WHICH STUDENTS (e.g. all? a subgroup?) will achieve WHAT LEVEL OF PERFORMANCE as measured by WHICH ASSESSMENT TOOL to demonstrate WHAT?

• 3. Make sure the tool that you will use to gather evidence of progress is valid and reliable and that

results can be easily and accurately measured.

Value Added Measures

General Guidance - DRAFT FOR PILOT

Hawaii State Public Charter School Commission

3

What measurement tools should I use to demonstrate progress on my VAM(s)?

The following guiding questions can be used to identify strong measurement tools whose results may be used as evidence of student academic performance:

• Does this tool enable all students to demonstrate their knowledge, skills and understandings

relevant to the given VAM?

• Is the tool valid – does it measure the skills or knowledge we intend it to measure?
• Is the tool reliable – does it provide consistent results when gathered repeatedly for the same

student at a given point in his/her development? Does it provide consistent results when gathered for other students at the same point in development?

• Is there a clearly written scoring rubric that is used to capture performance on the tool and that

is consistent with the purpose of the tool? Additionally, since evidence of progress must be submitted to the Commission at least once each school year, you should consider the data tracking and analysis burden that comes with the selected measurement tool. It is important that your school has the appropriate knowledge, skills, and resources to track the results from the tool within a reasonable timeline. For example, if you select a pre- and post-survey as the measurement tool for a VAM, you will need to ensure not only that the survey features strong, objective questions, but that your staff has the time and capacity to conduct the survey, collect the responses, store the responses in a centralized location (ideally electronically), and analyze the results (outside support may have to be sought to support data analysis).

How do I develop the scale that will be used to assess progress on my VAM?

Each school will determine their own scale that defines the performance levels for their overall ratings that the Commission will apply in evaluating their progress toward a VAM. Each scale will define the essential features (outcomes) of student performance that will indicate whether a school exceeds the target, meets the target, approaches the target, and does not meet the target. Each performance level is defined as follows:

• Does not meet target: School has clearly not met its VAM.
• Approaching target: School is making good progress toward meeting its VAM.
• Meets target: School has clearly met its VAM.
• Exceeds target: School has clearly exceeded its VAM.

Once a scale has been developed, school staff should be trained on applying the scale consistently.

Value Added Measures

General Guidance - DRAFT FOR PILOT

Hawaii State Public Charter School Commission

4 Sample Scale:

VAM Statement: Students in the 8th grade and higher have individual career plans that are realistic, well-considered and well-developed for their grade levels (earning a rating of “Good” or “Excellent” according to the evaluation tool) Charter Year Calendar Year Exceeds Target Meets Target Approaching Target Does Not Meet Target 1 2018-19 More than 70% of students in the 8th grade and higher have met the target. 65% to 70% of students in the 8th grade and higher have met the target. 50% to 65% of students in the 8th grade and higher have met the target. Fewer than 50% of students in the 8th grade and higher met the target. 2 2019-20 More than 80% of students in the 8th grade and higher have met the target. 75% to 80% of students in the 8th grade and higher have met the target. 60% to 75% of students in the 8th grade and higher have met the target. Fewer than 60% of students in the 8th grade and higher met the target. 3 2020-21 More than 90% of students in the 8th grade and higher have met the target. 85% to 90% of students in the 8th grade and higher have met the target. 70% to 85% of students in the 8th grade and higher have met the target. Fewer than 70% of students in the 8th grade and higher met the target. 4 2021-22 More than 90% of students in the 8th grade and higher have met the target. 85% to 90% of students in the 8th grade and higher have met the target. 70% to 85% of students in the 8th grade and higher have met the target. Fewer than 70% of students in the 8th grade and higher met the target. 5 2022-23 More than 95% of students in the 8th grade and higher met the target. 90% to 95% of students in the 8th grade and higher met the target. 80% to 89% of students in the 8th grade and higher met the target. Fewer than 80% of students in the 8th grade and higher met the target.

What if I have questions about VAMs or the guidance materials?

Please contact Jennifer Higaki, the Commission’s Academic Performance and Data Systems Manager, at jennifer.higaki@spcsc.hawaii.gov or 808.586.3882.

6 Exhibit 2 Draft VAM Proposal Template

Value Added Measures

Proposal Template - DRAFT FOR PILOT

Hawaii State Public Charter School Commission

School name:

1

Value-Added Measure Statement

• 1. Mission Statement
• 2. Value Added Measure
• 3. Rationale for VAM

Assessment and Measurement

• 4. Assessment Tools & Measures
• 5. Assessment Reliability & Scoring Consistency
• 6. Rationale for Selected Assessment Tool(s)

Data and Reporting

• 7. Baseline Data
• 8. Annual Targets

Value Added Measures

Proposal Template - DRAFT FOR PILOT

Hawaii State Public Charter School Commission

School name:

2

Contract Year School Year Exceeds Target Meets Target Approaching Target Does Not Meet Target 2 20xx-xx 3 4 5

• 9. Rationale for Targets
• 10. Data Reporting
• 11. List of Attachments
• 1. Mock-up of VAM report to Commission

7 Exhibit 3 Draft VAM Proposal Template Instructions

Value-Added Measures

Instructions for Proposal Template - DRAFT FOR PILOT

Hawaii State Public Charter School Commission

1

Value-Added Measure Statement

• 1. Mission Statement: In this space, provide the mission of your school, as stated in its current charter

school contract.

• 2. Value-Added Measure: In this space, describe the VAM and the desired student outcomes. It

should be a precise, declarative statement tied to a particular group of students (either all students at your school or a specific subgroup) and a specified timeframe or length of attendance.

• 3. Rationale for VAM: In this space, briefly explain (in about 2 to 3 sentences each) why you have

chosen to focus on this particular VAM. The rationale should articulate:

• how the VAM aligns with and supports the fulfillment of your school’s mission,
• how the VAM supports the achievement of your school’s goals,
• how your VAM addresses identified student needs, and
• a description of the student data used to identify these needs.

Assessment and Measurement

• 4. Assessment Tools & Measures: In this space, briefly describe (in about 1 to 2 sentences each):
• the tool(s) that you will use to evaluate achievement of, or progress toward, the desired

student outcomes (e.g., school-selected assessment, school-developed rubric),

• when and how often the tool(s) will be administered,
• to whom the tool(s) will be administered and the inclusion criteria for the target student

population (e.g., students who have attended the school for at least two consecutive years, students in tested grade levels, Native Hawaiian boys), and

• the definition of student success/achievement, as assessed by the tool(s) (e.g., a score

above a specified cut point or performance rating level). Note: If an existing standardized assessment tool does not adequately evaluate achievement of desired student outcomes, a school may create their own tool.

• 5. Assessment Reliability & Scoring Consistency: In this space, provide information regarding the

reliability and scoring consistency of any standardized assessments OR explain how you will demonstrate both the reliability and scoring consistency of any non-standardized assessment developed or administered by your school.

• 6. Rationale for Selected Assessment Tool(s): In this space, briefly explain (in about 2 to 3 sentences)

why the assessments you have chosen are appropriate, useful tools for measuring the performance

• f the identified students on the selected VAM.

Value-Added Measures

Instructions for Proposal Template - DRAFT FOR PILOT

Hawaii State Public Charter School Commission

2

Data and Reporting

• 7. Baseline Data: In this space, provide the baseline student achievement levels for the selected VAM

and an analysis of the data. Note: Baseline data is required for all selected assessment tools including those created by schools.

• 8. Annual Targets: In this space, describe how individual student results will be aggregated and used to

assess school-level performance (e.g., % of students who receive a rating of “Satisfactory” or higher). Use the table to indicate the performance targets for each remaining year of your school’s charter school contract. Below is the scale that the Commission will apply in evaluating a school’s attainment of, or progress toward, a particular VAM. Schools will define the essential features (outcomes) of student performance that will indicate whether they have exceeded the target, met the target, are approaching the target, or have not met the target. Each performance level is defined as follows:

• Exceeds target: School has clearly exceeded its VAM.
• Meets target: School has clearly met its VAM.
• Approaching target: School is making good progress toward meeting its VAM.
• Does not meet target: School has clearly not met its VAM.
• 9. Rationale for Targets: In this space, briefly describe (in about 2 to 3 sentences each):
• the rationale for the annual targets, and
• how the baseline data were used to set the targets.
• 10. Data Reporting: In this space, briefly describe (in about 2 to 3 sentences):
• the VAM data that will be reported to the Commission, including how it will be

aggregated/disaggregated (e.g., by grade level, by student cohort) and

• the reporting schedule/dates by which reports will be submitted to the Commission.

Note: This information should be consistent with the VAM report mock-up submitted as an attachment to the VAM proposal (see #11 below).

Additional Information

• 11. List of Attachments: In this space, list all documents submitted with the VAM proposal, such as

assessment tools, process documentation, or reference materials for standardized assessments. In the case of school-developed assessments, attachments might include an actual test that your school has developed for a particular purpose, and/or a scoring tool and instructions for evaluators. Required attachment: A mock-up of your school’s report to the Commission on the VAM data and performance level assessment (exceeds, meets, approaches, or does not meet target).