Standard 4: Program Impact Emerson Elliott - Special Projects, CAEP - PowerPoint PPT Presentation

Standard 4: Program Impact Emerson Elliott - Special Projects, CAEP Jennifer Carinci – Director of Research and Engagement, CAEP Washington, District of Columbia September 2017

Session Overview • the standard • what the standard is trying to do • challenges and how CAEP has addressed them • examples of evidence and ways of reviewing • illustrations from recent EPP submissions Fall 2017 | Washington, D.C. 2

Standard 4 and its Components Fall 2017 | Washington, D.C. 3

STANDARD 4: PROGRAM IMPACT • The provider demonstrates the impact of its completers on P-12 student learning and development, classroom instruction , and schools, and the satisfaction of its completers with the relevance and effectiveness of their preparation.  This standard must be met to be fully accredited.  Consider: What evidence do you have that would demonstrate graduates’ impact, effectiveness, and satisfaction? What research methodologies could you feasibly employ to gain such information? 4

4.1 Impact on P-12 Student Learning and Development • Direct measures of student learning and development for inservice teachers • Addresses diverse subjects and grades • P-12 impact or growth data from state teacher evaluations (if available) If state data are not available:  Teacher-linked student assessments from districts  Teacher conducted action research  Focus group producing qualitative data • For EPPs using qualitative data, a qualitative research method must be identified for the analyze of the data • Results must be reported

4.2 Indicators of Teaching Effectiveness The provider demonstrates, through structured and validated observation instruments AND/OR student surveys , that completers effectively apply the professional knowledge, skills, and dispositions that the preparation experiences were designed to achieve. • Multi-dimensional measures of preparation impact  State data  Work Sample  Case Study  Action Research Fall 2017 | Washington, D.C.

4.3 Satisfaction of Employers • Employer satisfaction data – EPP or State instruments  Descriptive of knowledge and skills that were developed during preparation • Surveys, focus groups, interviews, case studies, etc. • Data on employment milestones  Promotion, employment trajectory, retention Fall 2017 | Washington, D.C.

4.4 Satisfaction of Completers • Completer satisfaction data – EPP or State instruments  Descriptive of knowledge and skills that were developed during preparation • Surveys, focus groups, interviews, case studies, etc. (See CAEP Evidence Guide) – Comparison of exit surveys with in-service surveys Fall 2017 | Washington, D.C.

Purpose of Standard 4 Fall 2017 | Washington, D.C. 9

CAEP Standards Commission Rationale “...Standard 4 addresses the results of preparation at the point where they most matter —in classrooms and schools.” “...judgment (of candidate knowledge and skills) is finally dependent on the impact the completers have on-the-job with P-12 student learning and development.” Fall 2017 | Washington, D.C.

Emphasis on Results • inspired by the Baldridge Performance Excellence Program, Education Criteria for Performance Excellence • describes four complementary aspects of the results of educator preparation where it matters most — in the schools and classrooms where completers are employed • results matter – effort is not enough Fall 2017 | Washington, D.C.

Standard 4 in Context • Standards 1-3  NRC 2010 aspects of teacher preparation “likely to have the strongest effects” on outcomes • Standard 4  Lynchpin - assessment of outcomes critical to • testing inputs and • grounding data-informed improvement • STANDARD 5  Quality assurance system & organizational improvement • research advanced by the Carnegie Foundation for the Advancement of Teaching Fall 2017 | Washington, D.C. 12

Addressing the Challenges Fall 2017 | Washington, D.C. 13

Acknowledging the Challenges of Standard 4 • This is a new category of evidence  Data gathered on student learning and shared with EPPs differ from state to state – Within states –  Methodologies differ for measuring student learning • CAEP phase-In policy “Standard 4 asks the right questions” • Making the case, multiple measures • Accreditation Council Decision  CAEP Accreditation Policy pages 17, 33, and 34 Fall 2017 | Washington, D.C. 14

Guidelines for EPP self-study reports: Present Results Appropriately, in a way that Aligns with the Standard • Describe data sources, trends/patterns, and representativeness • Examine differences and similarities  Look across licensure areas; show differences across demographic categories  Identify comparison points and contextualize data • Comparisons over time; highlight any existing benchmarks; make normative comparisons to peers; show comparisons to performance standards • Discuss implications of the findings for subsequent action • Focus is on the standard’s holistic and overarching expectations Fall 2017 | Washington, D.C.

More guidelines for EPPs: Frame the Argument at the Standard Level • What points or data are provided as evidence? • Which points or data support the argument? • Which points or data are neutral? • Which points or data conflict with each other? • Draw conclusions about the extent to which the data support the standard • If appropriate, address triangulation and convergence of different forms or evidence to compensate for limitations of any one data source • Acknowledge weaknesses in data or evidence and how you are addressing weaknesses • Appropriate interpretations and conclusions are supported by data Fall 2017 | Washington, D.C.

Evidence and Sufficiency Criteria Fall 2017 | Washington, D.C. 17

EVIDENCE SUFFICIENCY: RESOURCES CONSULT: • Assessment Sufficiency Criteria  CAEP Evaluation Framework for EPP-Created Assessments • Evidence Sufficiency Criteria  CAEP Evaluation Rubric for Visitor Teams Fall 2017 | Washington, D.C. 18

EVIDENCE SUFFICIENCY: GENERAL RULES STANDARD 4 • All components of each standard are addressed • At least three cycles of data  Sequential and most recent available • Results disaggregated by licensure area, campus, delivery method • EPP-created assessments meet CAEP’s assessment sufficiency criteria • All data are from/about completers (in-service) • All phase-in requirements are met. • Site visits in Academic Year 2017-2018 can present plan along with progress data • Site visits in F18 and beyond are not eligible for phase-in • Fall 2017 | Washington, D.C. 19

POTENTIAL ISSUES: STANDARD 4 AREAS FOR IMPROVEMENT (AFIs) MAY BE CITED WHEN • Site team identifies specific gaps or inconsistencies in the coherence of the EPP’s case that it meets the standard E.g.:  Instrument Quality is Poor; EPP-created assessments used to collect Standard 4 data have significant deficiencies with respect to CAEP’s assessment evaluation framework  Phase- In Plans for one or more components do not meet CAEP’s guidelines for plans  Less than three cycles of data are provided  Less than one cycle of phase-in data collected by calendar 2018 • Interpretations of evidence are not well grounded in the provided evidence  Inaccuracies found when comparing original data to reported results  Measures are not relevant for the purpose for which they are used Fall 2017 | Washington, D.C. 20

POTENTIAL ISSUES: STANDARD 4 STIPULATIONS MAY BE CITED WHEN • No significant analysis or interpretation of evidence • Evidence not responsive to the Standard  Preservice data rather than inservice  Significant aspects of the standard not addressed by relevant measures  No efforts to document validity of evidence and/or no information on representativeness of the data  No explicit plans for the EPP to move toward stronger Standard 4 data in the future Fall 2017 | Washington, D.C. 21

POTENTIAL ISSUES: STANDARD 4 STANDARD 4 MAY BE DEEMED UNMET WHEN • Standard 4 will not be met when two or more stipulations are cited  Within a component  Across components • If required evidence is not provided for any component, a stipulation is assigned, and the Standard may or may not be met (depending on other accreditation findings) • If the standard is met — with one stipulation cited for insufficient evidence on a component, the EPP has 24 months from the decision to provide sufficient evidence to remedy the deficiency. • A document review will be conducted by a site team comprised of 2 or 3 site visitors. Fall 2017 | Washington, D.C. 22

These are examples of evidence that CAEP has suggested in Qs and As and in the Handbook When data are not available from the state , consider these options:  Teacher-linked P-12 student learning data from school districts or from individual schools  Teacher information from district or school tests  Teacher action research information on P-12 student learning, perhaps in the form of a “portfolio” of different teacher experiences and results with P -12 student learning CAEP encourages providers whose completers are employed by the same school districts to collaborate in development and conduct of such options. Fall 2017 | Washington, D.C. 23