What will be covered in the next 90 minutes? Point 1 The highlights - - PowerPoint PPT Presentation

Washington, District of Columbia September 2017

STANDARD 5: PROVIDER QUALITY,

CONTINUOUS IMPROVEMENT, AND CAPACITY IN INITIAL- LICENSURE AND ADVANCED-LEVEL PROGRAMS

CAEP PRESENTERS: GINA BURKHARDT, Vice President EMERSON J. ELLIOTT, Special projects

Fall 2017 | Washington, D.C.

Point 1—The highlights of Standard 5—PPT #3 Point 2—What Standard 5 is about and why is it part of CAEP standards—PPT #4 Point 3—Question prompts for the EPP self-study report and Standard 5—PPT #5-11 Point 4—Suggestions from the Carnegie Foundation for the Advancement of Teaching—PPT #13-15 Point 5—Potential issues for AFIs, stipulations and standard not met—PPT #16- 18 Point 6—Standard 5 and a culture of evidence—PPT # 19-23

2

What will be covered in the next 90 minutes?

Fall 2017 | Washington, D.C.

• 1. The provider maintains a quality assurance system [component 5.1]
• 2. comprised of valid data from multiple measures,[components 5.2 and 5.4]
• including evidence of candidates’ and completers’ positive impact on P-12 student

learning and development. [from components 3.5 and 4.1]

• 3. The provider supports continuous improvement [components 5.3, 5.4, for

which evidence is required, and 5.5] that is:

• sustained and evidence –based, and that
• evaluates the effectiveness of its completers.
• The provider uses the results of inquiry and data collection:
• to establish priorities,
• enhance program elements and capacity, and test innovations
• to improve completers’ impact on P-12 student learning and development.

3

The highlights of Standard 5:

Fall 2017 | Washington, D.C.

• It is about EPP capability to bring together useful evidence relevant to

preparation and its results

• It is about useful evidence (that is, information that can help you)
• And it is about effective use of that capability in answering faculty questions

and continuously improve

• Note that Standard 5 is written at the EPP level—It comprehends both initial-

licensure and advanced-level programs

4

What Standard 5 is about

Fall 2017 | Washington, D.C.

• Quality assurance and continuous improvement are means by which an

EPP can manage its responsibilities effectively

• Accreditation prioritizes continuous improvement—it is not just to focus on a

point in time or once each seven years

5

Why it is a CAEP Standard

Guiding Questions

Guiding Questions

Fall 2017 | Washington, D.C.

• How well is the quality assurance system working for the EPP and how do

you know? [component 5.1].

• Is it able to answer faculty questions about the adequacy of candidate

preparation in particular areas (e.g., common core state standards, use of data to monitor student progress, creating assessments appropriate for different instructional purposes)?

• What modifications has the faculty identified and carried out to change or

increase the capabilities of its quality assurance system?

8

• Question prompts for the EPP self-study-report and

Standard 5: THE QUALITY ASSURANCE SYSTEM

Fall 2017 | Washington, D.C.

• What strengths and weaknesses in the quality assurance system do faculty

find when they use data and analyses from the system? [component 5.2].

• Are the data “relevant, verifiable, representative, cumulative and

actionable?

• Can findings be triangulated with multiple data so they can be confirmed
• r found conflicting?
• What investigations into the quality of evidence and the validity of their

interpretations does the EPP conduct?

9

• Question prompts for the EPP self-study report and

Standard 5: DATA IN THE QUALITY ASSURANCE SYSTEM

Fall 2017 | Washington, D.C.

• What is the evidence that the EPP has improved programs in its continuous

improvement efforts? [component 5.3]

• How have perspectives of faculty and other EPP stakeholders been

modified by sharing and reflecting on data from the quality assurance system? [component 5.5]

• What “innovations” or purposeful changes has the EPP investigated and

what were the results? [component 5.3]

10

• Question prompts for the EPP self-study report and

Standard 5: USE OF DATA FOR CONTINUOUS IMPROVEMENT

Fall 2017 | Washington, D.C.

• What has the provider learned from reviewing its annual outcome measures
• ver the past three years? These are the measures in component 5.4 (initial-

licensure, reported to CAEP annually) and A.5.4 (advanced-level, not in annual report to CAEP):

• Licensure rate
• Completion rate
• Employment rate
• Consumer information such as places of employment and initial compensation (including student loan

default rates)

11

• Question prompts for the EPP self-study and Standard

5: OUTCOME MEASURES

Fall 2017 | Washington, D.C.

• As you examine the outcomes you currently achieve (i.e., data on the first

four standards) and identify gaps between current results and established standards, why is it that these results continue to occur?

• How do you understand the problem(s) you need to solve?
• Based on your systematic problem analysis, what is your working theory of

improvement?

12

Suggestions from Carnegie Foundation for the Advancement of Teaching on improvement research

Fall 2017 | Washington, D.C.

• How has this working theory been tested?
• More generally, as you cycle through your processes of continuous

improvement (iteratively refining your theories based on the results of the changes made) what are you learning about your instructional system

• Remember we often learn most from our failures. So, if relevant, what

perhaps might you have tried, found evidence that it did not work as you intended and what did you learn from this about what to try next?

13

More from Carnegie

Fall 2017 | Washington, D.C.

• A. The capabilities of the EPP’s quality assurance system
• B. Their confidence in EPP efforts to ensure data quality

14

What CAEP site teams will learn about Standard 5 while they review Standards 1-4

Fall 2017 | Washington, D.C.

• C. The EPP regularly analyzes and uses data (evidence is required

for effective continuous improvement in component 5.3)

• D. The EPP reports on the CAEP annual indicators of “impact” and

“outcomes” (evidence is required for outcome measures in component 5.4 and “impact” measures in Standard 4)

• E. Appropriate stakeholders are regularly involved in decision-making

15

In addition, site teams will need to find evidence in Standard 5 that:

Fall 2017 | Washington, D.C.

AREAS FOR IMPROVEMENT (AFIs) may be cited when:

• There are deficiencies in capabilities of the quality assurance system
• Data quality is not documented or equal to EPP claims
• EPP lacks efforts at continuous improvement
• Failures in EPP reporting on the 8 annual indicators
• Limited evidence that stakeholders are involved

16

POTENTIAL ISSUES: STANDARD 5--AFIs

Fall 2017 | Washington, D.C.

• STIPULATIONS may be cited when:
• There is limited evidence of a functioning quality assurance system
• There is limited evidence that data are regularly used as a basis for

continuous improvement (a required component)

• EPP use of the 8 annual measures is deficient (a required component)

17

POTENTIAL ISSUES: STANDARD 5--Stipulations

Fall 2017 | Washington, D.C.

• The Accreditation Council decides if AFIs or stipulations will be cited and

whether standards are met or unmet

• Standard 5 will not be met when two or more stipulations are cited
• Within a component
• Across components
• If required evidence is not provided for components 5.3 or 5.4, a stipulation

is assigned, and the standard may or may not be met (depending on other accreditation findings)

• If the standard is met—with one stipulation cited for insufficient evidence on

components 5.3 or 5.4, the EPP has 24 months from the decision to provide sufficient evidence to remedy the deficiency.

• A document review will be conducted by a site team comprised of 2 or 3

site visitors.

18

POTENTIAL ISSUES: STANDARD 5—Standard not met

Fall 2017 | Washington, D.C.

• The big picture—what CAEP means by a “culture of evidence”
• “A habit of using evidence in assessment, decision making, planning, resource

allocation and other institutional processes that is embedded in and characteristic of an institution’s actions and practices.”

• Western Association of Schools and colleges (2013)
• See CAEP Evidence Guide, pp. 5-7

http://caepnet.org/~/media/Files/caep/knowledge-center/caep-evidence- guide.pdf?la=en

19

A culture of evidence

Fall 2017 | Washington, D.C.

• Evidence is NOT
• Something that an EPP does for the accreditor
• A “compliance” mechanism
• Data are NOT
• An end in themselves
• “the answer” for accreditation
• WITHIN the EPP, DATA ARE THE MEANS TO SUPPORT A CONTINUING

EXAMINATION OF WHAT YOU DO

• FOR ACCREDITATION, DATA ARE THE BASIS TO BEGIN A CONVERSATION

20

More on a culture of evidence

Fall 2017 | Washington, D.C.

• Evidence comes from multiple sources
• Validity of evidence is systematically examined
• Data are USED by the EPP for purposes of continuous improvement

21

More on a culture of evidence

Fall 2017 | Washington, D.C.

• Evidence is intentional and purposeful
• Evidence always entails interpretation and reflection
• Good evidence is integrated and holistic
• What counts as evidence can be both quantitative and qualitative
• Good evidence can be either direct or indirect

22

Essential properties of evidence in a culture of evidence

Fall 2017 | Washington, D.C.

• FOR AN EPP:
• DATA ARE THE MEANS TO SUPPORT A CONTINUING EXAMINATION OF WHAT YOU DO
• FOR ACCREDITATION:
• DATA PROVIDE THE BASIS TO BEGIN A CONVERSATION

23

A take away for Standard 5