Measuring the Effectiveness of Wisconsin Principals: Two Studies - PowerPoint PPT Presentation

Measuring the Effectiveness of Wisconsin Principals: Two Studies From the State- wide Evaluation of Educator Effectiveness Curtis Jones & Leon Gilman, University of Wisconsin – Milwaukee Steve Kimball, University of Wisconsin – Madison Katharine Rainey, Wisconsin Department of Public Instruction Presented at the annual meeting of Association for Education Finance and Policy, Portland OR, 3.13.18 1

Presentation Outline • Background of the process and characteristics of principal ratings. • Study 1 – Measuring the concurrent validity of principal ratings with teacher perceptions of principal effectiveness across a number of aspects of principal leadership. • Study 2 – Using principal ratings to detect and measure the magnitude of principal effectiveness equity gaps within Wisconsin school districts. 2

The Wisconsin Educator Effectiveness Process • In 2011, Wisconsin passed Act 166, which required the state Department of Public Instruction (DPI) to develop an Educator Effectiveness (EE) system and school districts to implement principal and teacher evaluation systems based on the state EE System (Wisconsin, 2011). • As part of the EE process, districts are required to provide ongoing, formal feedback to principals about their practice using a standard process and leadership rubric. • Principals who are new to the profession or new to a district go through the evaluation process in their first year and every third year thereafter. • At the end of the year, district administrators provide principals evaluation ratings based on leadership documents and observations of practice collected throughout the year. 3

The Wisconsin Framework for Principal Leadership • With assistance from the Wisconsin Center for Education Research (WCER), DPI developed a principal effectiveness rubric called the Wisconsin Framework for Principal Leadership (WFPL). • About two-thirds of Wisconsin districts have chosen the WFPL to inform their principal evaluation and feedback process. • Following two pilot years, Wisconsin districts began fully implementing the principal EE process during the 2014-15 school year. • This paper focuses on WFPL ratings assigned to 322 principals at the end of the 2016-17 school year, across 61 school districts, which was the first year that WFPL ratings were collected from districts across the state. 4

Principals 277 40 Recruiting and Selecting 5 0 Human Resource Leadership Assignment of Teachers and Instructional 256 16 50 0 Staff 236 42 42 Observation and Performance Evaluation 2 227 30 65 Professional Development and Learning 0 201 55 65 Distributed Leadership 1 Wisconsin Framework for Principal Leadership Ratings 229 44 48 Mission and Vision 0 Instructional Leadership 207 63 50 Student Achievement Focus 0 Unsatisfactory 216 21 83 Staff Collaboration 1 201 73 47 School-wide Use of Data 0 Basic 267 19 34 Student Learning Objectives (Teacher SLOs) 1 126 185 Professionalism 1 9 Proficient Personal Behavior 236 42 42 Time Management and Priority Setting 1 266 28 26 Distinguished Use of Feedback for Improvement 1 223 Initiative and Persistence 29 69 0 Intentional and Collaborative 217 38 63 School Climate 2 School Culture 254 27 37 Communication 2 243 21 57 Conflict Management and Resolution 1 265 19 37 Consensus Building 1 School Management 227 10 85 Learning Environment Management 0 256 23 43 Financial Management 0 282 11 29 Policy Management 0 5

WFPL ratings of principal effectiveness – descriptive statistics Internal Std. Items Mean Consistency Dev. Human Resource Leadership 5 .75 3.07 0.36 Instructional Leadership 5 .80 3.02 0.40 Personal Behavior 4 .70 3.12 0.37 Intentional and Collaborative 4 .73 3.06 0.37 School Culture School Management 3 .59 3.12 0.32 Overall WFPL Rating 5 subdomains .88 3.08 0.30 6

Distribution of Overall WFPL Ratings 7

Study 1 – Measuring the Concurrent Validity of WFPL Ratings with Teacher Perceptions of Principal Effectiveness • As part of the state-wide evaluation of EE, each year SREed conducts a state-wide survey of teachers. • In the 2016-17 evaluation, over 21,000 teachers completed the survey, which represented about 42% of all Wisconsin classroom teachers. • The survey measures a number of aspects of schools. For the current study, we focus on teacher perceptions of: • Principal Leadership and Trust (5Essential Survey; Bryk, Sebring, Allensworth, Easton, & Luppescu, 2010) • Qualifications to provide performance feedback, Usefulness of feedback, and Accuracy of feedback (The Examining Evaluator Feedback Survey; Cherasaro, Brodersen, Yanoski, Welp, & Reale, 2015) 8

Study 1 – Measuring the Concurrent Validity of WFPL Ratings with Teacher Perceptions of Principal Effectiveness • Of the 322 schools with WFPL ratings, 162 had at least 40% of classroom teachers complete a survey. • 2,873 classroom teachers completed surveys in these schools, representing a 62% response rate. • 114 (70.3%) were elementary schools, 20 (12.3 %) were middle schools, 22 (13.6%) were high schools, and 6 (3.7%) were combined elementary and secondary schools. • In 56 schools (34.6%) the principal was new (in their first three years). • 60 schools were from the Milwaukee Public Schools. Min Max Mean Std. Dev. Enrollment 42 1,903 475 254 Percent Students with Disability 5.2% 77.8% 17.5% 7.7% Percent Economic Disadvantaged Students 2.1% 99.0% 60.7% 26.5% Percent White Students 0.6% 95.2% 44.9% 33.1% 9

Principals The principal participates in instructional 1,205 386 811 519 planning with teams of teachers. The principal knows what's going on in my 1,308 385 693 531 classroom. The principal carefully tracks student 1,530 170 478 724 academic progress. The principal encourages teachers to 1,557 141 351 860 implement what they have learned in Teacher Evaluations of Principal Leadership and Trust professional development. Leadership The principal communicates a clear vision 1,385 207 471 831 for our school. Strongly Disagree The principal makes clear to the staff his or 1,475 160 451 812 her expectations for meeting instructional goals. The principal understands how children 1,489 167 395 844 learn. Disagree The principal sets high standards for student 1,459 156 323 948 learning It's OK in this school to discuss feelings, 1,260 325 495 829 Agree worries, and frustrations with the principal. The principal looks out for the personal 1,292 287 499 829 Strongly Agree welfare of the staff. 1,255 281 559 806 I trust the principal at his or her word. The principal is an effective manager who 1,252 340 680 628 makes the school run smoothly. Trust The principal places the needs of children 1,355 190 417 942 ahead of personal and political interests. The principal has confidence in the expertise 1,344 208 463 891 of staff. The principal takes a personal interest in the 1,452 185 453 819 professional development of staff. 1,265 305 596 737 Staff feel respected by the principal. 10

Principals 278 286 636 740 included specific improvement suggestions. included specific suggestions to improve my 407 362 503 660 content/subject knowledge. included specific instructional strategies that 329 320 562 716 I could use to improve my teaching. Feedback Usefulness included specific classroom management 382 359 512 671 strategies that I could use to improve my teaching. Teacher Perceptions of Performance Feedback included recommendations for finding 446 384 632 459 Disagree resources or professional development to improve my teaching. 263 313 653 698 was provided as frequently as I needed it. Somewhat Disagree was provided in time for me to use it to 245 229 742 711 inform my practice. The feedback I received was an accurate 194 194 827 715 portrayal of my teaching. Feedback Accuracy Somewhat Agree The classroom observations or walkthroughs 167 220 708 837 that informed the feedback I received represented a typical day in my classroom. In our evaluation system, different evaluators 275 274 627 740 reviewing the same evidence would likely give the same ratings. Agree Evaluator Qualifications (In my opinion, my evaluator had knowledge of my subject/content to 211 241 831 643 effectively evaluate me. knowledge of how my students learn to 178 256 805 681 effectively evaluate me. sufficient…) knowledge of effective teaching practices to 118 187 965 652 effectively evaluate me. understanding of the curriculum being 188 269 797 664 observed to effectively evaluate me. 1,061 understanding of the established teacher 105 136 628 evaluation system to effectively evaluate me. 11

Study 1 – Measuring the Concurrent Validity of WFPL Ratings with Teacher Perceptions of Principal Effectiveness Teacher ratings of principal effectiveness – descriptive statistics Internal Std. Items Mean Responses Consistency Dev. Principal Leadership 8 .94 2.93 0.70 2,873 Principal – Teacher Trust 8 .96 2.91 0.79 2,872 Feedback Quality/Usefulness 7 .95 2.78 0.92 1,942 Feedback Accuracy 3 .86 3.05 0.87 1,924 Evaluator Qualifications 5 .94 3.18 0.84 1,915 12