[PPT] - Coaching with Fidelity Kaci Fleetwood, M.Ed, BCBA, LBA Jodie PowerPoint Presentation

SLIDE 1

Coaching with Fidelity

Kaci Fleetwood, M.Ed, BCBA, LBA Jodie Soracco, M.Ed, BCBA, LBA APBS 2020 J 11

SLIDE 2

Lens of Implementation

State Level District Level Doesn’t fit in a box Building Level

SLIDE 3

Identify a Buddy

Make eye contact with someone

close to you

Tap elbows or give each other a

thumbs up to confirm you’ve noticed each other

Plan to talk to your buddy during

think, pair, share prompts

SLIDE 4

5

“There was a moment in sports when employing a coach was unimaginable – and then came a time when not doing so was

unimaginable. We care about results in

sports, and if we care half as much about results in schools and in hospitals, we may reach the same conclusion.” Atul Gawande

SLIDE 5

Why is Coaching Important?

More accurate implementation fidelity scores (McIntosh et al., 2017)
Greater improvements in outcomes (Artman-Meeker, Hermeter, & Snyder, 2014; Capella

et al., 2012; Flynn, Lissy, Alicea, Tarzates, & Mckay, 2017)

Sustainability and scale-up (Fixen & Blasé, 1993; Horner et al., 2014)

6

SLIDE 6

“You would never have a team without a coach”

Shauna Bake (Pershing County MTSS Coordinator), 2019

7

SLIDE 7

Research to Practice Gap

Research* has demonstrated that coaching is essential for

translating training into effective implementation.

However, little explicit guidance exists on how to structure the

roles and activities of external coaches who support multiple implementation sites.

*(Joyce & Showers, 2002; LaPolla & Scott, 2019; McDaniel & Bloomfield, 2019; Sugai et al., 2012)

SLIDE 8

Default Approach

Those who have been responsible for recruiting, training, and

supervising external coaches at the district, regional, or state level often resort to simply hiring candidates with the most robust coach-related experience and permit a type of “free- range coaching”.

This results in a great deal of variability in what coaching

activities are performed, how school sites are supported.

SLIDE 9

Diverse Backgrounds

10

In Nevada Coaching Positions are filled by individuals with diverse professional histories

Licensed Teachers
School Psychologists
Board Certified Behavior

Analysts

Social Workers
RN/Psychiatric Nurses
School Administrators
Others

SLIDE 10

What are the Roles of a Coach?

“External Coaches”

– Coach teams – Train content – Evaluate outcomes – Coordinate implementation – Expand LEA capacity – Other duties as assigned

SLIDE 11

SOLUTION: OUTLINE PERFORMANCE EXPECTATIONS FOR DIVERSE COACHING FUNCTIONS

SLIDE 12

Leveraging a School-wide Approach

Expectations = professional values
Locations/Contexts= job functions
Coaching Competencies= observable behaviors

SLIDE 13

Value/ Expectation Value/ Expectation Value/ Expectation

Safe Respectful Responsible What expectations/ values work for your coach(es)?

SLIDE 14

What professional values apply to your context?

SLIDE 15

Insert Nevada Coach Matrix

MTSS External Coach Expectation Matrix

Overall (O) Coaching Teams (C) Training (T)

Responsible

1. Attend meetings and coach trainings with coordinator ______ 2. Self-monitor tasks assigned and completion _____ 3. Engage in formal dissemination ______ 1. Attend site-team meetings (data-driven) ______ 2. Collect monthly artifacts from teams ______ 3. Facilitate establishment of data collection systems ______ 4. Consultation notes completed ______ 1. Ask for clarification or assistance for concepts/topics with which you are not yet fluent ______ 2. Schedule & present targeted trainings _____

Data- Driven

4. Reports and data are delivered to the state on or before due date ______ 5. Use data to allocate time and resources ______ 5. Use TFI to collect data and drive action planning ______ 6. Use various data sources for assessment, planning, and reporting ______ 3. Collect evaluations on all formal presentations ______ 4. Review & reflect on feedback from: participants, district administrators, and coordinators ______

Professional

6. Assist district to build capacity for sustained implementation (redefine your role over time) ______ 7. Read professional literature at a minimum of 30 min/week ______ 8. Engage in professional and ethical standards (see Coaches Code) ______ 7. Advocate for smallest change that will have the largest impact ______ 8. Build credibility & rapport ______ 5. Engage in Effective Spokesperson techniques while presenting ______ Bonus: Present on MTSS/PBIS at district, state, and national forums ______

Professional Values/

Expectations

SLIDE 16

Insert Nevada Coach Matrix

MTSS External Coach Expectation Matrix

Overall (O) Coaching Teams (C) Training (T)

Responsible

1. Attend meetings and coach trainings with coordinator ______ 2. Self-monitor tasks assigned and completion _____ 3. Engage in formal dissemination ______ 1. Attend site-team meetings (data-driven) ______ 2. Collect monthly artifacts from teams ______ 3. Facilitate establishment of data collection systems ______ 4. Consultation notes completed ______ 1. Ask for clarification or assistance for concepts/topics with which you are not yet fluent ______ 2. Schedule & present targeted trainings _____

Data- Driven

4. Reports and data are delivered to the state on or before due date ______ 5. Use data to allocate time and resources ______ 5. Use TFI to collect data and drive action planning ______ 6. Use various data sources for assessment, planning, and reporting ______ 3. Collect evaluations on all formal presentations ______ 4. Review & reflect on feedback from: participants, district administrators, and coordinators ______

Professional

6. Assist district to build capacity for sustained implementation (redefine your role over time) ______ 7. Read professional literature at a minimum of 30 min/week ______ 8. Engage in professional and ethical standards (see Coaches Code) ______ 7. Advocate for smallest change that will have the largest impact ______ 8. Build credibility & rapport ______ 5. Engage in Effective Spokesperson techniques while presenting ______ Bonus: Present on MTSS/PBIS at district, state, and national forums ______

Locations/Contexts delineate job functions

SLIDE 17

SLIDE 18

Insert Nevada Coach Matrix

MTSS External Coach Expectation Matrix

Overall (O) Coaching Teams (C) Training (T)

Responsible

1. Attend meetings and coach trainings with coordinator ______ 2. Self-monitor tasks assigned and completion _____ 3. Engage in formal dissemination ______ 1. Attend site-team meetings (data-driven) ______ 2. Collect monthly artifacts from teams ______ 3. Facilitate establishment of data collection systems ______ 4. Consultation notes completed ______ 1. Ask for clarification or assistance for concepts/topics with which you are not yet fluent ______ 2. Schedule & present targeted trainings _____

Data- Driven

4. Reports and data are delivered to the state on or before due date ______ 5. Use data to allocate time and resources ______ 5. Use TFI to collect data and drive action planning ______ 6. Use various data sources for assessment, planning, and reporting ______ 3. Collect evaluations on all formal presentations ______ 4. Review & reflect on feedback from: participants, district administrators, and coordinators ______

Professional

6. Assist district to build capacity for sustained implementation (redefine your role over time) ______ 7. Read professional literature at a minimum of 30 min/week ______ 8. Engage in professional and ethical standards (see Coaches Code) ______ 7. Advocate for smallest change that will have the largest impact ______ 8. Build credibility & rapport ______ 5. Engage in Effective Spokesperson techniques while presenting ______ Bonus: Present on MTSS/PBIS at district, state, and national forums ______

SLIDE 19

Continuous Improvement

Contexts updated annually as needed
Competencies revised annually based on feedback/field test
V1 (2015); v2 (2016); v3 (2017); v4 (2018); v5 (2019)

2015 2016 2017 2018 2019 State/Overall District Teams Content/ Presentations Overall District Teams Content/ Presentations Overall District Teams Content/ Presentations Overall District Teams Individual Overall Coaching Teams Training

SLIDE 20

TEACHING COACHING COMPETENCIES

SLIDE 21

Leveraging a School-Wide Approach

Annual Coaches “Kick-Off” event at the beginning of each

school year

Reveal the current matrix
Use a Cool Tool format to teach each competency:

– Include rationale for competency – Include examples of that competency in observable behaviors – Include non-example of that competency

SLIDE 22

23

T1: Attend site-team meetings (data-driven)

Rationale: Training alone is a death sentence for initiatives. On-site technical assistance is key to translating what teams have learned in training into practice. Examples: Non-Examples:

Attending meetings to give guidance and

feedback

Allocating your time to support teams

based on their need

Proactively scheduling time to attend

their meeting (pre-arranged)

Being the leader and running meetings

for teams

Avoiding teams who are difficult to work

with

Attending meeting(s) and not providing

feedback

SLIDE 23

24

O6: Assist district to build capacity for sustained implementation (redefine your role over time)

Rationale: For an initiative to be durable and sustain, it will require multiple champions/ local experts Examples: Non-Examples:

Coordinating trainings for PBISMTSS

across departments (not a one person training)

Continuously working on sustainability

(not as a goal, but as a value)

Relooking at organizational structures as

needed

Advocating for schools & district needs
Leaving training to a lone soldier
Relying on others to make the change
Operating in a silo because it’s easier
Being the superhero who does everything

for everyone always

SLIDE 24

25

T5: Use TFI to collect data and drive action planning

Rationale: Using the TFI is required of all sites who are formally identified as implementing PBIS in Nevada. Site based activities should be in pursuit of critical implementation components of the tier they are on to move forward. Examples: Non-Examples:

Following up with teams after TFI

administrations to help support them in action planning

Requesting copies of team action

plans

Letting teams drift and tend to

topics/activities that are far removed from components of implementation

Not prompting sites to action plan

SLIDE 25

Notes about teaching the Competencies

Articulating the rationale is extremely important
Allow for veteran coaches to provide supplemental

examples/non-examples and reinforce or provide corrective feedback based on what they shared to enhance discrimination of appropriate ways to engage in the competencies

A through discussion of each competency is required. The cool

tools alone are insufficient

SLIDE 26

HOW TO MEASURE COMPETENCIES AND SET CRITERIA FOR HIGH-FIDELITY PERFORMANCE

SLIDE 27

How might you measure these behaviors?

SLIDE 28

Next Challenge

Needed a way to:

Quantifiably measure the activities and job duties executed by

external coaches

Capture variance within each competency

– More than yes/no

SLIDE 29

Training (T) Coach Expectations Red Yellow Green 1

Ask for clarification or assistance for concepts/topics with which you are not yet fluent

Assistance was needed but not sought after Assistance was needed but inconsistently sought after Assistance was needed and sought after/not applicable

2

Schedule & present targeted trainings

Training was not scheduled/delivered

r did not address

the data driven target Training was delivered but did not meet the needs of the topic/group Training was high quality, met the needs of the group; topic was relevant and based on data

3

Collect evaluations on all formal presentations

Evaluations were not collected Evaluations were partially collected and/or not turned in All evaluations were collected and turned in on time

4

Review & reflect

n feedback from:

participants, district administrators, and coordinators

Could not provide a concrete example Feedback was received, but did not drive action/change in practices Provided 1(+) concrete example of feedback that prompted adaptation

5

Engage in Effective Spokesperson techniques while presenting

Could not provide an example Can provide one example from a training, meeting, or

ther

presentation

pportunity

Can provide two examples from a training, meeting, or

ther

presentation

pportunity

Bonus

Present on MTSS/PBIS at district, state, and national forums

Opportunity was available but did not seek Sought

pportunity but

was denied Presented on MTSS/PBIS

Coaching Teams (C) Coach Expectations Red Yellow Green 1

Attend site-team meetings (data- driven)

Attends 1 meeting Attends 2 meetings based

n data

Attends 3 meetings, at least 2 for emerging or developing sites

2

Collect monthly artifacts from teams

0 artifacts collected per team 1 artifact collected per team for less than 80% of schools 2 relevant artifacts collected per team for 80% of schools

3

Facilitate establishment of data collection systems

0 – 4 contacts 5 – 7 contacts 8 – 10 contacts

4

Consultation notes completed

Documented less than 50% of TA Documented 50-79% of TA Documented 80% or more of TA

5

Use TFI to collect data and drive action planning

Could not provide a concrete example Provided 1 concrete example Provided 2 concrete examples

6

Use various data sources for assessment, planning, and reporting

Could not provide a concrete example Provided 1 concrete example Provided 2 concrete examples

7

Advocate for smallest change that will have the largest impact

Could not provide a concrete example Provided 1 concrete example Provided 2 concrete examples

8

Build credibility & rapport

Could not provide a concrete example Provided 1 concrete example Provided 2 concrete examples

Overall (O) Coach Expectations Red Yellow Green 1

Attend meetings and coach trainings with coordinator

Attend less than 2 Attend 2-3 Attended at least 4 meetings/trainings

2

Self-monitor tasks assigned and completion

Less than 50% of activities across weekly target sheets are completed 50-79% of activities across weekly target sheets are completed 80% (+) of activities across weekly target sheets are completed

3

Engage in formal dissemination

No formal dissemination of information/ presentations Newsletter published late/or lacking relevant content, data, or recognition High-quality newsletter with data, content, and recognition

4

Reports and data are delivered to the state on or before due date

Data for less than 80% of teams submitted on time Data for all teams submitted late (within 1 week) Data for all teams submitted on time in relevant months

5

Use data to allocate time and resources

Evidenced less than 2 weeks (50%) of time across target sheet expectations Evidenced 2/4 weeks (50%) of time across target sheet expectations Evidenced at least 3/4 weeks (75%

r greater) and

activities are distributed across the matrix

6

Assist district to build capacity for sustained implementation (redefine your role over time)

Cannot provide a concrete example for this month Can provide at least 1 concrete example for this month Can provide at least 2 concrete examples for this month

7

Read professional literature at a minimum of 30 min/week

Participated/read less than 1 hour of professional literature/webinars this month Participated/read under 2 hours of professional literature/webinars this month Read at least 2 hours of professional literature this month or/and attended a webinar

8

Engage in professional and ethical standards (see Coaches Code)

Can give no examples of adherence to the ethical standards or coaches code or had a violation of ethical standards Can give 1 example

f adherence to the

ethical standards or coaches code Can give 2 examples of instances where the coaches code was used in coaching, training,

r presentations

SLIDE 30

PROGRESS MONITORING

SLIDE 31

Feedback Rubrics

Monthly Feedback Session with

External Coach

Provides feedback on

performance with respect to:

– Expectations – Roles/Functions – Overall competency distribution

SLIDE 32

SLIDE 33

SLIDE 34

SLIDE 35

How might you use these data?

SLIDE 36

Same data, different purposes

Individual progress monitoring tool
Provides a view of districts coaching activities in action
Measurement of adherence to coaching activities by “coaches”

– Data leveraged for advocacy when “other duties as assigned” chronically interrupts coaching activities

At times, the data reveal district barriers to effective coaching

behaviors

SLIDE 37

PAYING HOMAGE: BEHAVIOR SCIENCE, OBM

SLIDE 38

Total Performance System

“Management Without Supervision”
Management with positive reinforcement
Reduces the need for direct supervision
Managers may spend more time:

– Planning – Coordinating – Optimizing performance

39

Abernathy (2000, 2011, 2014)

SLIDE 39

Alignment with Abernathy’s TPS

40

Observable, measureable strategic objectives

Expectation Matrix Self- Management Tools Feedback Rubrics

Personal scorecards with specific measures and goals

Feedback Scorecard

Managers assist in pinpointing ongoing

pportunities and

designing improvement plans

Performance Charts Tactical Improvement Plan

Employees share in the organization’s success

Performance- Based Incentives

SLIDE 40

TACTICAL IMPROVEMENT PLAN

Flowchart Pinpointing Obstacles to Maximum Performance

41

SLIDE 41

Abernathy Theory

How we typically address poor performance

– Look for problems inside the person – Managers point to characteristics like cooperativeness, dedication, commitment, work ethic

It is easier for managers to blame their employees than

themselves

– “If I only had good students, I could teach!”

42

SLIDE 42

Pinpointing Obstacles to Maximum Performance

43 Target High Opportunity Measure Measure Validity Issue? Performance Improvement Issue? Behavior Issue? Selection or Training Behavior Prompts Behavior Feedback Behavior Consequences Process Issue Coach Utilization Work Methods Adapted from Abernathy, 2000

SLIDE 43

Pinpointing Obstacles to Maximum Performance

44 Target High Opportunity Measure Measure Validity Issue? Performance Improvement Issue? Behavior Issue? Selection or Training Behavior Prompts Behavior Feedback Behavior Consequences Process Issue Coach Utilization Work Methods Adapted from Abernathy, 2000 Relevant and attainable measures and goals Skills and/or training Cues, signals, instructions, job aids, or

ther

antecedents to work actions Scorecards, coaching sessions, etc. Praise, recognition, feedback, criticism, etc. Pulled away for other projects/acti vities

Work process- es, quality

f input,

budget, etc.

SLIDE 44

LESSONS LEARNED & FUTURE DIRECTIONS

SLIDE 45

Lessons Learned

Be open to annual revisions

– Keep notes from monthly feedback sessions to drive future changes to contexts, competencies, and especially rubric criteria

Need to depersonalize individual performance

– Many activities are confounded by system barriers outside of coaches control

Additional lessons

SLIDE 46

Future Directions

Desire to investigate empirical questions

– How/does coaching performance on the TPS impact:

School outcomes (fidelity, implementation, sustainability)
District outcomes (capacity, sustainability, etc)

– Is there a threshold for maximum impact? – Are coaches utilizing this system more efficacious than those who do not have this system? (explore beyond anecdotal reports)

Analysis beyond individual progress
Additional things we want to research

SLIDE 47

What future directions would you recommend?

SLIDE 48

Questions

We appreciate you being here! Kaci Fleetwood; Kacif@unr.edu Jodie Soracco; Jodie@unr.edu