Chautauqua & Rockland Counties (New York State) & Manchester - - PowerPoint PPT Presentation

How to Continuously Evaluate System of Care Expansion by Combining Universal Big Data From All Human Services and Entire School Populations: Examples From Chautauqua & Rockland Counties (New York State) & Manchester City Council (UK)

Workshop Presented at University of Maryland Training Institutes, July 25-28,2018, Washington, D. C. Mansoor A. F. Kazi, PhD, President, Realist Evaluation Inc., Director, Program Evaluation Center, Fredonia SUNY; Rachel M Ludwig, LCSW, Chautauqua Tapestry Director; John Rosiak, Rosiak Associates, LLC Susan Hoerter, DO, Rockland County SOC Expansion Grant PI; Janet Sliva, Rockland County SOC Expansion Grant Manager

Acknowledgements

• Dr. Bret Apthorpe, Superintendent; Jessie Joy,

Director of Curriculum, Instruction & Assessment; Tina Sandstrom, Director of Schools, Jamestown Public Schools; John Panebianco, Director of Pupil Services

• Dr. Gary M. Blau, Chief, Child, Adolescent and

Family Branch, Center for Mental Health Services, Substance Abuse and Mental Health Services Administration

• Patricia

Brinkman, Commissioner; Chautauqua County Department of Mental Hygiene, Mayville

• Rachel Ludwig, Director Chautauqua Tapestry
• Manchester

City Council—Marie McLaughlin, Director Youth Justice Services

Acknowledgements

• Ed Day, Rockland County Executive
• Tom Zugibe, District Attorney
• Michael Leitzes, Mental Health Commissioner
• Mary Jean Marsico, Rockland BOCES
• Dr. Susan Hoerter, Rockland SOC Expansion PI
• Janet Sliva, Rockland SOC Expansion Grant Manager
• Barbara Gavin, Director, Child Welfare Services
• Carol Korngold, Safe Youth Center Executive Director
• Kathy Farber, Safe Youth Center Consulting Director
• Kathy Tower-Bernstein, Director of Probation

Tapestry ry of Chauta tauqu qua a County, ty, New York

Mansoor A. F. Kazi, President, Realist Evaluation Inc.; Affiliate Research Fellow, University at Buffalo; Director, Program Evaluation Center, Fredonia State University of New York; Adjunct Fordham University

kazim@fredonia.edu mkazi@buffalo.edu

• American Evaluation Association (eval.org) Co-Chair

Human Services Evaluation Topical Interest Group & Social Work Topical Interest Group.

Based on Kazi, M. A. F. (2003) ‘Realist Evaluation in Practice’, London: Sage

Realist Evaluation Partnerships SAMHSA’s Gold Award for Outstanding Local Evaluation 2010 Training Institute STAKES/THL

MANCHESTER CITY COUNCIL, England

ROCKLAND COUNTY NY

New York State System of Care SAMHSA Funding

Funded by SAMHSA—Substance Abuse and Mental Health Services Administration, Center for Mental Health Services:

• ‘Tapestry of Chautauqua’ Systems of Care Grant (with

Chautauqua County Department of Mental Hygiene, $9,000,000 2008-2015; $977,317 for evaluation, Funded by SAMHSA; Recipient of SAMHSA’s Gold Award for Outstanding Local Evaluation, July 2010).

• ‘Tapestry of Chautauqua’ Systems of Care Expansion Grant

(with Chautauqua County Department of Mental Hygiene, $4,000,000 2015-2019)

• Rockland County (NY) Systems of Care Expansion Grant,

with Rockland County, $4,000,000 2016-2020; $800,000 for evaluation, Funded by Substance Abuse and Mental Health Services Administration (SAMHSA).

Take the

Gary blau

challenge

T= (V + B + A) x (CQI)

2

EBP’s & Clinical Excellence Cultural & Linguistic Competence Continuous Quality Improvement Family Driven Youth Guided

T ransforming Children’s Mental Health in America!

System of Care: Focus on Values

Children’s Mental Health Initiative (CMHI): Systems of Care

CMHI has impacted 22% of the nations counties and 33 federally recognized Indian Tribes Since 2000, SAMHSA administers an average of 61 system of care grants per year CMHI sites have served over 100,000 children and youth There are currently 88 CMHI funded sites

Systems Of Care Work!

Reduced behavioral and emotional problems Increased behavioral and emotional skills Reduced suicidal ideation and attempts Reduced substance use problems Improved functioning in school and in the community Improved ability to build relationships

View all of the KSOC-TV webisodes on the SAMHSA YouTube channel or by going to www.samhsa.gov/children

The Goals of a System of Care

• To improve access and expand the array of services

and supports for children and youth with a serious emotional disturbance and their families.

• To promote the full potential of every child and

youth by addressing their physical, emotional, intellectual, cultural and social needs in family, school & community

Who Comes to School?

• 21% of children ages 9-17 have a diagnosable mental health

disorder (U.S. Surgeon General, 1999; NIMH, 20% currently)

• 20% of children with mental health disorders are identified

and receive mental health services (U.S. Surgeon General, 2000; mentalhealth.gov—20% currently)

• 50% of students age 14+ who have a mental illness drop out
• f high school…the highest dropout rate of any disability

group (U.S. Department of Education, 2001) (NIMH currently)

Chautauqua Tapestry

• Why “Tapestry”?
• Quick history
• Goals
• Partners

SAMHSA’s Theory of Change

Expansion = System Change

 System of care expansion is not a “project”  Goal is sustainable systemic changes in child-serving systems to improve services and outcomes  Occurs with or without federal grant  Grants are “venture capital” to achieve larger system change  Linked to other system reforms

Keys to Successful Expansion

• 1. Commitment of high-

level policy and decision makers

• 2. Realistic goals
• 3. High-priority goals
• 4. Specific, concrete

strategies

• 5. Strong leadership to

manage implementation

• 6. Linkages with larger

system reforms in environment

• 7. Cross-agency

partnerships

• 8. Commitment across key

stakeholders

• 9. Staff and resources

allocated to implementation work

Evaluation as Sustainability Strategy

• Similar organizations collecting quality

data, but uncoordinated and isolated

• Similar organizations collecting quality

data for the same outcomes in a coordinated and collaborative network

• Similar organizations collecting quality

data that align under similar outcomes for broad community impact

Collaboration Defined:

A mutually beneficial relationship between two or more parties who work toward common goals by sharing responsibility, authority and accountability for achieving results.

New York State

Evaluation Examples

• Jamestown Public Schools
• North Rockland School District
• Manchester City Council, UK

Critical Realism and Outcomes

• Sayer (1984): Realism links theoretical abstraction with

empirical research

• “Essentially eclectic, the realist approach seeks to move

away from what may be now rather sterile epistemological debates surrounding scientific versus pluralistic and constructivist paradigms, using all methodologies as appropriate within the realist

• framework. The approach facilitates the interrogation
• f empirical data with a view to identifying the nature
• f specific structural influences on outcomes” (p.45).
• Realist evaluation embraces outcome measurement

and also investigates the factors that influence the

• utcomes

Realist Evaluation

• Realist evaluation seeks to evaluate practice within the realities of

society, based on Emergent and Symbiotic elements of natural law.

• All systems are emergent and constantly in a state of evolution,

along with the reality that we are all symbiotically connected to nature and to each other.

• Practice takes place in an open system that consists of a

constellation of inter-connected structures, mechanisms and contexts.

• Realism aims to address all the significant variables… through a

realist effectiveness cycle which links the models of intervention with the circumstances in which practice takes place (Rom Harre’s models, analogous with reality)

• Explanation at any one time requires further investigation and

further explanation (continuous testing & development of Rom Harre’s models)

Purposes and Objectives of evaluation

• Research methods drawn from both epidemiology and

efficacy research traditions in a realist evaluation

• Partnership with human service agencies & schools to

investigate what works and for whom. Real live data mined from human service agencies is used to investigate the effectiveness of the human service interventions.

• Emphasis is on data naturally drawn from practice.
• Binary logistic regression as part of epidemiologic

evidence based on association, environmental equivalence, and population equivalence.

• How evaluators and agencies can make the best use of

the available data to inform practice.

Purposes and Objectives of evaluation

• Research indicates that the reform of the system of

care and the use of initiatives such as wraparound are effective in improving mental health and functioning in school (Kutash, 2006; Reback, 2010; Goldenson, 2011)

• However, most studies have focused on at risk

groups rather than the total school populations.

• The purpose of this study is to undertake a 100%

evaluation of all school-based services, utilizing data

• n the entire school populations, in a longitudinal

study.

Evaluation Strategy With Each Agency

• Evaluation resource and service for you and for

each participating agency

• How to access and to use your own MIS data
• How to analyze this data repeatedly to inform

practice

• Carried out with you and for you only—you

decide who to share with and how to use the findings

Realist Evaluation: What Interventions work & in what circumstances

• A combination of efficacy research & epidemiology traditions
• Management Information System (MIS) Data routinely collected but

typically not used for evaluation in agencies

• Investigate interrelationships between outcomes, client demographics,

client circumstances, & services provided (interventions)

• Methods such as binary logistic regression can predict the likelihood of

effectiveness of an intervention in given circumstances

• Use findings at regular intervals to better target and develop services

Education Partners in SOC

• Education plays a critical role in the

development of children.

• Positive learning experiences help prevent

emotional and behavioral problems.

.

School Examples of Realist Evaluation

• 17 NYS school districts have participated
• 100% data from school + participated agencies
• What works and for whom in achieving school,

agency and system of care outcomes

• Examples of school data

Demographics Intervention Outcome

• Ethnicity
• Gender
• Lunch status,
• Special education
• School based

interventions

• Pre-school

programs

• Mental health
• DSS
• GPA/ State tests
• Behavior incidents
• Attendance
• Drop out rates

Graduation rates

Data analysis and utilization

• Single system & one group pre/posttest
• Comparison of outcomes between baseline and

subsequent periods and those receiving and not receiving interventions

• Investigation of patterns between outcomes,

demographics and interventions

• Binary logistic regression
• Data analysis in partnership with schools and

agencies

• Utilization of findings to develop and improve

services for children and families

Outcomes: Jamestown Public Schools

• Strengths and Difficulties Questionnaire

(SDQ) April 2015 and April 2016

• Recorded Incidents of Behaviour 2014/15

and 2015/16

• North West Evaluation Association (NWEA)

Measures of Academic Progress (MAP) Fall 2015 , Spring 2016 & Fall 2016 (grade levels K to 10)

Universal Screening Tool: SDQ

• Strengths and Difficulties Questionnaire

(SDQ)

• Brief measure of pro-social behaviour and

psychopathology, 3-17 yr olds

• Goodman (2001)—reliability .73
• Five factors: emotional symptoms, conduct

problems, hyperactivity, peer relationships and pro-social

• Grade levels K to 4 in April 2012, 2013,

2014, 2015 & 2016

SDQ 2012 Normal Scores/Ethnicity

SDQ 2012 Predictors for abnormal/borderline total scores

Predictors for Change from 2012 to 2013 Total SDQ

Hyperactivity Level 2015 SDQ

Hyperactivity Level 2016 SDQ

Hyperactivity Level Change from 2015 to 2016 SDQ

Hyperactivity Level Change from 2015 to 2016 SDQ: Predictors for improvement

Hyperactivity Level Change from 2015 to 2016 SDQ: Striders Advocacy

Hyperactivity Level Change from 2015 to 2016 SDQ: Jamestown Behavioral Health Clinic

Tapestry Partners Again Predictors for Improved Behavior in Schools

• Recorded incidents of behavior Compared between

2014-15 and 2015-16 school years

• Overall mean was 0.96 in 2014/15 & 1.58 in 2015/16

recorded incidents per student ( n = 5134) in Jamestown Public Schools in both years

• Once again Tapestry partners’ services were

predictors for improved behavior, this time several more!

Safari Improved Behavior

Improving Measurement of Academic Achievement

• Northwest Evaluation Association (NWEA)
• Measures of Academic Progress Normative Data

for Reading and Mathematics

• More reliable than GPA
• Used three times a year up to grade level 10
• Beginning from Baseline Year 2015/16
• First two times used Rasch Unit Scale (RIT) but

for the third assessment will use expected growth

• Of those proficient at level 3 or 4,

51.8% demonstrated growth that was less than expected in the first two reading assessments

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016

NWEA Reading Fall 2015 & Spring 2016 Predictors for Improving RIT Score at Median 9

• r Higher

Plan for Chautauqua Tapestry 2.0(2015- 2019)

– Chautauqua County the only one to receive SAMHSA’s Gold Award for Outstanding Local Evaluation. – Tapestry 2.0 will deepen its evaluation with school districts and human service agencies in the county. – Chautauqua County is a world-leader in realist evaluation, and to date 28 agencies (including 7 school districts) have collaborated

New York State

Rockland County MACSHY

Rockland County MACSHY Leaders

Rockland County PSY

https://www.dropbox.com/s/9zjm3nie7slvl7w/P SY_Rafael_FINAL.mp4?dl=0

Outcomes: North Rockland Schools

• Recorded Incidents of Behaviour- 2016/17

(grade levels K-12th)

• Istation Universal Screener- Fall 2016 &

Spring 2017 (grade levels 3rd to 6th)

• Developmental Reading Assessment 2

(DRA2)- Fall 2016 & Spring 2017 (grade levels 1st- 4th)

• North West Evaluation Association (NWEA)

Measures of Academic Progress (MAP) Fall 2015 , Spring 2016 & Fall (grade levels K to 10) from 2017/18 School Year

North Rockland Schools 2017/18

Recorded Incidents of Behavior 2016/17 (8.9%) to 2017/18 (11%; N = 8026)

• Mental health affects school functioning (Reback, 2010; Desrocher, 2015;

Rossen & Cowan, 2014).

• However, most studies have focused on at risk groups rather than the

total school populations.

• Rockland County utilizes big data continuously on entire school

populations with North Rockland Schools.

• It was found that, out of the 85 that had received county mental health

services, 15.3% had improved on the number of recorded behavior incidents as compared with 4.4% of the rest of the school

• Out of the 62 that had received DSS interventions, it was found that 9.7%

had improved on behavior incidents as against 4.5% that had also improved in the school population not receiving these interventions.

• There were 16 that had been involved with Partnership for Safe Youth,

and of these 4 (25%) had improved as against 4.5% of the rest that had not been involved with the PSY; however, the PSY total was less than 20 at this stage and therefore not included in the inferential statistics.

Recorded Incidents of Behavior 2016/17 (8.9%) to 2017/18 (11%; N = 8026) Binary Logistic Regression: Incidents Improved or not--Variables in the Equation

B S.E. Wald df Sig. Exp(B ) Gender2017(1) 0.518 0.122 18.17 1 1 1.679 Ethnicity2017(2) 0.902 0.228 15.68 5 1 2.465 Ethnicity2017(3) 0.8 0.195 16.9 1 2.226 MHPatientOrNOTALLto1 22017(1) 0.793 0.38 4.356 1 0.037 2.209 FreeReducedornot(1) 0.408 0.14 8.486 1 0.004 1.504

Plan for Transforming Rockland County(2016-2020)

– National Evaluation Requirements – Local Evaluation Strategies – Continuous realist evaluation of services utilizing entire Management Information Systems – Department of Social Services – County Mental Health Services – Probation Service – North Rockland and Nyack School Districts – Continuous quality improvement

Plan for Rockland County(2016-2020)

– Rockland County Evaluation will expand from the largest North Rockland Schools to include all 8 school districts (n = 41,000) – One of 37 in NY, Board of Cooperative Educational Services (BOCES) plays central role with Partnership for Safe Youth – 100% data from County Mental Health, Department of Social Services, Probation Service and other human service agencies, and merged with whole school big data

Example: Manchester Youth Justice Service in England

• This interim report is based on electronic data

comprising of a 100% sample of 2,526 youth in the 4-year period January 2011 to December 2014

• 76.3% were male, and 62.2% of the youth were

white

• The average age at date of arrest for all youth

was 15 years, ranging from 10 to 18 years.

Manchester Youth Justice Service in England

• Offence database recorded every episode of an
• ffence reported on the youth,
• Work out those that had offended after the first

recorded offence, and those that had more than

• ne, in the 4-year period from January 2011 to

December 2014.

• It was found that 66.1% had not offended again

after the first recorded offence (as compared with 45.6% in the previous 10-year analysis reported at AEA 2012).

Binary Logistic Regression Model

Predictors for returning

• Age at first order. Those older by every year were 1.5

times more likely not to return

• Gender. Females were more likely not to return than

males.

• Those who had completed the order were 7 times more

likely not to return.

• Youth living with parents or relatives were 2.6 times more

likely not to return.

• Those with higher baseline ASSET scores were more

likely to return

• Those on youth caution more likely not to come back

Manchester Youth Justice Service: Linking Data with Practice

• Using our data differently
• Clearer picture of our cohort
• Understanding what is working and

what is not working

• Helping

to target resources more effectively

• Identifying predictors of re-offending

Manchester Youth Justice Service: Looking Forward

• Improving the way we record data
• Working with partners to improve use
• f data
• Merging this data with a wider pool of

data within the City of Manchester e.g. schools, police, health, benefits agency

• Targeting resources more effectively

to get better outcomes

Conclusion

• Realist evaluation strategies help to utilise the

available whole school data and to conduct deeper analysis

• Explanations for differences between those that

improve on an outcome and those that do not

• Identifies

patterns in the data where multiple factors are influencing the outcome, and selects the main factor or factors responsible for the

• utcome, with a prediction of the odds of achieving

a given outcome in particular circumstances

• Repeatedly

investigate changes in

• utcomes,

intervention & the contexts