HeadStart Kent Knowledge Seminar 2 21 st January 2015 Measuring - - PowerPoint PPT Presentation

HeadStart Kent Knowledge Seminar 2

21st January 2015 Measuring Outcomes

Time Agenda Who 9.30 – 9.40 Introductions (around the table and presenters) Florence/Angela 9.40 – 9.45 Recap of Learning from Seminar 1 Key Messages: Summary Broader Strategic Issues For Headstart Project Alex Hassett 9.45 – 9.55 Feedback on the Board and Shadow Youth Board Angela Ford 9.55 – 10.05 Broader Evaluation Programme Ugochi Nwulu 10.05 – 10.30 Activity: Challenges faced measuring resilience outcomes. Individual and group exercise – feedback to the wider group Alex Hassett 10.30 – 11.00

• Challenges and practical issues in measuring resilience
• Measures of resilience – some ideas
• A domains approach to measuring resilience

Mark Kerr 11.00 – 11.20 Coffee Break 11.20 – 12.00 Activity: Mapping where your service fits and what you measure Mark Kerr and Alex Hassett 12.00 – 12.15 Feedback on learning Alex Hassett 12.15 – 12.30 Way Forward Florence / Angela

 Resilience is not a trait but an interaction between risk and protective factors  We need to ensure we take an ecological and developmental view of resilience  It is useful to focus on resilience in terms of the areas or ‘domains’ of a person’s life that can be changed.  Negotiation and navigation  We need to consider what resources are available and how accessible are they

Key Messages: Summary

The following are the long term issues that need to be addressed:  Developing an overarching framework of resilience that the range of agencies can sign up to  Encouraging a long term interdependency between individuals, services, agencies on providing an holistic approach to young people  Providing a coherent system for evidence based evaluation ensuring that each element of the system is clear on how they evidence outcomes and impact

Broader Strategic Issues for Headstart Project

Reminder of Kent Activity

Canterbury

Penn State Resilience in secondary schools Penn State Resilience in primary schools. Penn State Resilience in community and target workers Family approach TBC

North West Kent

Safe Spaces in schools Safe Spaces in community hubs Coping packs Family Focus KS2 ACP Peer mentors Active listening mentors Youth MH First Aid Online counselling

Thanet

Restorative approaches in schools Restorative approaches in the community Target Restorative approaches in schools Restorative Ambassadors Restorative approaches families

Resilience Mentors: evidence based model of intensive support. FRIENDS Digital World: full services directory , volunteering & mentoring opportunities, self - referral form and sign posting to social media Social Marketing: skills roadshows, coping packs, phubbing.

Partnership Programme Board, Shadow Board, Knowledge Seminars

Coproduction throughout Young people and Families

Feedback from Young People

 Young people found the domains resilience approach useful  Identified areas of HeadStart Kent they felt would have most impact

 Resilience mentors  Coproduction  social marketing  Family resilience  Safe Spaces  Peer support  Online directory

 Ensure Kent’s Emotional Wellbeing Strategy is central to developments.

 Contributing to service redesign  Connecting to the system and enabling change  Wider stakeholders already mapping and exploring system redesign Outcomes  Early Help: improved emotional resilience and receive early support  Access: Receive timely, assessing and effective support  Whole Family: Recognises and strengthens and wider family relationships.  Recovery and Transition: Prepared for and experience positive transitions

Kent’s Emotional Wellbeing Strategy for Children, Young People and Young Adults

 Every one needs to consider how they contribute to building resilience, and what they could do enhance it further.  If our outcomes frameworks are to be guided by the domains (risk and protective factors), we need strategic cohesion across Kent including:

 Workforce being prepared to work systemically.  Shared language  Less duplication  Easy moving and less transitions.

Workshop

 There is a greater awareness of activity locally and countywide and a lot has happened over a short period of time  People fed back that the knowledge seminars have been useful and thought provoking.  Some of the challenges include:

 How to build coherence and ecological links when more than one intervention is working in the same area.  How to involve more young people of greater diversity  How to get passion and buy in from professionals  How to increase the understanding and scale of social marketing

Workshop Messages

HeadStart Kent Knowledge Seminar

21 January 2015

For HeadStart information http://kelsi.org.uk/pupil_support_and_wellbeing/targeted_ support/inclusion/kiass/headstart.aspx HeadStart Kent Twitter is: @HeadStartKent #headstartmatters

HeadStart Kent Knowledge Seminar

21st January 2015

Evaluation progress

Ugochi Nwulu

Ugochi.Nwulu@kent.gov.uk

 Ugochi Nwulu - KCC / University of Kent  Rob Comber - Education and YP’s Services, KCC  (Eileen McKibbin - Research and Evaluation, KCC)  Gabriela Sette - CHSS, University of Kent  (Prof Patricia Wilson - CHSS, University of Kent)

Evaluation team

Key evaluation questions:

• 1. What are the HeadStart interventions?
• 2. What is the theory of change across the programme?
• 3. How does each intervention contribute to the theory of change?
• 4. What is working well and not so well in the implementation and

the delivery of HeadStart Kent?

• 5. What are the critical and effective elements of the programme

which now need to be scaled up fora Kent wide approach to building emotional health and resilience

Process evaluation of HeadStart Kent

Kent area Intervention 1 Intervention 2 Intervention 3 Data sources: HeadStart operational team / Community practitioners / School staff / Resilience mentors Young people Evaluation methods Case study Focus group Questionnaires

Learning from participants and stakeholders

How we will measure the impact:  HeadStart Schools data

• demographics, risk factor profiles
• pupil absences, exclusions
• numbers accessing targeted support
• CYP who participate in the HeadStart programme

Baseline data collection

 January to March:

 National HeadStart conference  Synergies with the national evaluation - field work, surveys  Informal interviews and refinement of plans

 March to May:

 Focused data collection period  Data analysis and write up

 Evaluation report

 Will include plans for an impact evaluation of fully scaled up projects

Next steps

 Take a few minutes to think about the challenges you face in measuring resilience outcomes in the work that you do

• r the work that you commission. Please note down your

challenges and concerns  Now spend a few minutes discussing with your group what those challenges are. Please can each group decide on the 2 main challenges or concerns they face when considering measuring resilience outcomes?

Exercise One

Example outcome model

Educational Development Results In Increased Protective Factors Decreased Risk Factors Producing Personal Development Social Development Intrinsic Outcomes (individual well-being) Extrinsic Outcomes (wider social good)

Source: Young Foundation, 2012

 Can we measure resilience  The ‘what’ are we measuring – what is our Dependent Variable (DV)  Whether to use a global resilience scale or domain based measure  Self report versus third party rating  Age appropriateness of measure  Validity of measure  Requirements of analysis

Challenges in measuring

 ‘The development of a measurement instrument capable of assessing a range of protective mechanisms within multiple domains provides an approach to

• perationalising resilience as a dynamic process of

adaptation to adversity (Olsson et al., 2003)  Ideally, measures of resilience should be able to reflect the complexity of the concept and the temporal

• dimension. Adapting to change is a dynamic process

(Donoghue and Sturtevant, 2007)

Can we measure resilience?

 Assessments of resilience need to consider:

• I. a) the risk or adversity
• II. b) assets/resources that might offset the effect of the risk

III.c) the outcome  Quantitative direct measurement – using a resilience measurement scale as an outcome measure  Quantitative indirect measurement – modelling a range of data with multivariate statistics  Qualitative – understanding individual experiences

Measuring Resilience

 Several scales developed but not widely adopted and no clear preferred option  Definitional clarity needed which influences how we tackle this  Virtually no valid measures or children  Would need to measure availability of resources at all ecological levels to understand those that demonstrate  Only potential measure: California Healthy Kids Survey – The Resilience Scale of the Student Survey (Sun and Stuart, 2007)

Measuring Resilience cont’

 Remember ‘resilience’ is the ability to overcome adversity due to the interaction of risk and protective factors  Cannot assess resilience until the child experiences adversity  Current Headstart project requires us to focus on the antecedents of resilience i.e. protective factors

The ‘DV’ problem

 Headstart programme aims more suitable for a domain approach  The time limited (currently) nature of your work means for many children (hopefully) resilience will not be experienced due to the absence of significant adversity  The number of partner organisations, some with specific domains of focus, mean individually you are unlikely to provide all protective factors needed. However collectively you will

Resilience or Protective Factors?

Key domains What are you working on?

Secure Base Education Friendships Talents and interests Sociable Behaviour Positive Values

Resilience Domains (Daniel & Wassell)

 One of many decisions that must be taken alongside considerations such as age, ability, domain of interest and ecological level  Can be a combination – both self and others e.g. teacher, parent, carer or other professional and results combined  Research has indicated problems if incongruity between child, parent, and teacher reports

Self Report or Third Party Rating

 Variable focused

 Link among measures of degree of risk/or adversity,

• utcome, potential quality of individual or environment

(to compensate/protect)

 Person focused

 Compare people with different profiles (within or across time) on sets of criteria to ascertain what differentiates resilient from non resilient children

Models / Approach to Resilience

Source: Masten, (2001)

Variable focused studies

 Compensatory effects

 Enough positive assets could offset the burden in child’s life from one or many risk influences

 Three models

 Main effect  Indirect  Interaction

Variable focused studies

 Main effect models Asset Bipolar predictor Risk Outcome

• +
• +

Variable focused studies

 Indirect models (example) Asset Effective parenting Risk Outcome

• +

+

Variable focused studies

 Interaction models Moderator Risk activated moderator Risk Outcome – Not found very often (difficult to detect)

Pros and Cons

 Variable focused

+ Max. stats power, suitable for searching specific links between predictor and outcome

• Fail to capture striking patterns in lives of real people, risk
• f losing sense of the whole

 Person focused

+ variables assembled in naturally occurring configuration, well suited for search for common and uncommon patterns in lives

• Can obscure specific linkages

Resilience-Based Practice

Three Main Principles:

 Inclusive, respectful and engaged practice  Strengths-based practice  Solution focused approaches

Also,

 Fostering community and social connectedness  Attachment theory and Circle of Security

Differences in ecological emphases: Australia vs. UK

Community Family Individual

Resilience in Practice - UK

Intervention Themes:

 Improvement of self-esteem / to like self more  Improvement of peer relationships  Improvement in school experience / behaviour  Control of anger / managing disagreements  Naming feelings / emotional literacy

Target Anger, aggression Strategies Anger control, emotional intelligence Intended Outcomes Increased self-esteem, improved peer relationships and school experience

Resilience in Practice- Australia

Intervention Themes Subtypes Occurrence

• 1. Health

a) Physical/Medical 11% b) Mental/Behavioural 4%

• 2. Emotions & Attachment

a) Parent-child dyads 14% b) Broader family r/ships 3%

• 3. Parenting Skills & Confidence

a) Expectations/boundaries 22% b) Support: Peer/Playgroups 7%

• 4. Legal Issues

5%

• 5. Employment & Education or

Training 8%

• 6. Finances & Housing

5%

• 7. External Supports for CHILDREN

a) From the School 7% b) From the Community 4%

• 8. Reduce Social Isolation

10%

Australian Practice… Chains of Support

 NSWb1: Address mother’s social isolation  link mother with community supportive playgroup  mother-child bonding and attachment is facilitated  new social networks and connections with the community are created  NSWa1: Address uncontrolled behaviour, aggression in children/poor attachment evident  assist father in putting strong boundaries, routines and expectations in place at home  children seen as having greatly improved emotional regulation, able to cope in new spaces or with new people  father more competent and relaxed

Mapping interventions to Literature

• 1. Health
• 2. Emotions & attachment
• 3. Parenting confidence

and skills

• 4. Legal issues

5.Employment/E&T/edn/tr aining

• 6. Finances/Housing
• 7. External supports for

children

• 8. Reduce Social Isolation
• Res. Domains

Secure Base Education Social Competencies

Literature

Build Caring relationships Establish and maintain Self- Efficacy Mobilizing protective resources Creating Opportunities Foster Resilience Strings Ecological

Resilience-based Practice – Implications…

 Practice should aim to target all ecological levels to align to a comprehensive view of resilience  Policy: Ecological supports, especially community level investment  Building a common language around resilience to promote strategic change as disciplines and agencies endeavour to work in concert  Flexibility for assessment and practice frameworks – creativity and individualisation

Resilience-based Practice – Implications… cont’

 Comparative research: Underlying processes vs. behaviours; impacts of various ecological levels  Impact and unique predictive value of attachment as central to intervention efforts  Family definitions of adversity and their priorities for help  Evaluate, evaluate, evaluate! Assessment of outcomes to see if resilience-based interventions work!!

Essentially, resilience-based practice needs to aim for consistency in scope and application + flexibility Outcome evaluation as critical for determining best practice

 ‘Absence of a conceptually sound and psychometrically robust measure of resilience for children under 12’ (Windle et al., 2011)  Reading ability  Problems with cognitive processing of Likert Scales  Distinguishing between how they feel now, and how they typically feel – ‘situational effect’  Developmental age versus chronological age

Issues with Age (Self Report)

Measuring Resilience - Outcomes

OUTCOMES:  Measured through informal channels and processes:  Feedback from parents and other agencies, observations, anecdotal evidence, children’s art  Positive (presence) and negative (absence) indicators of resilience:  Improved social skills, decreased anxiety, increased emotional regulation, better school performance, engagement in community activities

Measuring Resilience – Implications…

 Consistency in outcome measurement to aid evaluation  Policy: Focused drive to incorporate sound outcome measurement  Research to examine breadth of assessment and measurement tools – recommend utility and relevance in the context of actual

• utcomes for clients

 Again, the emphasis lies on consistency of usage: Assessment

and outcome measurement procedures need to align to a resilience-based framework if this is the approach being explicitly espoused by the organization

 To be a valid measure it must have been subject to a number of validation checks including:

 Content validity  Internal consistency  Criterion validity  Construct validity  Reproducibility  And others…  Should also go through peer review  So always try and use and existing measure!

Validity of Measure

Selection of Measures

 What are you going to do with the data?  What questions do you want the data to answer?  What type of analysis do you need to do?  Do you want to be able to generalise your findings?  The type of data you collect influences the analysis you can do

Considering Analysis

 Within group or between groups?  The need for ‘controls’  Score cut off points – defining expected levels, what does success look like?  Sample size and the impact on power and effect

Technical Considerations

 Who are the focus of your activities?  What is their adversity?  Why do you think creative activities might build resilience? (your ‘theory of change’)  How might they benefit from a resilience promoting activity?  How will you know they have benefited?  How do you currently demonstrate this?

Your Work

Creative Activities New skills/learning Social connections/networks Efficacy Cognitive Health

You have 40 minutes for this exercise. As a group we would like you to think about each of your services or interventions that you or your service provides or you commission. Using the domains framework, and the chart provided, think about which domains you have an impact on and how you measure or could measure outcomes within that domain. Once each group member has mapped their service and interventions look at the gaps and think about how they could be filled. Spend the last 5 minutes of the exercise discussing what you feel your key learning has been from this exercise.

Activity