London Office of Technology & Innovation Workshop: Evaluation - - PowerPoint PPT Presentation

Workshop: Evaluation Framework for Assistive Technology pilots

13 November 2019

Lead boroughs: Newham, Camden and Hackney

London Office of Technology & Innovation

LOTI_LDN

medium.com/loti

#LOTI

Welcome and Introductions

10:00 Priya Javeri

1. To learn from boroughs’ experiences of evaluating previous AT trials 2. To create a template evaluation framework for ATs

Objectives for today:

10:05

Agenda

9:30 Arrival 10:00 Welcome and Introductions - Priya 10:05 Objectives - Priya 10:10 Exercise 1: Sharing of Experiences - LOTI 10:40 Exercise 2: Creating Personas - LOTI 11:10 Break 11:20 Recap of LOTI AT evaluation research - LOTI 11:30 Exercise 3: Creating a prototype evaluation framework - Priya 12:25 Wrap up and next steps 12:30 Workshop Close

Exercise 1

10:10

Sharing of experiences

• What AT pilots have you run / are

you running now?

• What has been your approach to

evaluation?

• What have been the strengths and

weaknesses of your evaluations?

Exercise 2

10:40

Creating personas

• From your experiences with AT, what

factors about a person, their life and context are most important to understand before implementing a new AT intervention?

• Which factors are common to almost any

person and scenario?

• Which factors are unique to certain types
• f person and scenario?

Break

11:10

AT evaluation - Research Summary

11:20

Key points from interviews with Sutton and Hackney Learnings from existing projects and considerations for future evaluations

• create a baseline
• Set up control groups
• Involve end users
• Gather qualitative and quantitative data
• Consider: a) the technology and any processes in place aspects before

commissioning, b) the technical practicalities, c) borough context, d) the purpose Existing tools: LGA’s Digital Social Care Programme and Global Disability Innovation Hub

What we’ve learned from our desk research From other frameworks

• NICE framework
• The Nesta Standards of Evidence
• SETT Framework

From case studies:

• RNIB - highlights the need to provide training including takeaway factsheets /

user manuals to users of AT devices.

• CQC - highlights that different ATs (independent living, condition

management etc.) might require bespoke qualitative and quantitative evaluation.

How LOTI can add value

Develop a consistent methodology for evaluating AT pilots that can enable LOTI member boroughs to compare like for like technologies and make better investment decisions.

LOTI’s outcomes-based methodology could be used to help boroughs frame their hypothesis about the intended change they believe an AT might make to help someone live independently. PROBLEM SOLUTION

Phase

2 - Discover 3 - Define 4 - Develop 5 - Deliver

Generate Focus Generate Focus Key Question

What are the real problems and what do we know about them? What specific problem(s) will we try to solve? What hypotheses do we want to test by prototyping? Which prototypes perform best?

Tech & Data

What problems relate to technology and data? What role can technology and data play in enabling the desired outcome?

Non Tech & Data

What problems relate to people, processes and conditions? Beyond technology, what is needed to achieve the desired outcome?

OUTCOME

What real-world

• utcome do

we want to enable? (This is not the solution, tech or data!)

1 - Motivate

LOTI draft AT Evaluation Framework template

Desired Real-World Outcome (applying slide 5):

For each of the below, we need to capture: 1) How are things now? What’s our baseline? 2) What’s are the specific problems the AT will address / what are the user needs? 3) What’s our hypothesis for change and how it will we measure it? 4) What results and insights have we captured?

INDIVIDUAL ENVIRONMENT

Goal: Assess the individual’s current abilities, unique needs, and functional areas of concern. Goal: Assess the individual’s environment (including the context of any public service support being delivered), how the proposed AT integrates, and the individual’s current capabilities within those environments in comparison to other individuals.

TASKS TOOLS

Goal: Assess what actually happens in the environment (including the context of any public service support being delivered) and the impact of current demands on the individual. Goal: Assess the devices, services, resources and strategies that are currently in place to support the individual. Assess whether the proposed tools are user-centred and/or task orientated.

Consider what Standard of Evidence you need to make the desired decision:

• What are the variables and which can you control?
• What are the limitations of your sample size?
• What qualitative and quantitative data can you realistically capture?

Exercise 3

11:30

Creating a prototype evaluation framework

• What are the objectives of AT evaluations

/ what needs must they serve, individually and collectively?

• Against what headings do we need to

capture information?

• What is realistic to measure?

Wrap up

12:25

Next steps

• Agreeing and prototyping the evaluation

framework

• Refining it for more systematic testing of

ATs

Workshop close

12:30

London Office of Technology & Innovation

LOTI_LDN

medium.com/loti

#LOTI