Shelley Hughes Occupational Therapist Senior Product Manager - - PDF document

9/23/2016 1

Learning to Administer and Interpret Assessments to Support your Continuing Professional Development (CPD)

Shelley Hughes Occupational Therapist 22 Sept 2016

1

CPD and Standardised Assessments

Shelley Hughes

Occupational Therapist Senior Product Manager Pearson Clinical Assessment Twitter: #OTLearn

CPD and Standardised Assessments | 2

9/23/2016 2 By the end of this session, attendees will be able to…

… explain the CPD requirements for Occupational Therapists in the UK … describe the steps required to begin learning to use a standardised assessment … share ideas for working with colleagues to learn to use standardised assessments as a CPD activity … explain some of the concepts of standardised assessments to evidence this as a CPD activity

3

CPD and Standardised Assessments

Agenda

I CPD requirements UK Occupational Therapists II The assessment process III The essentials of getting started IV Learning with colleagues V Graduating from basic to advanced administration and interpretation

4

CPD and Standardised Assessments

9/23/2016 3

CPD Requirements CPD can be defined as: “a range of learning activities through which health and care professionals maintain and develop throughout their career to ensure that they retain their capacity to practice safely, effectively and legally within their evolving scope of practice”

Health and Care Professions Council (HCPC)

6

CPD and Standardised Assessments

9/23/2016 4

CPD Requirements in the UK

• 1. Standards of proficiency

‒ Meeting the standards ‒ Scope of practice

• 2. CPD and Audit Process

‒ Need to document CPD ‒ Demonstrate a variety of learning activities ‒ Seek to ensure that CPD contributes to quality of practice ‒ Must benefit the service user[s] ‒ May be audited

• 3. Examples of CPD activities

‒ Work based learning ‒ Professional activities ‒ Formal education ‒ Self-directed learning ‒ Other

7

CPD and Standardised Assessments

COT Guidelines

1.College of Occupational Therapists Code of Continuing Professional Development – 2014

• 2. Aligned with and complimentary to the HCPC guidelines

(does not replace these) 3.Underpinning concept = critical reflection

8

CPD and Standardised Assessments

9/23/2016 5

The Assessment Process The Assessment Process

1.Hypothesis testing

‒ A problem solving process ‒ Identifying strengths

2.Types of Assessment

‒ Screeners ‒ Non-standardised ‒ Criterion-referenced ‒ Standardised ‒ Full battery ‒ More specific ‒ Predictors for problems in everyday life

10

CPD and Standardised Assessments

9/23/2016 6

background information develop hypothesis screening assessment revise hypothesis full battery assessment specific assessment prove / disprove hypothesis Intervention planning /

• utcome measure

The Assessment Process

CPD and Standardised Assessments

Standardised Assessments: Key Features

• 1. Allows you to compare your patients score to a criterion

‒ the patient’s score is compared to the norms - the scores of the reference group

• 2. Is always administered and scored in exactly the same way
• 3. Comparisons made using standardised scores

‒ Raw Scores (non-standardised) ‒ Percentile Ranks ‒ Standard (or Scaled) Scores

• 4. The aim is to decide whether:

‒ patient’s performance is evidently different to most or all of the typical sample ‒ the patients performance is significantly different to what you would expect in view of age, ability etc

12

CPD and Standardised Assessments

9/23/2016 7

The Essentials of Getting Started Where to Start…

• 1. Understand the population for which the assessment is

intended 2.What service needs will the assessment meet 3.How was the assessment developed 4.How are assessment findings reported

• 5. Practice administration

14

CPD and Standardised Assessments

9/23/2016 8

How was the Assessment Developed…

• 1. Theoretical Underpinnings
• 2. Data Collection

15

CPD and Standardised Assessments

Evidence of Consistency & Usefulness…Reliability

Is the test a reliable and consistent measure?

Inter-rater reliability

‒ are the scores likely to be the same no matter who is administering the test?

Test-retest reliability

‒ are the scores likely to be the same if the test is given on two

• ccasions separated by a short interval?

Parallel form reliability

‒ where the test has parallel forms, are these equivalent in difficulty?

16

CPD and Standardised Assessments

9/23/2016 9

Evidence of Consistency & Usefulness…Validity

Does the test measure what it claims to measure?

Construct validity

‒ Is the test measuring the theoretical concept it says it is measuring?

Content validity

‒ Does the test assess a reasonable range of behaviour related to the concept being measured?

Criterion validity

‒ Are the test results consistent with results from other measures of the concept being measured?

Face validity

‒ Does it appear as if it is measuring what it is supposed to be measuring?

Ecological validity

‒ Do test results predict behaviour in everyday situations?

17

CPD and Standardised Assessments

How we Measure Reliability and Validity: Correlation Coefficients

• Measure of the degree of relationship between 2

variables e.g. age and scores on a test

• Range +/- 1 to 0
• +1 and –1 are perfect correlations in different directions
• 0 = no relationship between the variables
• Correlation of .6 = acceptable, .8 or higher = very good

18

CPD and Standardised Assessments

9/23/2016 10

How are assessment findings reported

‒ The patient’s score is compared to the norms - the scores of the reference group

The aim is to decide whether:

‒ the patient’s performance is evidently impaired i.e. worse than most or all of the typical sample ‒ the patients performance is significantly different to what you would expect in view of age, ability etc

19

CPD and Standardised Assessments

Normal Distribution:

‒ Distribution of test scores for large groups of heterogeneous test takers ‒ 68.26% of population should fall within 1 Standard Deviation (S.D.)

• f the mean

‒

2 S.D.’s = 95.44%

‒

3 S.D.’s = 99.73% ‒ Few extremes at both ends ‒ Scores must be normally distributed for these to be useful

20

CPD and Standardised Assessments

9/23/2016 11

Normal Distribution:

21

CPD and Standardised Assessments

Interpretation of Scores

‒ A score on a test does not prove or disprove anything ‒ It just gives an estimate of performance in a particular

area of behaviour

‒ Must be interpreted in the wider context of results from

a range of tests

‒ Take into account previous abilities and achievements

22

CPD and Standardised Assessments

9/23/2016 12

Practice

Administration

23

CPD and Standardised Assessments

The Learning Process

Questions to ask yourself

1.Understand the basics 2.How would the assessment support your decision making and practice 3.What is the evidence base for use of the assessment

24

CPD and Standardised Assessments

9/23/2016 13

The Learning Process

Documenting

• 1. Reflection

‒ How important is this activity to your professional practice? ‒ How will this influence your practice? ‒ Your next steps for learning ‒ Review your learning after several months

• 2. Sharing with others

‒ Teaching students ‒ Share a case study with colleagues ‒ Share with other professions

25

CPD and Standardised Assessments

Learning with Colleagues

9/23/2016 14

Teamwork

• 1. Practice administration
• 2. Inter-disciplinary
• 3. Application to different clinical

populations

• 4. Establish a community of practice

27

CPD and Standardised Assessments

Graduating from Basic to Advanced Administration and Interpretation

9/23/2016 15

The Next Steps

• 1. Plan for your next steps
• 2. Review the research
• 3. Become more advanced in

administration and interpretation

29

CPD and Standardised Assessments

Advanced Administration

• 1. Understanding the assessment

in context

• 2. Interpreting the assessment in

detail

30

CPD and Standardised Assessments

9/23/2016 16

Confidence Bands

Reflect the uncertainty in scores arising from measurement error i.e. – you would not expect the same score on every single occasion.

31

CPD and Standardised Assessments

Confidence Bands

• 1. Reported as 68% or 95% confidence band
• 2. 95% confidence band = you are 95% confident that the true

score will lie within this band

• 3. Based on the reliability measurements of the test
• 4. The higher the confidence interval is the wider the band will be
• 5. Which band is better for your situation?

32

CPD and Standardised Assessments

9/23/2016 17

Confidence Bands

Overlapping Confidence Bands 86 (Standard Score)

95 (Standard Score)

33

79 Verbal Memory 93 91 Visual Memory 99

CPD and Standardised Assessments

Composite vs Subtest Level Interpretation

• 1. Composite more robust
• 2. Some subtests are sufficiently robust to use in isolation
• 3. Comparing subtest level data helps build understanding

about an individual’s profile

• 4. Statistical difference vs clinical importance

34

CPD and Standardised Assessments

9/23/2016 18

Pairwise comparisons BOT-2 Example

35

CPD and Standardised Assessments

Pairwise comparisons

36

CPD and Standardised Assessments

9/23/2016 19

Consolidating and Documenting your Learning

• 1. Case studies

‒ Work through examples in the Manual ‒ Listen to webinars

• 2. Talk through with colleagues

‒ Explain your findings ‒ Check your understanding

• 3. Reflect on whether a more in-

depth interpretation changes your practice

‒ Certainty behind recommendations ‒ Communication with colleagues ‒ More targeted intervention planning

37

CPD and Standardised Assessments

Now have a go

Document what you have learned from this session

38

CPD and Standardised Assessments

9/23/2016 20

There’s so much more to learn

Find out more about us at Pearsonclinical.co.uk