Take Your Level 2s Up a Notch: Start Measuring Application Not - - PowerPoint PPT Presentation

▶
take your level 2s up a notch start measuring application
SMART_READER_LITE
LIVE PREVIEW

Take Your Level 2s Up a Notch: Start Measuring Application Not - - PowerPoint PPT Presentation

Aug 30, 2023 397 likes •873 views

Take Your Level 2s Up a Notch: Start Measuring Application Not Just Recall Presented by: Ken Phillips Phillips Associates February 9, 2016 Phillips Associates 1 Agenda 1. Examine Level 2 evaluation facts 2. Analyze test creation errors in

slide-1
SLIDE 1

Phillips Associates

1

Take Your Level 2s Up a Notch: Start Measuring Application Not Just Recall

Presented by: Ken Phillips Phillips Associates February 9, 2016
slide-2
SLIDE 2

Phillips Associates

2

Agenda

  • 1. Examine Level 2 evaluation facts
  • 2. Analyze test creation errors in a sample

Level 2 knowledge test

  • 3. Examine the difference between test

questions that measure application versus those that measure recall

  • 4. Discover 15 tips for creating valid,

scientifically sound multiple-choice test questions

slide-3
SLIDE 3

Phillips Associates

3

Kirkpatrick / Phillips Evaluation Model

Levels of Evaluation Measurement Focus Time Frame

Level 1: Reaction Participant favorable reaction to a learning program Conclusion of learning program Level 2: Learning Degree to which participants acquired new knowledge, skills

  • r attitudes

Conclusion of learning program or within 6 to 8 weeks after Level 3: Behavior Degree to which participants applied back-on-the-job what was learned 2 to 12 months Level 4: Results Degree to which targeted business

  • utcomes were achieved

9 to 18 months Level 5: ROI Degree to which monetary program benefits exceed program costs 9 to 18 months

slide-4
SLIDE 4

Phillips Associates

4

Level 2 Evaluation Facts

Source: ASTD Research Study, “The Value of Evaluation: Making Training Evaluations More Effective,” 2009

81%

Organizations evaluate some programs at Level 2

  • f all programs

evaluated

49% 55%

Organizations view data collected as having high

  • r very high value
slide-5
SLIDE 5

Phillips Associates

5

Why the Disconnect?

Test items too simple or too difficult Test items focus on recall not application Test administered too close to learning event The data collected is not used

slide-6
SLIDE 6

Phillips Associates

6

What’s the Solution?

Create valid, scientifically sound Level 2 knowledge tests that measure job application not just recall

1.

Include knowledge tests as part of learning program design and use for both evaluation and reinforcement

2.

Aggregate knowledge test score data and use to improve program design, demonstrate program value and improve test item validity

3.

slide-7
SLIDE 7

Phillips Associates

7 Source: Donald & James Kirkpatrick, “Evaluating Training Programs: The Four Levels,” 2006.

Data Collection Methods

slide-8
SLIDE 8

Phillips Associates

8

Knowledge Test Overall Goals

Learner Organization

Source: Nanette Minor, “The Art of Test Creation,” T+D, July 2007.

Fair to:

slide-9
SLIDE 9

Phillips Associates

9

6 Test Creation Guidelines

  • 1. Use knowledge tests for both evaluation &

reinforcement plus knowledge checks

  • 2. If concept not taught, don’t test
  • 3. Provide clear instructions
  • 4. Allow test aids if used on job
  • 5. Randomize test items for administration, but

group by topic for scoring

  • 6. Test items should discriminate
Source: Adapted from Nanette Minor, July 2007
slide-10
SLIDE 10

Phillips Associates

10

Sample Level 2 Knowledge Test

slide-11
SLIDE 11

Phillips Associates

11

Instructions 1.

Form a group of 3, 4 or 5 persons

2.

Review sample Level 2 knowledge test in handout and see how many different test creation errors you can find (Hint: 11 different errors are built into the test)

3.

Be prepared to share your findings with the whole group

slide-12
SLIDE 12

Phillips Associates

12

What’s one thing all 16 multiple-choice test questions have in common?

slide-13
SLIDE 13

Phillips Associates

13

They all test for knowledge or recall – the lowest thinking skill level!

slide-14
SLIDE 14

Phillips Associates

14

Bloom’s Taxonomy of Thinking Skills

Knowledge/Recall Comprehension Application Analysis Synthesis Evaluation Low High

slide-15
SLIDE 15

Phillips Associates

15

Examples

What does TV stand for? What is the main function of a TV? What physical principle is used to display images on a TV? Your TV is not working properly. What is the most likely cause of the following set

  • f symptoms:…?
slide-16
SLIDE 16

Phillips Associates

16

Example: Recall

What does the letter “I” in the ADDIE model stand for?

  • A. Initiate
  • B. Implement
  • C. Improve
  • D. Investigate
slide-17
SLIDE 17

Phillips Associates

17

Example: Application

You sent a detailed design document to a group of SMEs to solicit their feedback regarding a program design. This is an example of what element in the ADDIE model?

  • A. Analysis
  • B. Design
  • C. Development
  • D. Implementation
  • E. Evaluation
slide-18
SLIDE 18

Phillips Associates

18

Test Creation Tip 1

Source: ATD Test Design and Delivery certificate program participant workbook, 2006

Test for application not just knowledge or recall.

slide-19
SLIDE 19

Phillips Associates

19

What’s Wrong with This?

  • 10. Effective listening is defined as:
  • A. Attempting to understand the person speaking

from his or her perspective and not your own.

  • B. Encouraging another person to talk.
  • C. Consolidating the important parts of a

discussion.

  • D. Creating a win/win communication situation.
slide-20
SLIDE 20

Phillips Associates

20

Test Creation Tip 2

Keep response options equivalent in length and writing style.

Source: Test Design and Delivery, 2006

slide-21
SLIDE 21

Phillips Associates

21

What’s Wrong with This?

  • 4. A collaborative communication style is also

known as a:

  • A. Win/Win communication style.
  • B. Low concern for self and high concern

for other communication style.

  • C. Engaging and outgoing communication

style.

  • D. Intermediate concern for self and high

concern for other communication style.

slide-22
SLIDE 22

Phillips Associates

22

Test Creation Tip 3

The question should not reveal the correct answer.

Source: Test Design and Delivery, 2006

slide-23
SLIDE 23

Phillips Associates

23

Example

  • 4. A collaborative communication style is also

known as a(n):

  • A. Win/Win communication style.
  • B. Low concern for self and high concern for
  • ther communication style.
  • C. Engaging and outgoing communication

style.

  • D. Intermediate concern for self and high

concern for other communication style.

slide-24
SLIDE 24

Phillips Associates

24

What’s Wrong with This?

  • 14. What communication technique is used

to consolidate what has been previously discussed and to move a conversation from one topic to another?

  • A. Arguing
  • B. Interrupting
  • C. Summarizing
  • D. Initiating
slide-25
SLIDE 25

Phillips Associates

25

Test Creation Tip 4

All response options must be plausible.

Source: Test Design and Delivery, 2006

slide-26
SLIDE 26

Phillips Associates

26

What’s Wrong with This?

  • 6. A benefit of being an effective listener is:
  • A. It facilitates getting to know the other

person better.

  • B. It increases the likelihood of having a

mutual exchange.

  • C. It helps another person get to know

him/herself better.

  • D. All the above.
slide-27
SLIDE 27

Phillips Associates

27

Test Creation Tip 5

Avoid “All the above” or “None of the above”.

Source: Test Design and Delivery, 2006

slide-28
SLIDE 28

Phillips Associates

28

What’s Wrong with This?

  • 12. Which is not one of the four

communication styles?

  • A. Intuitive
  • B. Avoiding
  • C. Aggressive
  • D. Collaborative
slide-29
SLIDE 29

Phillips Associates

29

Test Creation Tip 6

Write test questions the same way the material is taught.

Source: Test Design and Delivery, 2006

slide-30
SLIDE 30

Phillips Associates

30

What’s Wrong with This?

  • 9. A perception check is a:
  • A. Technique for moving a conversation from

the objective, where the focus is on facts, to the subjective, where the focus is on feelings.

  • B. Technique for pulling together the important

parts of one long conversation.

  • C. Technique for helping someone to express

his/her real self.

  • D. Technique for conveying interest in what

another person is saying.

slide-31
SLIDE 31

Phillips Associates

31

Test Creation Tip 7

Include the central idea and most of the phrasing in the stem.

Source: Test Design and Delivery, 2006

slide-32
SLIDE 32

Phillips Associates

32

Example

  • 9. A perception check communication

technique is used to:

  • A. Move a conversation from the objective

to the subjective.

  • B. Pull together the important parts of one

long conversation.

  • C. Help someone express his/her real self.
  • D. Convey interest in what another person

is saying.

slide-33
SLIDE 33

Phillips Associates

33

What’s Wrong with This?

  • 2. ____________________ are used to report

specific, observable actions or details in an

  • bjective, non-evaluative manner.
  • A. Behavior descriptions
  • B. Open probes
  • C. Evaluative statements
  • D. Direct statements
slide-34
SLIDE 34

Phillips Associates

34

Test Creation Tip 8

Write stem as a question – response options can finish question, but don’t use fill-in-the-blank.

Source: Test Design and Delivery, 2006

slide-35
SLIDE 35

Phillips Associates

35

What’s Wrong with This?

  • 1. Interpersonal communication is often

rendered ineffective due to the presence of communication barriers. What are the three categories of communication barriers?

  • A. Environmental, interpersonal and intrapersonal
  • B. Environmental, interpersonal and cultural
  • C. Structural, interpersonal and cultural
  • D. Organizational, intrapersonal and structural
slide-36
SLIDE 36

Phillips Associates

36

Test Creation Tip 9

Eliminate excessive verbiage

  • r irrelevant information

from the stem.

Source: Test Design and Delivery, 2006

slide-37
SLIDE 37

Phillips Associates

37

Example

  • 1. What are the three categories of

communication barriers?

  • A. Environmental, interpersonal and intrapersonal
  • B. Environmental, interpersonal and cultural
  • C. Structural, interpersonal and cultural
  • D. Organizational, intrapersonal and structural
slide-38
SLIDE 38

Phillips Associates

38

What’s Wrong with This?

  • 16. According to Robert Merabian, what

percentage of interpersonal communication is transmitted through voice tone and body language?

  • A. 90 percent
  • B. 50 percent
  • C. 75 percent
  • D. 60 percent
slide-39
SLIDE 39

Phillips Associates

39

Test Creation Tip 10

When possible, present the response options in some logical order (e.g. chronological, most to least or alphabetical).

Source: Test Design and Delivery, 2006

slide-40
SLIDE 40

Phillips Associates

40

Test Creation Tips 11-15

Spread the correct answers across the A-D response choices. Be sure there is only one correct answer. Use at least four response options with each item. Make all response options mutually exclusive. If administering the test electronically, use the diminishing response technique.

Source: Test Design and Delivery, 2006

slide-41
SLIDE 41

Phillips Associates

41

Overall Summary

Develop a Level 2 knowledge test that is:

ü Valid ü Fair to the learner ü Fair to the organization ü Tests for application not recall

slide-42
SLIDE 42

Phillips Associates

42

Test items should be easy for those who understand what you taught, difficult for those who don’t understand, and a total mystery to the uninitiated.

Source: Odin Westegaard

slide-43
SLIDE 43

Phillips Associates

43
slide-44
SLIDE 44

Phillips Associates

44

Phillips, Ken, “Eight Tips on Developing Valid Level 1 Evaluation Forms”, Training Today, Fall 2007, pps. 8 & 14. Phillips, Ken, “Developing Valid Level 2 Evaluations”, Training Today, Fall 2009, pps. 6-8. Phillips, Ken, “Capturing Elusive Level 3 Data: The Secrets of Survey Design”, Unpublished article, 2013. Phillips, Ken, “Level 1 Evaluations: Do They Have a Role in Organizational Learning Strategy?”, Unpublished article, 2013. Phillips, Ken, “Business Results Made Visible: Designing Proof Positive Level 4 Evaluations”, Unpublished article, 2013.

Free Articles

slide-45
SLIDE 45

Phillips Associates

45

Phillips Associates ken@phillipsassociates.com (847) 231-6068 www.phillipsassociates.com 34137 N. Wooded Glen Drive Grayslake, Illinois 60030

Ken Phillips