Take Your Level 2s Up a Notch: Start Measuring Application Not - PowerPoint PPT Presentation

Take Your Level 2s Up a Notch: Start Measuring Application Not Just Recall Presented by: Ken Phillips Phillips Associates February 9, 2016 Phillips Associates 1

Agenda 1. Examine Level 2 evaluation facts 2. Analyze test creation errors in a sample Level 2 knowledge test 3. Examine the difference between test questions that measure application versus those that measure recall 4. Discover 15 tips for creating valid, scientifically sound multiple-choice test questions Phillips Associates 2

Kirkpatrick / Phillips Evaluation Model Levels of Evaluation Measurement Focus Time Frame Participant favorable reaction Conclusion of learning Level 1: Reaction to a learning program program Degree to which participants Conclusion of learning Level 2: Learning acquired new knowledge, skills program or within 6 to 8 or attitudes weeks after Degree to which participants applied Level 3: Behavior 2 to 12 months back-on-the-job what was learned Degree to which targeted business Level 4: Results 9 to 18 months outcomes were achieved Degree to which monetary program Level 5: ROI 9 to 18 months benefits exceed program costs Phillips Associates 3

Level 2 Evaluation Facts Organizations Organizations view 81% 49% 55% evaluate some of all programs data collected programs evaluated as having high at Level 2 or very high value Source: ASTD Research Study, “The Value of Evaluation: Making Training Evaluations More Effective,” 2009 Phillips Associates 4

Why the Disconnect? Test items too simple or too difficult Test items focus on recall not application Test administered too close to learning event The data collected is not used Phillips Associates 5

What’s the Solution? Create valid, scientifically sound Level 2 1. knowledge tests that measure job application not just recall Include knowledge tests as part of learning 2. program design and use for both evaluation and reinforcement Aggregate knowledge test score data and use 3. to improve program design, demonstrate program value and improve test item validity Phillips Associates 6

Data Collection Methods Source: Donald & James Kirkpatrick, “Evaluating Training Programs: The Four Levels,” 2006. Phillips Associates 7

Knowledge Test Overall Goals Fair to: Learner Organization Source: Nanette Minor, “The Art of Test Creation,” T+D , July 2007. Phillips Associates 8

6 Test Creation Guidelines 1. Use knowledge tests for both evaluation & reinforcement plus knowledge checks 2. If concept not taught, don’t test 3. Provide clear instructions 4. Allow test aids if used on job 5. Randomize test items for administration, but group by topic for scoring 6. Test items should discriminate Source: Adapted from Nanette Minor, July 2007 Phillips Associates 9

Sample Level 2 Knowledge Test Phillips Associates 10

Instructions Form a group of 3, 4 or 5 persons 1. Review sample Level 2 knowledge test in 2. handout and see how many different test creation errors you can find ( Hint : 11 different errors are built into the test) Be prepared to share your findings with the 3. whole group Phillips Associates 11

What’s one thing all 16 multiple-choice test questions have in common? Phillips Associates 12

They all test for knowledge or recall – the lowest thinking skill level! Phillips Associates 13

Bloom’s Taxonomy of Thinking Skills High Evaluation Synthesis Analysis Application Comprehension Knowledge/Recall Low Phillips Associates 14

Examples What does TV stand for? What is the main function of a TV? What physical principle is used to display images on a TV? Your TV is not working properly. What is the most likely cause of the following set of symptoms:…? Phillips Associates 15

Example: Recall What does the letter “I” in the ADDIE model stand for? A. Initiate B. Implement C. Improve D. Investigate Phillips Associates 16

Example: Application You sent a detailed design document to a group of SMEs to solicit their feedback regarding a program design. This is an example of what element in the ADDIE model? A. Analysis B. Design C. Development D. Implementation E. Evaluation Phillips Associates 17

Test Creation Tip 1 Test for application not just knowledge or recall. Source: ATD Test Design and Delivery certificate program participant workbook, 2006 Phillips Associates 18

What’s Wrong with This? 10. Effective listening is defined as: A. Attempting to understand the person speaking from his or her perspective and not your own. B. Encouraging another person to talk. C. Consolidating the important parts of a discussion. D. Creating a win/win communication situation. Phillips Associates 19

Test Creation Tip 2 Keep response options equivalent in length and writing style. Source: Test Design and Delivery , 2006 Phillips Associates 20

What’s Wrong with This? 4. A collaborative communication style is also known as a: A. Win/Win communication style. B. Low concern for self and high concern for other communication style. C. Engaging and outgoing communication style. D. Intermediate concern for self and high concern for other communication style. Phillips Associates 21

Test Creation Tip 3 The question should not reveal the correct answer. Source: Test Design and Delivery , 2006 Phillips Associates 22

Example 4. A collaborative communication style is also known as a(n): A. Win/Win communication style. B. Low concern for self and high concern for other communication style. C. Engaging and outgoing communication style. D. Intermediate concern for self and high concern for other communication style. Phillips Associates 23

What’s Wrong with This? 14. What communication technique is used to consolidate what has been previously discussed and to move a conversation from one topic to another? A. Arguing B. Interrupting C. Summarizing D. Initiating Phillips Associates 24

Test Creation Tip 4 All response options must be plausible. Source: Test Design and Delivery , 2006 Phillips Associates 25

What’s Wrong with This? 6. A benefit of being an effective listener is: A. It facilitates getting to know the other person better. B. It increases the likelihood of having a mutual exchange. C. It helps another person get to know him/herself better. D. All the above. Phillips Associates 26

Test Creation Tip 5 Avoid “All the above” or “None of the above”. Source: Test Design and Delivery , 2006 Phillips Associates 27

What’s Wrong with This? 12. Which is not one of the four communication styles? A. Intuitive B. Avoiding C. Aggressive D. Collaborative Phillips Associates 28

Test Creation Tip 6 Write test questions the same way the material is taught. Source: Test Design and Delivery , 2006 Phillips Associates 29

What’s Wrong with This? 9. A perception check is a: A. Technique for moving a conversation from the objective, where the focus is on facts, to the subjective, where the focus is on feelings. B. Technique for pulling together the important parts of one long conversation. C. Technique for helping someone to express his/her real self. D. Technique for conveying interest in what another person is saying. Phillips Associates 30

Test Creation Tip 7 Include the central idea and most of the phrasing in the stem. Source: Test Design and Delivery , 2006 Phillips Associates 31

Example 9. A perception check communication technique is used to: A. Move a conversation from the objective to the subjective. B. Pull together the important parts of one long conversation. C. Help someone express his/her real self. D. Convey interest in what another person is saying. Phillips Associates 32

What’s Wrong with This? 2. ____________________ are used to report specific, observable actions or details in an objective, non-evaluative manner. A. Behavior descriptions B. Open probes C. Evaluative statements D. Direct statements Phillips Associates 33

Test Creation Tip 8 Write stem as a question – response options can finish question, but don’t use fill-in-the-blank. Source: Test Design and Delivery , 2006 Phillips Associates 34

What’s Wrong with This? 1. Interpersonal communication is often rendered ineffective due to the presence of communication barriers. What are the three categories of communication barriers? A. Environmental, interpersonal and intrapersonal B. Environmental, interpersonal and cultural C. Structural, interpersonal and cultural D. Organizational, intrapersonal and structural Phillips Associates 35

Test Creation Tip 9 Eliminate excessive verbiage or irrelevant information from the stem. Source: Test Design and Delivery , 2006 Phillips Associates 36

Example 1. What are the three categories of communication barriers? A. Environmental, interpersonal and intrapersonal B. Environmental, interpersonal and cultural C. Structural, interpersonal and cultural D. Organizational, intrapersonal and structural Phillips Associates 37

What’s Wrong with This? 16. According to Robert Merabian, what percentage of interpersonal communication is transmitted through voice tone and body language? A. 90 percent B. 50 percent C. 75 percent D. 60 percent Phillips Associates 38