Test-item Writing Workshop: The Power of Peer-review October 4, - - PowerPoint PPT Presentation

Test-item Writing Workshop: The Power

• f Peer-review

Fidel Lim, DNP, CCRN

October 4, 2017

• Identify 4 flaws in test questions
• Utilize best practices for writing

multiple choice questions

• Utilize best practices when writing

alternate items

• Correctly classify questions based
• n Bloom’s Taxonomy

1/12/2018 2

Objectives

• An effective test item:
• Discriminates students who understand the content

from those who do not.

• Focuses on important information.

3

What are we testing?

• 1. Test-wise flaws - Flaws that make it easier

for “test-wise” students to answer the question correctly, giving them an unfair advantage over students who are not test-

• wise. (3%)

(Option length, grammatical errors in options that are not correct answer, repetition of key word in correct answer, “all of the above”)

Presentation Title Goes Here 4

Four Categories of Test-item Flaws

• 2. Irrelevant difficulty - Flaws in either the stem or
• ptions that make items difficult to understand for

reasons unrelated to the content or focus of the assessment.

Difficult to understand = completion format

Four Categories of Test-item Flaws

1/12/2018 Presentation Title Goes Here 5

• 3. Linguistic/structural bias - Unnecessary

linguistic complexity in the stem or options, grammatical errors, and lack of clarity or consistency in the wording. Four Categories of Test-item Flaws

1/12/2018 Presentation Title Goes Here 6

• 4. Cultural bias - Use of culturally specific information

that is not equally available to all cultural groups.

(Questions with information not equally available to all cultural groups – “gin & tonic”)

Four Categories of Test-item Flaws

1/12/2018 Presentation Title Goes Here 7

• A practice analysis study aims to investigate the

applicability of the 2013 NCLEX-RN Test Plan. This periodic performance of practice analyses (i.e., job analysis) studies assists NCSBN in evaluating the validity of the test plan that guides content distribution

• f the licensure examination.

https://www.ncsbn.org/15_RN_Practice_Analysis_Vol62_ web.pdf

2014 RN Practice Analysis: Linking the NCLEX-RN Examination to Practice

1/12/2018 Presentation Title Goes Here 8

Lowest percentage of applicability

• “Provide care and education to a client in labor and

antepartum client” (21.5%),

• “Provide post-partum care and education” (23.0%)
• “Provide care and education for the newborn less than

1 month old through the infant or toddler client through 2 years” (26.7%).

Applicability of Activities to Practice Setting

1/12/2018 Presentation Title Goes Here 9

The activities with the highest percentage of applicability:

• “Organize workload to manage time effectively” (100.0%)
• “Apply principles of infection control (e.g., hand hygiene,

surgical asepsis, isolation, sterile technique, universal/standard precautions)” (100.0%).

Applicability of Activities to Practice Setting

1/12/2018 Presentation Title Goes Here 10

Setting-Specific The activities performed with the lowest frequencies:

• “Implement emergency response plans (e.g., internal/external

disaster, bomb threat, community planning)” (0.67),

• “Implement and monitor phototherapy” (0.67),
• “Perform hemodialysis or continuous renal replacement therapy”

(0.68), and

• “Perform and manage care of client receiving peritoneal dialysis”

(0.70).

Frequency of Activity Performance

1/12/2018 Presentation Title Goes Here 11

Setting-Specific

The activities performed with the highest frequencies:

• “Apply principles of infection control (e.g., hand hygiene, surgical

asepsis, isolation, sterile technique, universal/ standard precautions)” (4.94),

• “Provide care within the legal scope of practice” (4.82), and
• “Ensure proper identification of client when providing care” (4.82).

Frequency of Activity Performance

1/12/2018 Presentation Title Goes Here 12

Total Group The activities performed with the lowest frequency were

• “Implement and monitor phototherapy” (0.20),
• “Perform hemodialysis or continuous renal

replacement therapy” (0.22), and

• “Provide post-partum care and education (0.31).

Frequency of Activity Performance

1/12/2018 Presentation Title Goes Here 13

Total Group

The activities performed with the highest frequencies were:

• “Apply principles of infection control (e.g., hand hygiene,

surgical asepsis, isolation, sterile technique, universal/ standard precautions)” (4.94),

• “Provide care within the legal scope of practice” (4.82), and
• “Ensure proper identification of client when providing care”

(4.80).

Frequency of Activity Performance

1/12/2018 Presentation Title Goes Here 14

Setting-Specific The activities with the lowest importance ratings were:

• “Recognize complementary therapies and identify potential

contraindications (e.g., aromatherapy, acupressure, supplements)” (3.52),

• “Plan and/or participate in community health education” (3.55),

and

• “Perform post-mortem care” (3.73).

Importance of Activity Performance

1/12/2018 Presentation Title Goes Here 15

Setting-Specific The activities with the highest importance ratings were:

• “Apply principles of infection control (e.g., hand hygiene,

surgical asepsis, isolation, sterile technique, universal/standard precautions)” (4.91),

• “Prepare and administer medications, using rights of

medication administration” (4.89), and

• “Ensure proper identification of client when providing care”

(4.88).

Importance of Activity Performance

1/12/2018 Presentation Title Goes Here 16

Total Group The activities with the lowest importance ratings were:

• “Recognize complementary therapies and identify potential

contraindications (e.g., aromatherapy, acupressure, supplements)” (3.31),

• “Plan and/or participate in community health education” (3.37),

and

• “Implement and monitor phototherapy” (3.42).

Importance of Activity Performance

1/12/2018 Presentation Title Goes Here 17

Total Group

The activities with the highest importance ratings were:

• “Apply principles of infection control (e.g., hand hygiene,

surgical asepsis, isolation, sterile technique, universal/standard precautions)” (4.91),

• “Ensure proper identification of client when providing care”

(4.88), and

• “Prepare and administer medications, using rights of

medication administration” (4.88).

Importance of Activity Performance

1/12/2018 Presentation Title Goes Here 18

Remembering: Can the student recall or remember the information? define, duplicate, list, memorize, recall, repeat, reproduce state Understanding: Can the student explain ideas or concepts? classify, describe, discuss, explain, identify, locate, recognize, report, select, translate, paraphrase Applying: Can the student use the information in a new way? choose, demonstrate, dramatize, employ, illustrate, interpret, operate, schedule, sketch, solve, use, write. Analyzing: Can the student distinguish between the different parts? appraise, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, test.

Bloom’s Taxonomy

Remembering Level

• Requires recall or recognition of information

previously learned.

• Question requires no understanding or judgment.
• Good for testing factual information.
• Limited value for testing nursing competence.

Understanding Level

• Evaluates a student’s understanding of the information.

Examples

• Pathophysiology (e.g., you do this because …, why do

you do something) Behavioral objective:

• Explain the cause of complications of diagnostic

procedures for the liver.

Applying Level

• Requires students to apply their knowledge in a situation.
• Tests ability to use learned concepts in specific situations.
• This is where you are asking the student to make some sort of

judgment. Examples:

• Evaluating client responses or statements.
• Determining which client situation requires immediate action.

Analyzing Level

• Requires students to break down information to
• btain the meaning of the material or demonstrate

how it is organized or structured.

• Requires students to analyze the situation, interpret

data, set goals, establish nursing diagnoses, set priorities, and determine actions.

• This is a multiple step process – the student has to

figure out what’s going on with the patient and then what to do about it.

Question Stem Should Should not:

Contain a subject and a verb Contain irrelevant information CAPITALIZE, italicize, or underline qualifier word to support the critical thinking process “Teach" information Always be a client and a nurse

Write the Stem: Example Stem

Joe, aged 25, is admitted to the hospital with a ruptured

• appendix. His temperature is 101°F and WBC is

15,000. During his post-op recovery phase Joe receives TPN and complains of thirst and frequent urination. This complaint most likely indicates:

Example Stem

Good - The nurse cares for the client receiving TPN. The patient reports increased thirst and frequent urination. These symptoms MOST likely indicate which complication?

Write the Correct Answer/s

• Write and validate the correct response/s
• Vary the position of the correct response/s
• Should be single action

Answer Choices Should Should not:

Eliminate unnecessary words. Contain key words or phrases related to information in the stem. Enable students to choose the correct answer because they know the information tested, not because of the way answer choices are written. Vary in length, vocabulary, or syntax. Be mutually exclusive. One answer choice should not be included in another. Include rationale for nurse’s action

Distracters

• Can be diagnostic and provide information about the

student's lack of knowledge.

• Can be a predictable error, a common misconception,

a clinical error, an incomplete but related idea, or a common procedural mistake.

Distracters

Should Should not:

Be realistic and plausible. Be close in meaning to the correct response. Be the same length and grammatically similar. Be direct opposites of distracters or the correct answer. Be consistent with the correct answer. Be absolute: such as always or never. Be homogenous. Be made-up “garbage” Be inviting and well written, but not tricky! Contain rationale for answer

Write the Rationales

• Explain why the correct response is correct
• Explain why the incorrect response is

incorrect

Rationales

Should Should not:

Explain why the correct response is correct Say “not the best/correct response” Explain why the incorrect response is incorrect Teach about the topic Present information essential to answering the question

• Step 1 – Select topic
• Step 2 – Write the Stem
• Step 3 – Write the Answer/s
• Step 4 – Write the Distracters
• Step 5 – Write rationales

To Review: Question Construction Concepts

Creating Higher Level Questions

• Remember Bloom’s verbs
• Write questions that ask for “best”, “most appropriate”

action

• Requires at least two steps to answer the question

Creating Higher Level Questions

• In the stem of the question, give clinical assessment

data rather than the diagnosis. This requires students to analyze the data to answer the question. Example:

• Describe symptoms of pulmonary edema rather than

state "client with pulmonary edema...”

• “The patient reports shortness of breath…”

Creating Higher Level Questions

• Write questions that include integrated concepts.

Examples:

• Diabetic woman in postpartum situation
• Pediatric patient in surgery
• Elderly patient receiving medication

Creating Higher Level Questions

Sample Item Stems

• Which of these questions would be most important

for a nurse to ask a client who has XX?

• To identify whether a client is developing (side effect
• f medication, complication of a procedure,

progression of a disease), which of these questions should a nurse ask the client? Which of these assessments should a nurse make?

McDonald, M. (2007). A nurse educator’s guide to assessing learning outcomes.

Creating Higher Level Questions

Sample Item Stems

• A nurse identifies that a client who has a (XX)

demonstrate (manifestations, lab values). This finding supports which priority nursing diagnosis?

• Which outcome is most appropriate to establish for

the client diagnosed with … (nursing diagnosis)?

McDonald, M. (2007). A nurse educator’s guide to assessing learning outcomes.

Creating Higher Level Questions

Sample Item Stems

• A nurse observes a (colleague) including all these

measures when (performing a procedure). Which one would require the nurse to intervene? Which of these actions is most appropriate for the nurse to take?

• When a client has a (dx) is being treated with

(medication, treatment), which of these (manifestations, lab data) would indicate that the client’s condition is (improving, worsening)?

McDonald, M. (2007). A nurse educator’s guide to assessing learning outcome.

General Rules

• Avoid bias by gender.
• Use a description of the patient without a proper name.
• Write questions at appropriate grade reading level. (PN

textbooks 10th – 11th grade level; RN textbooks 11th - 13th: varies by book).

• Review each other’s questions.
• Write in the active voice and put a patient and nurse in

stem.

Alternate Format Questions

• Multiple response - select all that apply, rank in order
• Hot spots
• Fill-in-the-blank
• Chart exhibit
• Drag-and-drop
• Audio and video

Multiple Response

• Don’t include an answer in other answers.
• Don’t test priorities or “best”.
• Do keep question broad to encompass several correct

answers.

• Don’t present one correct or all correct.
• Do monitor placement of correct answers.
• Do change existing priority questions.
• Do include multiple wrong answers.

1 The nurse starts an IV with a 19-gauge needle. 2 The nurse begins the transfusion at a slow rate. 3 The nurse obtains a history of transfusion reactions. 4 The nurse obtains the blood product from the blood bank.

Rank in Order

The nurse administers packed cells to the client. In which order should the nurse perform the actions?

• Rank in order from first activity to last activity.
• Use all answer choices.

The nurse adm inisters packed cells to the client. In which order should the nurse perform the actions? Rank in order from first activity to last activity. Answer:

3 The nurse obtains a history of transfusion reactions. 1 The nurse starts an IV with a 19-gauge needle. 4 The nurse obtains the blood product from the blood bank. 2 The nurse begins the transfusion at a slow rate.

Rank in Order

The nurse reviews the records of clients in the m edical clinic who are at risk for developing type 2 diabetes. In which order will the nurse assess the clients? Arrange the following clients in order from greatest risk to least risk for developing type 2 diabetes. Use all answer choices.

1 A 26-year-old African American with a history of gestational diabetes and following a weight reduction diet. 2 A 36-year-old Caucasian with a serum triglyceride level of 310 mg/dL (1.24 mmol/L) and whose mother-in-law has type 1 diabetes. 3 A 42-year-old Hispanic American with a history of hypertension and a serum high- density lipoprotein (HDL) level of 30 mg/dL (0.78 mmol/L). 4 A 56-year-old Native American who is 5' 8" tall, weighs 200 lb, and has two siblings with type 2 diabetes.

Rank in Order

The nurse reviews the records of clients in the medical clinic who are at risk for developing type 2 diabetes. In which order will the nurse assess the clients?

Answer:

4 A 56-year-old Native American who is 5' 8" tall, weighs 200 lb, and has two siblings with type 2 diabetes. 3 A 42-year-old Hispanic American with a history of hypertension and a serum high-density lipoprotein (HDL) level of 30 mg/dL (0.78 mmol/L). 1 A 26-year-old African American with a history of gestational diabetes and following a weight reduction diet. 2 A 36-year-old Caucasian with a serum triglyceride level of 310 mg/dL (1.24 mmol/L) and whose mother-in-law has type 1 diabetes.

Rank in Order

Hot Spots

• Involve anatomy and physiology
• Identify front and back
• Determine margin of error

Fill-in-the-Blank

• Usually math calculation
• Give rounding rules up front
• Make decisions about spelling errors or synonyms

Drag-and-Drop

• Use all options.
• Link two columns with arrows.
• Pulls first option to the top of the ordered options list.

Categories - NCLEX

• Safe and Effective Care
• Management of Care
• Safety and Infection Control
• Health Promotion and Maintenance
• Psychosocial Integrity
• Physiological Integrity
• Basic Care and Comfort
• Pharmacological/Parenteral Therapies
• Reduction of Risk Potential
• Physiological Adaptation

Categories – Nursing Process

• Assessment
• Diagnose
• Planning
• Implementation
• Evaluation

The nurse collaborates with the unlicensed assistive personnel (UAP) to provide care for a patient with heart failure. Which instructions should the nurse provide to the UAP when delegating care for this patient? Select all that apply.

• A. Reposition the patient every 2 hours.
• B. Teach the patient to perform deep breathing exercises.
• C. Accurately record intake and output.
• D. Use the same scale each morning to weigh the patient.
• E. Place patient on oxygen in case of respiratory distress.
• F. Notify the nurse if the patient’s vital signs are abnormal.

Sample Question

1/12/2018 Presentation Title Goes Here 53

The nurse collaborates with the unlicensed assistive personnel (UAP) to provide care for a patient with heart failure. Which instructions should the nurse provide to the UAP when delegating care for this patient? Select all that apply.

• A. Reposition the patient every 2 hours.
• B. Teach the patient to perform deep breathing exercises.
• C. Accurately record intake and output.
• D. Use the same scale each morning to weigh the patient.
• E. Place patient on oxygen in case of respiratory distress.
• F. Notify the nurse if the patient’s vital signs are abnormal.

Sample Question

1/12/2018 Presentation Title Goes Here 54

The UAP can reposition the patient every 2 hours to improve

• xygenation and prevent atelectasis. The UAP can also accurately

record intake and output, and use the same scale to weigh the patient each morning before breakfast. UAPs are not qualified to teach patients or assess the need for and provide oxygen therapy. However, they can notify the RN if there are abnormalities in vital signs and changes in patient’s condition. Bloom’s Category: Analysis. Content Category/Topic: Delegation.

Rationale

1/12/2018 Presentation Title Goes Here 55

The family of a neutropenic patient reports the patient “is not acting right.” Which of these actions should be the nurse’s priority?

• A. Ask the patient about pain.
• B. Assess the patient for infection.
• C. Delegate taking a set of vital signs
• D. Review the patient’s food intake.

Sample Question

1/12/2018 Presentation Title Goes Here 56

The family of a neutropenic patient reports the patient “is not acting right.” Which of these actions should be the nurse’s priority?

• A. Ask the patient about pain.
• B. Assess the patient for infection.
• C. Delegate taking a set of vital signs
• D. Review the patient’s food intake.

Sample Question

1/12/2018 Presentation Title Goes Here 57

Neutropenic patients often do not have classic manifestations of infection, but infection is the most common cause of death in neutropenic patients. The nurse should assess for signs and symptoms of infection. The nurse should assess for pain but this is not the priority. The nurse should take the patient’s vital signs instead

• f delegating them since the patient has had a change in status.

Reviewing the patient’s food intake is part of comprehensive assessment, but it is not the priority in the scenario described. Bloom’s Category: Analysis. Content Category/Topic: Neutropenia

Rationale

1/12/2018 Presentation Title Goes Here 58

A nurse assesses a patient with diabetes mellitus and notes the patient only responds to a sternal rub by moaning, has fingerstick blood glucose of 33 mg/dL, and has no intravenous line. Which action should the nurse take first?

• A. Administer 1 mg of intramuscular glucagon
• B. Encourage the patient to drink orange juice.
• C. Insert a new intravenous (IV) access line.
• D. Administer 25 mL dextrose 50% (D50) IV push.

Sample Question

1/12/2018 Presentation Title Goes Here 59

A nurse assesses a patient with diabetes mellitus and notes the patient only responds to a sternal rub by moaning, has fingerstick blood glucose of 33 mg/dL, and has no intravenous line. Which action should the nurse take first?

• A. Administer 1 mg of intramuscular glucagon
• B. Encourage the patient to drink orange juice.
• C. Insert a new intravenous (IV) access line.
• D. Administer 25 mL dextrose 50% (D50) IV push.

Sample Question

1/12/2018 Presentation Title Goes Here 60

Treatment of severe hypoglycemia in a patient with altered mental status who has not IV access is to give an IM injection of glucagon. A patient with decreased mental status will not be able to drink orange

• juice. Inserting an IV line would delay correction of the critically low

blood sugar of 33 g/dL. If the patient has no IV access, the nurse won't be able to give the 50% dextrose. Bloom’s Category:

• Application. Content Category/Topic: Hypoglycemia
• Rationale

1/12/2018 Presentation Title Goes Here 61

The patient takes beclomethasone by metered dose inhaler (MDI). Which statement made by the patient indicates that teaching is successful?

• A. "I know it is time to have the prescription refilled when the

canister floats in water."

• B. "I will rinse my mouth and throat with water after each dose."
• C. "I will be sure not to shake the canister before I use it."
• D. "If the dose does not help, I will take extra and let the health care

provider know the results."

Sample Question

1/12/2018 Presentation Title Goes Here 62

The patient takes beclomethasone by metered dose inhaler (MDI). Which statement made by the patient indicates that teaching is successful?

• A. "I know it is time to have the prescription refilled when the

canister floats in water."

• B. "I will rinse my mouth and throat with water after each dose."
• C. "I will make sure not to shake the canister before I use it."
• D. "If the dose does not help, I will take extra and let the healthcare

provider know the results."

Sample Question

1/12/2018 Presentation Title Goes Here 63

Beclomethasone is a corticosteroid and inhaled corticosteroids can predispose to fungal oropharynx infection (candidiasis). Rinsing the mouth and gargling with warm water when each treatment is completed is imperative to remove residual medication and prevent onset of infection. A more accurate method of calculating doses is to count them and refill the prescription when the canister is nearly

• empty. The canister should be shaken thoroughly before use to disperse and mix

the medication with the propellant. The medication should be used exactly as it was prescribed. If the does is not effective, the healthcare provider (HCP) is

• consulted. An overdose could result in hypertension, palpitations, angina, and
• dysrhythmias. Bloom’s Category: Application. Content Category/Topic: Inhalers.

Rationale

1/12/2018 Presentation Title Goes Here 64

Questions?