Pedagogical Best Practices: Pedagogical Best Practices: The Power - - PowerPoint PPT Presentation

▶
pedagogical best practices pedagogical best practices the
SMART_READER_LITE
LIVE PREVIEW

Pedagogical Best Practices: Pedagogical Best Practices: The Power - - PowerPoint PPT Presentation

Sep 26, 2023 16 likes •389 views

Pedagogical Best Practices: Pedagogical Best Practices: The Power of Quizzing The Power of Quizzing Gordon Hodge Gordon Hodge University of New Mexico & University of New Mexico & The National Center for Academic Transformation The

slide-1
SLIDE 1

1

Pedagogical Best Practices: The Power of Quizzing Pedagogical Best Practices: The Power of Quizzing

Gordon Hodge

University of New Mexico & The National Center for Academic Transformation

Gordon Hodge

University of New Mexico & The National Center for Academic Transformation

The Redesign Alliance Third Annual Conference

Orlando • March 24, 2009

The Redesign Alliance Third Annual Conference

Orlando • March 24, 2009

Acknowledgements Institute of Education Sciences Grant #R305H030339 NCAT Pew Grant Program in Redesign Acknowledgements Institute of Education Sciences Grant #R305H030339 NCAT Pew Grant Program in Redesign

slide-2
SLIDE 2

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

2

Intro Psych 105 is the largest of UNM’s 20 “killer” courses for lower division undergraduate students. Before the redesign, the drop, withdrawal, failure rate was 42%.

slide-3
SLIDE 3

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

3

While maintaining or exceeding standards …

Improve grades (but without grade inflation) Use merit system (no curving) Comprehensive coverage (all of textbook) Achieve consistency across sections Increase student time on task

Problem

slide-4
SLIDE 4

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

4

¸ Quizzes Required ¸ 3,000 Question Pool ¸ 3 Quizzes per week ¸ 20 Questions per quiz ¸ Each Quiz different ¸ Timed & Deadlines ¸ Unlimited repeats ¸ Highest score counts ¸ 1 Point per question ¸ 1 Exam per week ¸ Quizzes Required ¸ 3,000 Question Pool ¸ 3 Quizzes per week ¸ 20 Questions per quiz ¸ Each Quiz different ¸ Timed & Deadlines ¸ Unlimited repeats ¸ Highest score counts ¸ 1 Point per question ¸ 1 Exam per week

Solution: Lots of WebCT Quizzes

slide-5
SLIDE 5

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

5

Instructions to Students

Quizzes are not just a way to review or to test yourself. Quizzes are an important way to learn the material in the first place.

slide-6
SLIDE 6

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

6

Quiz Resistance Quiz Resistance

“It’s weird to use a quiz to study.” “You don’t take a quiz to study, you study first and then you take a quiz!” “It’s weird to use a quiz to study.” “You don’t take a quiz to study, you study first and then you take a quiz!”

Traditionally, what does it mean to study? Traditionally, what does it mean to study?

slide-7
SLIDE 7

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

7

  • Attend Lectures
  • Read Textbook
  • Review Notes
  • Attend Lectures
  • Read Textbook
  • Review Notes
  • How do you know what’s important?
  • How do you know you’ll remember it?

But . . . But . . .

Traditionally, what does it mean to study? Traditionally, what does it mean to study?

7

slide-8
SLIDE 8

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

8

Active Learning + Practice

What should it mean to study? What should it mean to study?

slide-9
SLIDE 9

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

9

Results

slide-10
SLIDE 10

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

10

Do Quizzes Matter?

  • Take quizzes many times (5-15x)
  • Get perfect or near perfect scores
  • Complete later quizzes in a set in under 5 min
  • Take quizzes many times (5-15x)
  • Get perfect or near perfect scores
  • Complete later quizzes in a set in under 5 min

Students who do well on exams typically: Students who do well on exams typically:

slide-11
SLIDE 11

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

11

Do Quizzes Matter?

r = .93

Fall 2005: Correlations Fall 2005: Correlations

Figure 1. High quiz scores predicted high exam scores (n = 847 students).

slide-12
SLIDE 12

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

12

Do Quizzes Matter?

Average Score (80 Questions)

Fall 2005: Pretest vs. Posttest Fall 2005: Pretest vs. Posttest

Figure 2. Students took an 80-question Pretest during the first week

  • f the semester and the same test as the Final Exam. Students

who performed well in the course performed better on the Posttest Final than students who performed less well.

slide-13
SLIDE 13

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

13

Do Quizzes Matter?

Average Score (80 Questions)

Fall 2008: Pretest vs. Posttest Fall 2008: Pretest vs. Posttest

Figure 3. Students took an 80-question Pretest during the first week

  • f the semester and the same test as the Final Exam.

Pre- vs. Posttest Percentage F08 • All Sections Percentage Pretest Posttest (Final Exam)

slide-14
SLIDE 14

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

14

Do Quizzes Matter?

Percentage of Students

Fall 2005: Ethnicity Fall 2005: Ethnicity

Figure 4. Students of different ethnic backgrounds performed comparably.

slide-15
SLIDE 15

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

15

Do Quizzes Matter?

Fall 2000 – Fall 2008: 16 Semesters Fall 2000 – Fall 2008: 16 Semesters

slide-16
SLIDE 16

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

16

Do Required Quizzes Matter?

Figure 6. Patterns of Quiz Taking (left). When quizzes were required, students scored higher, took them more times, and for longer durations than when quizzes were optional.

Percentage of Students

(Figure 5)

High Score

Quizzes Optional

Number of Attempts

Weekly Mastery Quizzes Weekly Mastery Quizzes

Quizzes Required

Mean Duration (min)

Spring 2002: Two Sections/Same Instructor Spring 2002: Two Sections/Same Instructor

slide-17
SLIDE 17

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

17

Do Required Quizzes Matter?

Spring 2005: Two Sections/Different Instructors Spring 2005: Two Sections/Different Instructors

Course Grade Percentage of Students Figure 7. In spring 2005, to determine whether quizzes need to be required in order for students to perform well in the class, two sections were taught by different instructors. In Sec 001, quizzes were required; in Sec 002, quizzes were available but optional (i.e., students were encouraged to take them, but they received no points for doing so). Students performed better when quizzes were required (Sec 001) than when quizzes were

  • ptional (Sec 002).
slide-18
SLIDE 18

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

18

Do Required Quizzes Matter?

Spring 2007: Alternating Weeks—Rqd vs Opt Qzs Spring 2007: Alternating Weeks—Rqd vs Opt Qzs

Figure 8. In weeks when quizzes were required, students performed better across sections than when quizzes were optional. *p < .05 or better

Required

* * * * * * * * * * * * * * * *

Required Required Required Required Required Required Required Required Required Required Required Required Required Required Required

Section 001 Quizzes Required on Odd # Exam Days Section 002 Quizzes Required on Even # Exam Days

Required

slide-19
SLIDE 19

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

19

Are students just memorizing the questions? Are students just memorizing the questions?

Do Quiz Questions Matter?

slide-20
SLIDE 20

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

20

QUIZ STEM — In elementary school and high school, Charlie got away with copying his test answers from classmates. Because the college has test proctors who are very observant, Charlie spends as many hours devising new ways to cheat as it would take him to study and perform well in an honest fashion. Charlie's strategy for passing tests illustrates the consequences of:

  • functional fixedness.
  • a mental set.
  • confirmation bias.
  • the availability heuristic.

What students see Do Quiz Questions Matter?

slide-21
SLIDE 21

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

21

QUIZ STEM — In elementary school and high school, Charlie got away with copying his test answers from classmates. Because the college has test proctors who are very

  • bservant, Charlie spends as many hours devising new ways to cheat as it would take him

to study and perform well in an honest fashion. Charlie's strategy for passing tests illustrates the consequences of:

  • functional fixedness.
  • a mental set.
  • confirmation bias.
  • the availability heuristic.

What students remember? Do Quiz Questions Matter?

Charlie = mental set

slide-22
SLIDE 22

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

22

TEST STEM — A mental set is a:

  • methodical step-by-step procedure for solving problems.
  • mental grouping of similar objects, events, or people.
  • tendency to approach a problem in a way that has been successful in the past.
  • group of conclusions derived from certain assumptions or general principles.

QUIZ STEM — In elementary school and high school, Charlie got away with copying his test answers from classmates. Because the college has test proctors who are very observant, Charlie spends as many hours devising new ways to cheat as it would take him to study and perform well in an honest fashion. Charlie's strategy for passing tests illustrates the consequences of:

  • functional fixedness.
  • a mental set.
  • confirmation bias.
  • the availability heuristic.

Do Quiz Questions Matter? Use different Exam questions Use different Exam questions

slide-23
SLIDE 23

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

23

Do Required Quizzes Matter?

Spring 2007: Alternating Weeks—Rqd vs Opt Qzs Spring 2007: Alternating Weeks—Rqd vs Opt Qzs

Figure 9. Difference in performance between same- or different-stem questions. Performance was different when quiz and exam stems were very different from each other. *p < .05

* * * * * * * * * * * * * * * *

Sections 001 & 002 SAME Quiz & Exam Stems Sections 001 & 002 DIFFERENT Quiz & Exam Stems

slide-24
SLIDE 24

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

24

To determine whether studying by taking quizzes versus going to lectures and reading the textbook differentially affects exam performance, students were divided into two groups (Group 1 and Group 2). All students were required to complete quizzes, to attend lectures, and to read the textbook. All exam questions were based on quiz-question content but were not the same questions. Students in both Groups saw a common pool of quiz questions from which 30 exam questions were based. In addition, students in Group 1 saw quiz questions from which 10 exam questions were based; but only Group-1 students saw the corresponding quiz questions. Group-2 students saw their own set of quiz questions from which 10 exam questions were based; but only they saw these corresponding quiz questions. Exams were comprised of 50 questions; 30 common questions, 10 Group-1 Questions, and 10 Group-2 Questions. The hypothesis was that all students would perform comparably on the Common Questions but that Group-1 Students would perform better on exam questions they had prepared for by taking quizzes and Group-2 Students would perform better on exams question they had prepared for by taking quizzes.

Do Quiz Questions Matter? Do Quiz Questions Matter?

slide-25
SLIDE 25

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

25

Do Quiz Questions Matter? Do Quiz Questions Matter?

Figure 10. Students studied for the exam by taking quizzes, attending lectures, and reading the textbook. Students in both groups performed comparably on exam questions they had studied by taking quizzes. Students in Group 1, however, performed better on exam questions based on items only they had studied on quizzes. Group-2 students, correspondingly, did better on exam questions based on quiz items they had studied.

10 Group-2 Questions 30 Group-1 and Group-2 Common Questions 10 Group-1 Questions Number of Students With Correct Answers

Effects of Studying Different Quiz Questions on Exam01 Performance Spring 2008 Figure 11. Collapsed percentages of correct

  • questions. Common questions were

comparable between groups. Group 1 performed better on their questions; Group 2 performed better on their questions.

slide-26
SLIDE 26

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

26

Time spent reading vs. taking quizzes

Do Quiz Questions Matter?

slide-27
SLIDE 27

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

27

Are students simply

reading the quizzes?

Do Quiz Questions Matter?

Plato's assumption that certain ideas are inborn is most directly relevant to the controversy regarding: A) conscious versus unconscious. B)

  • bservation versus introspection.

C) nature versus nurture. D) correlation versus causation.

QUIZ ITEM STUDYBIT

Plato's assumption that certain ideas are inborn is most directly relevant to the controversy regarding nature versus nurture.

  • I’ll read it later.
  • I’ve read it.
  • I’m learning it.
  • I know it.
slide-28
SLIDE 28

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

28

Do Quiz Questions Matter?

28

slide-29
SLIDE 29

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

29

Do Quiz Questions Matter? Do Quiz Questions Matter?

Group-2 Questions 30 Group-1 and Group-2 Common Questions Group-1 Questions Percentage Correct

slide-30
SLIDE 30

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

30

r = 0.169 p < 0.01 n = 386 0 50 r = 0.229 p < 0.01 n = 293 0 15 r = 0.102 p = 0.07 n = 317 10 5 0 20 Same-Group Exam Qs Correct

Number of Correct Quiz Responses EXAM 04 Personality Sec 001 Groups 1&2 Spring 09 Same-Group Exam Questions vs. Same-Group Quiz Questions

MasteryQz A MasteryQz B LectureQz

slide-31
SLIDE 31

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

31

r = 0.121 p < 0.05 n = 385 0 75 r = 0.178 p < 0.01 n = 293 0 50 r = 0.01 p > 0.2 n = 409 0 20

EXAM 04 Personality Sec 001 Groups 1&2 Spring 09 Opposite-Group Exam Questions vs. StudyBit Responses

MasteryQz A MasteryQz B LectureQz

Number of StudyBit Responses

Oppo-Group Exam Qs Correct 10 5

slide-32
SLIDE 32

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

32 EXAM 04 Personality Sec 002 Groups 1&2 Spring 09 Same-Group Exam Questions vs. Same-Group Quiz Questions

r = 0.177 p < 0.01 n = 265 r = 0.204 p < 0.01 n = 236 r = 0.209 p < 0.01 n = 334

MasteryQz A MasteryQz B LectureQz

Number of Correct Quiz Responses

Same-Group Exam Qs Correct 0 20 0 20 0 40 10 5

slide-33
SLIDE 33

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

33 EXAM 04 Personality Sec 002 Groups 1&2 Spring 09 Opposite-Group Exam Questions vs. Opposite-Group Qz Questions

MasteryQz A MasteryQz B LectureQz

Number of Correct Quiz Responses

r = 0.245 p < 0.01 n = 265 r = 0.202 p < 0.01 n = 236 r = 0.242 p < 0.01 n = 334 Oppo-Group Exam Qs Correct 0 30 0 40 0 40 10 5

slide-34
SLIDE 34

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

34 Number of Correct Quiz Responses

r = 0.136 p < 0.01 n = 474 r = 0.146 p < 0.01 n = 409 r = 0.119 p < 0.05 n = 354

EXAM 09 Personality Sec 001 Groups 1&2 Fall 08 Same-Group Exam Questions vs. Same-Group Quiz Questions

MasteryQz A MasteryQz B LectureQz

0 40 0 15 0 40 Same-Group Exam Qs Correct 10 5

slide-35
SLIDE 35

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

35

MasteryQz A MasteryQz B LectureQz

Number of StudyBit Responses EXAM 09 Personality Sec 001 Groups 1&2 Fall 08 Opposite-Group Exam Questions vs. Opposite-Group Qz Questions

r = 0.001 p > 0.2 n = 409 10 5 0 0 30 r = -0.073 p > 0.1 n = 354 0 25 r = 0.008 p > 0.2 n = 474 0 50 Oppo-Group Exam Qs Correct

slide-36
SLIDE 36

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

36

So far, what’s been of greatest use to you for studying for the exams?

78%

Do Quizzes Matter to Students?

  • A. Reading the textbook
  • B. Attending lectures
  • C. Participating in class
  • D. Taking the quizzes
  • E. Other
  • A. Reading the textbook
  • B. Attending lectures
  • C. Participating in class
  • D. Taking the quizzes
  • E. Other
slide-37
SLIDE 37

NCAT • March 24, 2009 • Gordon Hodge • ghodge@unm.edu

37

So far, what’s been of greatest use to you for studying for the exams?

  • A. Reading the textbook
  • B. Attending lectures
  • C. Participating in class
  • D. Taking the quizzes
  • E. Other

2%

What else matters to Students?