Extreme Quizzing: Extreme Quizzing: Increasing Student Success and - - PowerPoint PPT Presentation

▶
extreme quizzing extreme quizzing increasing student
SMART_READER_LITE
LIVE PREVIEW

Extreme Quizzing: Extreme Quizzing: Increasing Student Success and - - PowerPoint PPT Presentation

Dec 03, 2022 215 likes •425 views

Extreme Quizzing: Extreme Quizzing: Increasing Student Success and Increasing Student Success and Retention Retention Gordon Gordon Hodge Hodge Gordon Gordon Hodge Hodge Mexico & Mexico & University of University of New New

slide-1
SLIDE 1

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

Extreme Quizzing: Increasing Student Success and Retention Extreme Quizzing: Increasing Student Success and Retention

Gordon Gordon Hodge Hodge

University University of

  • f New

New Mexico Mexico & National National Center Center for for Academic Academic Transformation Transformation

Gordon Gordon Hodge Hodge

University University of

  • f New

New Mexico Mexico & National National Center Center for for Academic Academic Transformation Transformation

NCAT Second Annual Redesign Alliance Conference Orlando • March 16–18, 2008 NCAT Second Annual Redesign Alliance Conference Orlando • March 16–18, 2008

slide-2
SLIDE 2

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

Intro Psych 105 is the largest of UNM’s 20 “killer” courses for lower division undergraduate students. Before the redesign, the drop, withdrawal, failure rate was 42%.

Support Pew Grant Program in Redesign National Center for Academic Transformation Proposal for Redesign of General Psychology DOE Institute for Education Science (IES) Grant Test-Enhanced Learning.

slide-3
SLIDE 3

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

While maintaining or exceeding standards …

  • Improve grades (but without inflation)
  • Use merit system (no curving)
  • Comprehensive coverage
  • Achieve consistency across sections
  • Increase time on task

Goals:

slide-4
SLIDE 4

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

 Quizzes Required  3,000 Question Pool  3 Quizzes per week  20 Questions per quiz  Timed & Deadlines  Unlimited repeats  Highest score counts  1 Point per question  1 Exam per week  Quizzes Required  3,000 Question Pool  3 Quizzes per week  20 Questions per quiz  Timed & Deadlines  Unlimited repeats  Highest score counts  1 Point per question  1 Exam per week

Solution: WebCT Quizzes

slide-5
SLIDE 5

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

Instructions to Students

Quizzes are not just a way to review or to test yourself. Quizzes are an important way to learn the material in the first place.

slide-6
SLIDE 6

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

Do Quizzes Matter?

  • Take quizzes many times (10-20x)
  • Get perfect or near perfect scores
  • Complete later quizzes in under 5 min
  • Take quizzes many times (10-20x)
  • Get perfect or near perfect scores
  • Complete later quizzes in under 5 min

Students who do well on exams typically: Students who do well on exams typically:

slide-7
SLIDE 7

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

Do Quizzes Matter?

r = .93

Fall Fall 2005 2005: Correlations Fall Fall 2005 2005: Correlations

Exam Points

Figure 1. High quiz scores predicted high exam score (n = 847 students).

slide-8
SLIDE 8

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

Do Quizzes Matter?

Average Score (80 Questions)

Fall Fall 2005 2005: Pretest vs. Posttest Fall Fall 2005 2005: Pretest vs. Posttest

Figure 2. Students took an 80-question Pretest during the first week of the semester and the same test as the Final Exam. Students who performed well in the course, performed better on the Posttest Final than students who performed less well.

slide-9
SLIDE 9

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

Do Quizzes Matter?

Percentage of Students

Fall Fall 2005 2005: Ethnicity Fall Fall 2005 2005: Ethnicity

Figure 3. Students of different ethnic backgrounds performed comparably.

slide-10
SLIDE 10

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

Passing Grades for Psych 105

Fall 2000 – Fall 2007

59% 71% 63% 77% 74% 78% 77% 77% 80% 81% 73% 75% 63% 75%

41% 29% 37% 23% 26% 22% 23% 23% 20% 19% 27% 25% 37% 25%

0% 20% 40% 60% 80%

F00 Tradi- tional F01 Pilot S02 F02 S03 F03 S04 F04 S05 F05 S06 F06 S07 F07

Semester

Percentage of Students C or Above C- or Below (& Ws)

Do Quizzes Matter?

Fall Fall 2000 2000 – – Fall Fall 2007 2007: 14 Semesters Fall Fall 2000 2000 – – Fall Fall 2007 2007: 14 Semesters

Figure 4. Compared to traditional designs (i.e., lectures, readings, exams), students who were required to complete quizzes performed better in the course.

slide-11
SLIDE 11

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

Do Required Quizzes Matter?

Spring Spring 2002 2002: Two Sections/Same Instructor Spring Spring 2002 2002: Two Sections/Same Instructor

Figure 5. To determine whether quizzes need to be required in order for students to perform well in the class, two sections were taught by the same instructor. In Sec 001, quizzes were required; in Sec 002, quizzes were available but optional (i.e., students were encouraged to take them, but they received no points for doing so). Students performed better when quizzes were required (Sec 001) than when quizzes were optional (Sec 002). Percentage of Students

slide-12
SLIDE 12

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

Do Required Quizzes Matter?

Figure 6. Patterns of Quiz Taking (left). When quizzes were required, students scored higher, took them more times, and for longer durations than when quizzes were optional.

Percentage of Students

(Figure 5)

High Score

2 4 6 8 10 Number of Attempts 4 8 12 16 20 High Score

Quizzes Optional

3 6 9 12 Weekly Mastery Quizzes Average Duration (Min) 2 4 6 8 10 Number of Attempts 4 8 12 16 20 High Score A B C+ and C C- and Below

Number of Attempts

Weekly Mastery Quizzes

3 6 9 12 Weekly Mastery Quizzes

Weekly Mastery Quizzes

Quizzes Required

Mean Duration (min)

Spring Spring 2002 2002: Two Sections/Same Instructor Spring Spring 2002 2002: Two Sections/Same Instructor

slide-13
SLIDE 13

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

Do Required Quizzes Matter?

Spring Spring 2005 2005: Two Sections/Different Instructors Spring Spring 2005 2005: Two Sections/Different Instructors

Course Grade Percentage of Students Figure 7. In spring 2005, to determine whether quizzes need to be required in order for students to perform well in the class, two sections were taught by different instructors. In Sec 001, quizzes were required; in Sec 002, quizzes were available but optional (i.e., students were encouraged to take them, but they received no points for doing so). Students performed better when quizzes were required (Sec 001) than when quizzes were optional (Sec 002).

slide-14
SLIDE 14

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

Figure 8. In spring 2007, to determine whether quizzes need to be required in order for students to perform well on exams, two sections were counterbalanced each week for whether quizzes were required or optional. For weeks when quizzes were required, students performed better across sections than when quizzes were optional.

Mean In-Class Exam Performance S07 Sec 001 vs Sec 002

50 52 54 56 58 60 62 64 66 68 Exam01 Meth p<0.04 Exam02 Neuro p<0.00 Exam03 Devel p<0.61 Exam04 S&P p<0.00 Exam05 Consc p<0.19 Exam06 Learn p<0.01 Exam07 ThkIntel p<0.01 Exam08 MoEmo p<0.01 Exam09 Stress p<0.01 Exam10 Pers p<0.01 Exam11 AbTher p<0.26 Exam12 Soc p<0.25

Exam p value (Single Factor Btwn ANOVA: Sec 001 vs Sec 002)

Points (out of 80)

Sec 001 Qzs Rqd on Odd Exams Sec 002 Qzs Rqd on Even Exams

Do Required Quizzes Matter?

Spring Spring 2007 2007: Alternating Weeks—Rqd vs Opt Qzs Spring Spring 2007 2007: Alternating Weeks—Rqd vs Opt Qzs

* * * * * * * * * * * * * * * * * *

slide-15
SLIDE 15

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

Are Are students students just just memorizing memorizing the the questions? questions? Are Are students students just just memorizing memorizing the the questions? questions?

Do Quiz Questions Matter?

slide-16
SLIDE 16

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

QUIZ STEM — In elementary school and high school, Charlie got away with copying his test answers from classmates. Because the college has test proctors who are very

  • bservant, Charlie spends as many hours devising new ways to cheat as it would take him

to study and perform well in an honest fashion. Charlie's strategy for passing tests illustrates the consequences of:

  • functional fixedness.
  • a mental set.
  • confirmation bias.
  • the availability heuristic.

What students see Do Quiz Questions Matter?

slide-17
SLIDE 17

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

What students may learn

QUIZ STEM — In elementary school and high school, Charlie Charlie got away with copying his test answers from classmates. Because the college has test proctors who are very

  • bservant, Charlie

Charlie spends as many hours devising new ways to cheat as it would take him to study and perform well in an honest fashion. Charlie Charlie's strategy for passing tests illustrates the consequences of:

  • functional fixedness.
  • a

a mental mental set. set.

  • confirmation bias.
  • the availability heuristic.

Charlie Charlie = = mental mental set set

Do Quiz Questions Matter?

slide-18
SLIDE 18

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

TEST STEM — A mental set is a:

  • methodical step-by-step procedure for solving problems.
  • mental grouping of similar objects, events, or people.
  • tendency to approach a problem in a way that has been successful in the past.
  • group of conclusions derived from certain assumptions or general principles.

QUIZ STEM — In elementary school and high school, Charlie got away with copying his test answers from classmates. Because the college has test proctors who are very observant, Charlie spends as many hours devising new ways to cheat as it would take him to study and perform well in an honest fashion. Charlie's strategy for passing tests illustrates the consequences of:

  • functional fixedness.
  • a mental set.
  • confirmation bias.
  • the availability heuristic.

Do Quiz Questions Matter? Use different Exam questions Use different Exam questions

slide-19
SLIDE 19

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

To determine whether studying by taking quizzes versus going to lectures and reading the textbook differentially affects exam performance, students were divided into two groups (Group 1 and Group 2). All students were required to complete quizzes, to attend lectures, and to read the textbook. All exam questions were based on quiz-question content but were not the same questions. Students in both Groups saw a common pool of quiz questions from which 30 exam questions were based. In addition, students in Group 1 saw quiz questions from which 10 exam questions were based; but only Group-1 students saw the corresponding quiz questions. Group-2 students saw their own set of quiz questions from which 10 exam questions were based; but

  • nly they saw these corresponding quiz questions.

Exams were comprised of 50 questions; 30 common questions, 10 Group-1 Questions, and 10 Group-2 Questions. The hypothesis was that all students would perform comparably on the Common Questions but that Group-1 Students would perform better on exam questions they had prepared for by taking quizzes and Group-2 Students would perform better on exams question they had prepared for by taking quizzes.

Do Quiz Questions Matter? Do Quiz Questions Matter?

slide-20
SLIDE 20

NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu NCAT Redesign Alliance • March 16-18, 2008 • Gordon Hodge • ghodge@unm.edu

Do Quiz Questions Matter? Do Quiz Questions Matter?

Figure 9. Students studied for the exam by taking quizzes, attending lectures, and reading the textbook. Students in both groups performed comparably on exam questions they had studied by taking quizzes. Students in Group 1, however, performed better on exam questions based on items only they had studied on quizzes. Group-2 students, correspondingly, did better on exam questions based on quiz items they had studied.

100 200 300 400

Number ofStudents With Correct Answer Group 1 Group 2

Group 1 35 31 19 35 25 39 38 31 23 20 19 12 13 23 29 34 35 29 12 34 41 38 34 37 33 31 34 40 34 42 40 36 38 38 10 37 36 38 28 29 40 39 38 38 24 23 27 32 31 Group 2 22 21 99 30 18 30 37 27 15 31 27 20 20 36 36 36 36 30 14 32 39 36 33 36 31 29 32 39 31 41 39 35 37 36 12 35 34 34 27 29 39 37 36 36 22 23 24 29 30 8 9 11 12 13 14 15 41 42 43 4 5 6 7 10 16 44 45 46 47 1 2 3 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 48 49 50

10 Group-2 Questions 30 Group-1 and Group-2 Common Questions 10 Group-1 Questions Number of Students With Correct Answers

Effects of Studying Select Quiz Questions on Exam01 Performance

Exam 1 (All Common Qs)

6 12 18 24 30

Common Qs

Number Questions Correct

Group 1 Group 2 Exam 1 (All Common Qs)

6 12 18 24 30

Common Qs

Number Questions Correct

Group 1 Group 2 Group 1 & Group 2 Qs

2 4 6 8 10

Group 1 Qs Group 2 Qs

Number Questions Correct

Group 1 Group 2 Group 1 & Group 2 Qs

2 4 6 8 10

Group 1 Qs Group 2 Qs

Number Questions Correct

Group 1 Group 2

Figure 10. Collapsed percentages of questions gotten correct. Common questions were comparable between

  • groups. Group 1 performed better on their

questions; Group 2 performed better on their questions.