Deterring Cheating in Online Environments Henry Corrigan-Gibbs - - PowerPoint PPT Presentation

▶
deterring cheating in online environments
SMART_READER_LITE
LIVE PREVIEW

Deterring Cheating in Online Environments Henry Corrigan-Gibbs - - PowerPoint PPT Presentation

Aug 24, 2022 160 likes •1.19k views

Deterring Cheating in Online Environments Henry Corrigan-Gibbs Nakull Gupta Curtis Northcutt Stanford University Microsoft Research India MIT Edward Cutrell William Thies Microsoft Research India Microsoft

slide-1
SLIDE 1

Deterring Cheating in
 Online Environments

Henry Corrigan-Gibbs
 Stanford University Nakull Gupta
 Microsoft Research India Curtis Northcutt
 MIT Edward Cutrell William Thies
 Microsoft Research India Microsoft Research India

slide-2
SLIDE 2

Online
 Education

slide-3
SLIDE 3

Online
 Education Online
 Microwork

slide-4
SLIDE 4

Online
 Education Online
 Marketplaces Online
 Microwork

slide-5
SLIDE 5

Online
 Education Online
 Marketplaces Online
 Reviews Online
 Microwork

slide-6
SLIDE 6

But…

slide-7
SLIDE 7

But…

Disposable identities

slide-8
SLIDE 8

But…

Disposable identities Few enforceable rules

slide-9
SLIDE 9

But…

Disposable identities Few enforceable rules Different social norms

slide-10
SLIDE 10
slide-11
SLIDE 11
slide-12
SLIDE 12
slide-13
SLIDE 13
slide-14
SLIDE 14
slide-15
SLIDE 15
slide-16
SLIDE 16
slide-17
SLIDE 17

Research Questions

  • 1. How can we deter dishonest behavior in online

environments?

  • 2. How will we know when we have succeeded? 


(i.e., How do we measure the rate of cheating?)

Experiments

I. Microsoft’s online course platform (India)

  • II. Amazon Mechanical Turk (US & India)
slide-18
SLIDE 18

Research Questions

  • 1. How can we deter dishonest behavior in online

environments?

  • 2. How will we know when we have succeeded? 


(i.e., How do we measure the rate of cheating?)

Experiments

I. Microsoft’s online course platform (India)

  • II. Amazon Mechanical Turk (US & India) } In the paper
slide-19
SLIDE 19

Research Questions

  • 1. How can we deter dishonest behavior in online

environments?

  • 2. How will we know when we have succeeded? 


(i.e., How do we measure the rate of cheating?)

Experiments

I. Microsoft’s online course platform (India)

  • II. Amazon Mechanical Turk (US & India)
slide-20
SLIDE 20

Research Questions

  • 1. How can we deter dishonest behavior in online

environments?

  • 2. How will we know when we have succeeded? 


(i.e., How do we measure the rate of cheating?)

Experiments

I. Microsoft’s online course platform (India)

  • II. Amazon Mechanical Turk (US & India)
slide-21
SLIDE 21

A Theory of Dishonesty

slide-22
SLIDE 22

A Theory of Dishonesty

Moral Consequences

  • Will this make me

feel guilty?

  • What would my

parents think?

slide-23
SLIDE 23

A Theory of Dishonesty

Moral Consequences

  • Will this make me

feel guilty?

  • What would my

parents think?

Material Consequences

  • Will I get caught?
  • How much do I

stand to gain / lose?


slide-24
SLIDE 24

Intervention I: Honor Code

  • Make cheating “morally costly”
  • Require signed statement declaring intent

to behave honestly

  • Evidently effective in in-person interactions

[Mazar et al. 2008], [Pruckner and Sausgruber 2013], [Rosenbaum et al. 2014]

slide-25
SLIDE 25

Intervention I: Honor Code

  • Make cheating “morally costly”
  • Require signed statement declaring intent

to behave honestly

  • Evidently effective in in-person interactions

[Mazar et al. 2008], [Pruckner and Sausgruber 2013], [Rosenbaum et al. 2014]

“I shall neither give nor receive
 help during this examination.”

slide-26
SLIDE 26

Intervention I: Honor Code

  • Make cheating “morally costly”
  • Require signed statement declaring intent

to behave honestly

  • Evidently effective in in-person interactions

[Mazar et al. 2008], [Pruckner and Sausgruber 2013], [Rosenbaum et al. 2014]

slide-27
SLIDE 27

Intervention II: Warning

  • Inform the participant of the potential

material costs of cheating

[Fellner et al. 2013]

slide-28
SLIDE 28

Intervention II: Warning

  • Inform the participant of the potential

material costs of cheating

Warning

If we discover that you violated the
 rules of this online exam, we may: – Cancel your exam.
 – Cancel your account.
 – Notify your university.

[Fellner et al. 2013]

slide-29
SLIDE 29

Intervention II: Warning

  • Inform the participant of the potential

material costs of cheating

[Fellner et al. 2013]

slide-30
SLIDE 30

General Method

  • 1. Randomize participants into treatment groups:


(a) Control
 (b) Honor Code
 (c) Warning

  • 2. Measure rate of cheating in each group
  • 3. Determine whether treatment had any

measurable effect

slide-31
SLIDE 31

General Method

  • 1. Randomize participants into treatment groups:


(a) Control
 (b) Honor Code
 (c) Warning

  • 2. Measure rate of cheating in each group
  • 3. Determine whether treatment had any

measurable effect

???

slide-32
SLIDE 32

Research Questions

  • 1. How can we deter dishonest behavior in online

environments?

  • 2. How will we know when we have succeeded? 


(i.e., How do we measure the rate of cheating?)

Experiments

I. Microsoft’s online course platform (India)

  • II. Amazon Mechanical Turk (US & India)
slide-33
SLIDE 33

Research Questions

  • 1. How can we deter dishonest behavior in online

environments?

  • 2. How will we know when we have succeeded? 


(i.e., How do we measure the rate of cheating?)

Experiments

I. Microsoft’s online course platform (India)

  • II. Amazon Mechanical Turk (US & India)
slide-34
SLIDE 34

Measuring Cheating

slide-35
SLIDE 35

Measuring Cheating

  • nlineexam.edu
slide-36
SLIDE 36

Measuring Cheating

  • nlineexam.edu
slide-37
SLIDE 37

Measuring Cheating

  • nlineexam.edu
slide-38
SLIDE 38

Measuring Cheating

  • nlineexam.edu
slide-39
SLIDE 39

Measuring Cheating

  • nlineexam.edu

wikipedia.org

slide-40
SLIDE 40

Measuring Cheating

  • nlineexam.edu

wikipedia.org

slide-41
SLIDE 41

Measuring Cheating

  • nlineexam.edu

wikipedia.org

slide-42
SLIDE 42

Measuring Cheating

slide-43
SLIDE 43

Measuring Cheating

  • For free-response questions, can use


plagiarism detection tools

slide-44
SLIDE 44

Measuring Cheating

  • For free-response questions, can use


plagiarism detection tools

“Did Alice copy verbatim from Wikipedia?”

slide-45
SLIDE 45

Measuring Cheating

  • For free-response questions, can use


plagiarism detection tools

“Did Alice copy verbatim from Wikipedia?” “Did Alice and Bob submit exactly the same response?”

slide-46
SLIDE 46

Measuring Cheating

  • For free-response questions, can use


plagiarism detection tools

“Did Alice copy verbatim from Wikipedia?” “Did Alice and Bob submit exactly the same response?”

  • In use today (e.g., Turnitin),


and we use these techniques too


slide-47
SLIDE 47

Measuring Cheating

  • For free-response questions, can use


plagiarism detection tools

“Did Alice copy verbatim from Wikipedia?” “Did Alice and Bob submit exactly the same response?”

  • In use today (e.g., Turnitin),


and we use these techniques too
 … but for multiple-choice questions?

slide-48
SLIDE 48

“Honey pot”

  • nlineexam.edu

[Provos 2004]

slide-49
SLIDE 49

“Honey pot”

  • nlineexam.edu

exam-answers.org

[Provos 2004]

slide-50
SLIDE 50

“Honey pot”

  • nlineexam.edu

exam-answers.org

[Provos 2004]

slide-51
SLIDE 51

“Honey pot”

  • nlineexam.edu

exam-answers.org

[Provos 2004]

slide-52
SLIDE 52
  • nlineexam.edu

exam-answers.org

slide-53
SLIDE 53
  • nlineexam.edu

exam-answers.org

slide-54
SLIDE 54
  • nlineexam.edu

exam-answers.org

slide-55
SLIDE 55
  • nlineexam.edu

exam-answers.org

slide-56
SLIDE 56
  • nlineexam.edu

exam-answers.org

slide-57
SLIDE 57
  • nlineexam.edu

exam-answers.org

slide-58
SLIDE 58
  • nlineexam.edu

exam-answers.org

slide-59
SLIDE 59
  • nlineexam.edu

exam-answers.org

slide-60
SLIDE 60
  • nlineexam.edu

exam-answers.org

slide-61
SLIDE 61
  • nlineexam.edu

exam-answers.org

slide-62
SLIDE 62
  • nlineexam.edu

exam-answers.org We use the cookie to
 identify dishonest behavior

slide-63
SLIDE 63

Caveat: False Negatives

  • nlineexam.edu

exam-answers.org

slide-64
SLIDE 64

Caveat: False Negatives

  • nlineexam.edu

exam-answers.org

slide-65
SLIDE 65

Caveat: False Negatives

  • nlineexam.edu

exam-answers.org

slide-66
SLIDE 66

Caveat: False Negatives

  • nlineexam.edu

exam-answers.org

slide-67
SLIDE 67

Caveat: False Negatives

  • nlineexam.edu

exam-answers.org

slide-68
SLIDE 68

Caveat: False Negatives

  • nlineexam.edu

exam-answers.org

slide-69
SLIDE 69

Research Questions

  • 1. How can we deter dishonest behavior in online

environments?

  • 2. How will we know when we have succeeded? 


(i.e., How do we measure the rate of cheating?)

Experiments

I. Microsoft’s online course platform (India)

  • II. Amazon Mechanical Turk (US & India)
slide-70
SLIDE 70

Research Questions

  • 1. How can we deter dishonest behavior in online

environments?

  • 2. How will we know when we have succeeded? 


(i.e., How do we measure the rate of cheating?)

Experiments

I. Microsoft’s online course platform (India)

  • II. Amazon Mechanical Turk (US & India)
slide-71
SLIDE 71

Research Questions

  • 1. How can we deter dishonest behavior in online

environments?

  • 2. How will we know when we have succeeded? 


(i.e., How do we measure the rate of cheating?)

Experiments

I. Microsoft’s online course platform (India)

  • II. Amazon Mechanical Turk (US & India)
slide-72
SLIDE 72

Experiment I: Online Course

  • MSR India free online algorithms course (2014-15)
  • Online final examination


– 14 multiple-choice questions + 1 free response
 – 409 students in spring 2014
 – 223 students in winter 2015

  • “Closed book, closed Internet” exam


 
 All students had an opportunity to opt out and we did not penalize students we suspected of cheating.

slide-73
SLIDE 73
slide-74
SLIDE 74
slide-75
SLIDE 75
slide-76
SLIDE 76
slide-77
SLIDE 77
slide-78
SLIDE 78
slide-79
SLIDE 79
slide-80
SLIDE 80
slide-81
SLIDE 81

Study population 603 students

slide-82
SLIDE 82

Plagiarized on
 free-response 102 students Study population 603 students

slide-83
SLIDE 83

Plagiarized on
 free-response 102 students Study population 603 students Visited honey pot 50 students

slide-84
SLIDE 84

Plagiarized on
 free-response 102 students Study population 603 students Visited honey pot 50 students Both 5 students

slide-85
SLIDE 85

Scores (Honest) Scores (Cheating) Frequency 10 20 30 40

0% 50% 100% 0% 50% 100%

Results: Cheating vs. Score

slide-86
SLIDE 86

Scores (Honest) Scores (Cheating) Frequency 10 20 30 40

0% 50% 100% 0% 50% 100%

Results: Cheating vs. Score

Cheating doesn’t pay! (p < 0.0001)

slide-87
SLIDE 87

Control Honor Code Warn Percentage cheating 0% 10% 20% 30% 40% 50%

Online Exam

31.7% 25.9% 14.6%

Results: HC and Warning

slide-88
SLIDE 88

Control Honor Code Warn Percentage cheating 0% 10% 20% 30% 40% 50%

Online Exam

31.7% 25.9% 14.6%

Results: HC and Warning

Honor code group saw a drop in cheating rate
 (not stat. signif.)

slide-89
SLIDE 89

Control Honor Code Warn Percentage cheating 0% 10% 20% 30% 40% 50%

Online Exam

31.7% 25.9% 14.6%

Results: HC and Warning

slide-90
SLIDE 90

Control Honor Code Warn Percentage cheating 0% 10% 20% 30% 40% 50%

Online Exam

31.7% 25.9% 14.6%

Results: HC and Warning

Warning has an effect
 (N=398, p < 0.001)

slide-91
SLIDE 91

Control Honor Code Warn Percentage cheating 0% 10% 20% 30% 40% 50%

Online Exam

31.7% 25.9% 14.6%

Results: HC and Warning

slide-92
SLIDE 92

Control Honor Code Warn Percentage cheating 0% 10% 20% 30% 40% 50%

Online Exam

31.7% 25.9% 14.6%

Results: HC and Warning

Control Honor Code Warn

MTurk (India)

41.6% 35.8% 18.3%

slide-93
SLIDE 93

Where we go from here…

slide-94
SLIDE 94

Where we go from here…

  • Does the effect of a warning diminish over time?
slide-95
SLIDE 95

Where we go from here…

  • Does the effect of a warning diminish over time?
  • Do warnings scare off honest and dishonest

users at different rates?

slide-96
SLIDE 96

Where we go from here…

  • Does the effect of a warning diminish over time?
  • Do warnings scare off honest and dishonest

users at different rates?

  • Can we use honey pots to automate the detection
  • f unethical behavior online? Abuse, etc.
slide-97
SLIDE 97

Where we go from here…

  • Does the effect of a warning diminish over time?
  • Do warnings scare off honest and dishonest

users at different rates?

  • Can we use honey pots to automate the detection
  • f unethical behavior online? Abuse, etc.
  • Can warnings help deter other bad behavior
  • nline? Nasty comments, etc.
slide-98
SLIDE 98

Summary

Control Honor Code Warn Percentage cheating 0% 10% 20% 30% 40% 50%

Online Exam

31.7% 25.9% 14.6%

slide-99
SLIDE 99

Summary

  • We studied the effect of an honor

code and a warning at deterring cheating in an online course and Mechanical Turk.

Control Honor Code Warn Percentage cheating 0% 10% 20% 30% 40% 50%

Online Exam

31.7% 25.9% 14.6%

slide-100
SLIDE 100

Summary

  • We studied the effect of an honor

code and a warning at deterring cheating in an online course and Mechanical Turk.

  • We find evidence that the warning

was effective in online settings… less so the honor code.

Control Honor Code Warn Percentage cheating 0% 10% 20% 30% 40% 50%

Online Exam

31.7% 25.9% 14.6%

slide-101
SLIDE 101

Summary

  • We studied the effect of an honor

code and a warning at deterring cheating in an online course and Mechanical Turk.

  • We find evidence that the warning

was effective in online settings… less so the honor code.

  • “Honey pots” may be a useful tool

for measuring rates of cheating in future work.

Control Honor Code Warn Percentage cheating 0% 10% 20% 30% 40% 50%

Online Exam

31.7% 25.9% 14.6%