Monitoring Electronic Exams Ali Kassem 1 , Ylis Falcone 2 and Pascal - - PowerPoint PPT Presentation

monitoring electronic exams
SMART_READER_LITE
LIVE PREVIEW

Monitoring Electronic Exams Ali Kassem 1 , Ylis Falcone 2 and Pascal - - PowerPoint PPT Presentation

Monitoring Electronic Exams Ali Kassem 1 , Ylis Falcone 2 and Pascal Lafourcade 3 1 Univ. Grenoble Alpes, VERIMAG, Grenoble, France 2 Univ. Grenoble Alpes, Inria, LIG, Grenoble 3 Universit Clermont Auvergne, LIMOS, France The 15th International


slide-1
SLIDE 1

Monitoring Electronic Exams

Ali Kassem1, Yliès Falcone2 and Pascal Lafourcade3

  • 1Univ. Grenoble Alpes, VERIMAG, Grenoble, France
  • 2Univ. Grenoble Alpes, Inria, LIG, Grenoble

3Université Clermont Auvergne, LIMOS, France

The 15th International Conference on Runtime Verification Vienna, September 28, 2015

1 / 30

slide-2
SLIDE 2

Traditional Exam

2 / 30

slide-3
SLIDE 3

e-exam

Information technology for the assessment of knowledge and skills.

3 / 30

slide-4
SLIDE 4

Reality

4 / 30

slide-5
SLIDE 5
  • Threats. . .

◮ Candidate cheating ◮ Bribed, corrupted or unfair examiners ◮ Dishonest/untrusted exam authority ◮ Outside attackers ◮ . . .

5 / 30

slide-6
SLIDE 6

. . . and their Mitigation

Most existing e-exam systems assume trusted authorities and focus on student cheating:

◮ Exam centers ◮ Software solutions, e.g. ProctorU

6 / 30

slide-7
SLIDE 7

. . . and their Mitigation

Most existing e-exam systems assume trusted authorities and focus on student cheating:

◮ Exam centers ◮ Software solutions, e.g. ProctorU

Yet also the other threats are real:

◮ Atlanta Public Schools cheating scandal (2009) ◮ UK student visa tests fraud (2014)

6 / 30

slide-8
SLIDE 8

. . . and their Mitigation

Most existing e-exam systems assume trusted authorities and focus on student cheating:

◮ Exam centers ◮ Software solutions, e.g. ProctorU

Yet also the other threats are real:

◮ Atlanta Public Schools cheating scandal (2009) ◮ UK student visa tests fraud (2014)

So what about dishonest authorities or hackers?

6 / 30

slide-9
SLIDE 9

Several Security Properties

Secrypt’14 Authentication Properties: Mark Authenticity, Answer Origin Authentication, Form Authorship, Form Authenticity. Privacy Properties: Anonymous Marking, Question Indistinguishability, Anonymous Examiner, Mark Privacy, Mark Anonymity ISPEC’15 Individual Verifiability: Question Validity, Marking Correctness, Exam-Test Integrity, Exam-Test Markedness, Marking Integrity, Marking Notification Integrity Universal Verifiability: Eligibility (Registration), Marking Correctness Exam-Test Integrity, Exam-Test Markedness, Marking Integrity.

7 / 30

slide-10
SLIDE 10

Several Security Properties

Secrypt’14 Authentication Properties: Mark Authenticity, Answer Origin Authentication, Form Authorship, Form Authenticity. Privacy Properties: Anonymous Marking, Question Indistinguishability, Anonymous Examiner, Mark Privacy, Mark Anonymity ISPEC’15 Individual Verifiability: Question Validity, Marking Correctness, Exam-Test Integrity, Exam-Test Markedness, Marking Integrity, Marking Notification Integrity Universal Verifiability: Eligibility (Registration), Marking Correctness Exam-Test Integrity, Exam-Test Markedness, Marking Integrity. How can we use it on real e-exam?

7 / 30

slide-11
SLIDE 11

Plan

Introduction Model Properties Case Study: UJF E-exam Conclusion

8 / 30

slide-12
SLIDE 12

Plan

Introduction Model Properties Case Study: UJF E-exam Conclusion

9 / 30

slide-13
SLIDE 13

E-exam: Players and Organization

Three Roles: Candidate Examination Authority Examiner

10 / 30

slide-14
SLIDE 14

E-exam: Players and Organization

Three Roles: Candidate Examination Authority Examiner Four Phases:

  • 1. Registration
  • 2. Examination
  • 3. Marking
  • 4. Notification

10 / 30

slide-15
SLIDE 15

Event Based Model

slide-16
SLIDE 16

Event Based Model

  • 1. Registration
slide-17
SLIDE 17

Event Based Model

  • 1. Registration

register( )

Register

slide-18
SLIDE 18

Event Based Model

  • 1. Registration

register( )

Register

  • 2. Examination
slide-19
SLIDE 19

Event Based Model

  • 1. Registration

register( )

Register

  • 2. Examination

begin( )

slide-20
SLIDE 20

Event Based Model

  • 1. Registration

register( )

Register

  • 2. Examination

begin( ) get( , )

Question

slide-21
SLIDE 21

Event Based Model

  • 1. Registration

register( )

Register

  • 2. Examination

begin( ) get( , )

Question

change( , , )

slide-22
SLIDE 22

Event Based Model

  • 1. Registration

register( )

Register

  • 2. Examination

begin( ) get( , )

Question

change( , , ) submit( , , ) accept( , , )

Answer

slide-23
SLIDE 23

Event Based Model

  • 1. Registration

register( )

Register

  • 2. Examination

begin( ) get( , )

Question

change( , , ) submit( , , ) accept( , , )

Answer

end( )

11 / 30

slide-24
SLIDE 24

Event Based Model

  • 3. Marking
slide-25
SLIDE 25

Event Based Model

  • 3. Marking

corr( , )

Correct Answer

slide-26
SLIDE 26

Event Based Model

  • 3. Marking

corr( , )

Correct Answer

mark( , , , )

Evaluation

slide-27
SLIDE 27

Event Based Model

  • 3. Marking

corr( , )

Correct Answer

mark( , , , )

Evaluation

  • 4. Notification
slide-28
SLIDE 28

Event Based Model

  • 3. Marking

corr( , )

Correct Answer

mark( , , , )

Evaluation

  • 4. Notification

assign( , )

Mark

12 / 30

slide-29
SLIDE 29

Plan

Introduction Model Properties Case Study: UJF E-exam Conclusion

13 / 30

slide-30
SLIDE 30

Quantified Event Automata (QEAs)

◮ Properties expressed as QEAs [BFH+12]: event automaton

with quantified variables.

◮ An event automaton is a finite-state machine with

transitions labeled by parametric events.

◮ Transitions may include guards and assignments. ◮ We extend the initial definition of QEAs by:

  • 1. variable declaration and initialization before reading the trace
  • 2. global variable shared among all event automaton instances.

◮ event(parameters)

[guard] assignment

14 / 30

slide-31
SLIDE 31

Candidate Eligibility

No answer is accepted from an unregistered candidate ∀i 1 2 register(i) Σ = {register(i), accept(i, q, a)}

slide-32
SLIDE 32

Candidate Eligibility

No answer is accepted from an unregistered candidate ∀i 1 2 register(i) Σ = {register(i), accept(i, q, a)} Σ 3 accept(i, q, a)

15 / 30

slide-33
SLIDE 33

Candidate Eligibility with Auditing

All candidates that violates the requirement are collected in a set F. Initially: I : ˆ = ∅ 1 2 register(i) I:=I∪{i} accept(i, q, a)

[i / ∈I] F:ˆ ={i}

register(i) I:=I∪{i} accept(i, q, a)

[i / ∈I] F:=F∪{i}

16 / 30

slide-34
SLIDE 34

Properties

Candidate Registration: an unregistered candidate tried to take the exam.

17 / 30

slide-35
SLIDE 35

Properties

Candidate Registration: an unregistered candidate tried to take the exam. Answer Authentication:

◮ an unsubmitted answer was considered as accepted; or ◮ more than one answer were accepted from a candidate.

17 / 30

slide-36
SLIDE 36

Properties

Candidate Registration: an unregistered candidate tried to take the exam. Answer Authentication:

◮ an unsubmitted answer was considered as accepted; or ◮ more than one answer were accepted from a candidate.

Questions Ordering:

◮ a candidate got a question before validating the previous ones.

17 / 30

slide-37
SLIDE 37

Properties (continued)

Exam Availability: an answer was accepted outside exam time.

18 / 30

slide-38
SLIDE 38

Properties (continued)

Exam Availability: an answer was accepted outside exam time. Exam Availability with Flexibility:

◮ supports different duration and starting time between

candidates.

18 / 30

slide-39
SLIDE 39

Properties (continued)

Exam Availability: an answer was accepted outside exam time. Exam Availability with Flexibility:

◮ supports different duration and starting time between

candidates. Marking Correctness: an answer was marked in a wrong way.

18 / 30

slide-40
SLIDE 40

Properties (continued)

Exam Availability: an answer was accepted outside exam time. Exam Availability with Flexibility:

◮ supports different duration and starting time between

candidates. Marking Correctness: an answer was marked in a wrong way. Mark Integrity:

◮ an accepted answer was not marked; or ◮ a candidate was not assigned the corresponding mark.

18 / 30

slide-41
SLIDE 41

Plan

Introduction Model Properties Case Study: UJF E-exam Conclusion

19 / 30

slide-42
SLIDE 42

E-exam at Université Joseph Fourier (UJF)

Registration:

◮ 2 weeks before the exam. ◮ Using login/password.

20 / 30

slide-43
SLIDE 43

E-exam at Université Joseph Fourier (UJF)

Examination in a supervised room Authentication and answers questions as follows:

◮ In a fixed order. ◮ Once validates the current question, he gets the next one. ◮ He can change the answer unlimited times before validating. ◮ Once he validates, then he cannot go back and change any of

the validated answers.

21 / 30

slide-44
SLIDE 44

E-exam at Université Joseph Fourier (UJF)

Marking:

◮ For each question, the professor specifies the correct answer(s). ◮ For each question, all the answers provided by the candidates

are collected.

◮ Each answer is evaluated by an examiner to 0 or 1. ◮ The mark for each candidate is calculated as the summation of

all the scores attributed to his answers. Notification:

◮ The marks are notified to the candidates. ◮ A candidate can consult his submission and check the marking.

22 / 30

slide-45
SLIDE 45

Analysis

Verification of two real e-exam executions using MarQ tool [RCR15]. From the logs: register(i), change(i, q, a), submit(i, q, a), accept(i, q, a).

4 Properties

◮ Candidate Registration ◮ Candidate Eligibility ◮ Answer Authentication ◮ Exam Availability

23 / 30

slide-46
SLIDE 46

5 new properties

◮ Answer Authentication ∗:

◮ All accepted answers are submitted by candidates. ◮ Allow the acceptance of the same answer again. ◮ But, still forbids the acceptance of a different answer. 24 / 30

slide-47
SLIDE 47

5 new properties

◮ Answer Authentication ∗:

◮ All accepted answers are submitted by candidates. ◮ Allow the acceptance of the same answer again. ◮ But, still forbids the acceptance of a different answer.

◮ Answer Authentication Reporting: Collects in a set F every

candidate from which more than one answer are accepted.

24 / 30

slide-48
SLIDE 48

5 new properties

◮ Answer Authentication ∗:

◮ All accepted answers are submitted by candidates. ◮ Allow the acceptance of the same answer again. ◮ But, still forbids the acceptance of a different answer.

◮ Answer Authentication Reporting: Collects in a set F every

candidate from which more than one answer are accepted.

◮ Answer Editing: A candidate cannot change an answer after

validation it.

24 / 30

slide-49
SLIDE 49

5 new properties

◮ Answer Authentication ∗:

◮ All accepted answers are submitted by candidates. ◮ Allow the acceptance of the same answer again. ◮ But, still forbids the acceptance of a different answer.

◮ Answer Authentication Reporting: Collects in a set F every

candidate from which more than one answer are accepted.

◮ Answer Editing: A candidate cannot change an answer after

validation it.

◮ Question Ordering ∗: A candidate cannot changes the answer

to a future question before validating the current question.

24 / 30

slide-50
SLIDE 50

5 new properties

◮ Answer Authentication ∗:

◮ All accepted answers are submitted by candidates. ◮ Allow the acceptance of the same answer again. ◮ But, still forbids the acceptance of a different answer.

◮ Answer Authentication Reporting: Collects in a set F every

candidate from which more than one answer are accepted.

◮ Answer Editing: A candidate cannot change an answer after

validation it.

◮ Question Ordering ∗: A candidate cannot changes the answer

to a future question before validating the current question.

◮ Acceptance Order: A candidate has to validate the questions

in order, but he can skip some questions.

24 / 30

slide-51
SLIDE 51

Results: Exam 1

233 students, 40875 events Property Result Time (ms) Candidate Registration

  • 538

Candidate Eligibility

  • 517

Answer Authentication × 310 Exam Availability

  • 518

Answer Authentication ∗

  • 742

Answer Authentication Reporting ×[1] 654 Answer Editing

  • 641

Question Ordering ∗ × 757 Acceptance Order

  • 697

25 / 30

slide-52
SLIDE 52

Results: Exam 2

90 students, 4641 events Property Result Time (ms) Candidate Registration

  • 230

Candidate Eligibility

  • 214

Answer Authentication

  • 275

Exam Availability ×[1] 237 Answer Authentication ∗

  • 223

Answer Authentication Reporting

  • 265

Answer Editing × 218 Question Ordering ∗ × 389 Acceptance Order

  • 294

26 / 30

slide-53
SLIDE 53

Plan

Introduction Model Properties Case Study: UJF E-exam Conclusion

27 / 30

slide-54
SLIDE 54

Conclusion

◮ Event-based model of e-exams. ◮ Several properties defined as QEAs. ◮ Analysis of 2 real e-exams at UJF using MarQ tool. ◮ Discovering some misbehaviours.

28 / 30

slide-55
SLIDE 55

Future Work

◮ Analyze more existing e-exams from other universities. ◮ Perform on-line verification with our monitors during live

e-exams.

◮ Study more expressive and quantitative properties that can

detect colluded students through similar answer patterns.

◮ Automatic transformation from verifiability to monitors.

29 / 30

slide-56
SLIDE 56

Thank you for your attention!

Questions? pascal.lafourcade@udamail.fr

30 / 30

slide-57
SLIDE 57

Howard Barringer, Yliès Falcone, Klaus Havelund, Giles Reger, and David E. Rydeheard. Quantified event automata: Towards expressive and efficient runtime monitors. In FM 2012: Formal Methods - 18th International Symposium, Paris, France, August 27-31, 2012. Proceedings, volume 7436

  • f Lecture Notes in Computer Science, pages 68–84. Springer,

2012. Giles Reger, Helena Cuenca Cruz, and David E. Rydeheard. MarQ: Monitoring at runtime with QEA. In Tools and Algorithms for the Construction and Analysis of Systems - 21st International Conference, TACAS, London, UK, pages 596–610, 2015.

31 / 30

slide-58
SLIDE 58

Candidate Eligibility

No answer is accepted from an unregistered candidate. ∀i 1 2 register(i)

31 / 30

slide-59
SLIDE 59

Candidate Eligibility with Auditing

Initially: I : ˆ = ∅ 1 2 register(i) I:=I∪{i} accept(i, q, a)

[i / ∈I] F:ˆ ={(i,q,a)}

register(i) I:=I∪{i} accept(i, q, a)

[i / ∈I] F:=F∪{(i,q,a)}

32 / 30

slide-60
SLIDE 60

Answer Authentication

◮ All accepted answers are submitted by candidates. ◮ Exactly one answer is accepted from each candidate.

∀i, ∀q 1 2 3 submit(i, q, a) A:ˆ

={a}

submit(i, q, a) A:=A∪{a} accept(i, q, a) [a∈A] submit(i, q, a)

33 / 30

slide-61
SLIDE 61

Exam Availability

A candidates can take the exam only during the examination time. 1 ΣEA(i, t) [t0≤t≤tf ] 1 2 ΣEA(i, t) [t0>t∨t>tf ]

F:ˆ ={i}

ΣEA(i, t) [t0>t∨t>tf ]

F:=F∪{i} ◮ ΣEA = {get(i, t), change(i, t), submit(i, t), accept(i, t)}. ◮ t0 is the starting instant of the exam. ◮ tf is the ending instant of the exam.

34 / 30

slide-62
SLIDE 62

Exam Availability with Flexibility

Exam Availability with flexible starting time and duration. ∀i 1 2 3 begin(i, t) [t1≤t≤t2]

tb:ˆ =t

accept(i, t) [tb≤t≤t2∧t−tb≤durationi] end(i)

◮ t1 is the starting instant of the allowed period. ◮ t2 is the ending instant of the allowed period.

35 / 30

slide-63
SLIDE 63

Marking Correctness

All answers were marked correctly. ∀q, A : ˆ =∅ 1 2 corrAns(q, a) A:ˆ

=A∪{a}

marked(q, a, b) [(b=1⇔a∈A)] marked(q, a, b) [b=1⇔a∈A]

36 / 30

slide-64
SLIDE 64

Mark Integrity

◮ All accepted answers were marked; ◮ each candidate was assigned the mark attributed to his

answers. ∀i 1 2 3 4 marked(q, a, b) accept(i, q, a) A:ˆ

={(q,a)}

accept(i, q, a) A:=A∪{(q,a)} marked(q, a, b) [(q,a)/

∈A]

marked(q, a, b)

[(q,a)∈A] A:=A\{(q,a)}; s:ˆ =b

marked(q, a, b) [(q,a)/

∈A]

accept(i, q, a) A:=A∪{(q,a)} assign(i, m) [m=s∧A=∅] marked(q, a, b)

37 / 30

slide-65
SLIDE 65

Answer Authentication ∗

A weaker variant of Answer Authentication:

◮ All accepted answers are submitted by candidates. ◮ Allow the acceptance of the same answer again. ◮ But, still forbids the acceptance of a different answer.

∀i, ∀q 1 2 3 submit(i, q, a) A:ˆ

={a}

submit(i, q, a) A:=A∪{a} accept(i, q, a) [a∈A]

av:=a

submit(i, q, a) accept(i, q, a) [a=av] Motivation: UJF exam allows the acceptance of the same answer twice.

38 / 30

slide-66
SLIDE 66

Answer Authentication Reporting

Collects in a set F every candidate from which more than one answer are accepted. Global:F : ˆ =∅ ∀q 1 2 3 accept(i, q, a) A:ˆ

={i}

accept(i, q, a)

[i / ∈A] A:=A∪{i}

accept(i, q, a)

[i∈A] F:ˆ ={i}

accept(i, q, a)

[i / ∈A] A=A∪{i}

accept(i, q, a)

[i∈A] F=F∪{i}

39 / 30

slide-67
SLIDE 67

Answer Editing

A candidate cannot change an answer after validation it. ∀i, ∀q 1 2 change(i, q) accept(i, q, av) accept(i, q, a) [a=av] Motivation: UJF exam does not allow a candidate to change any of the previously validated answers.

40 / 30

slide-68
SLIDE 68

Question Ordering ∗

A candidate cannot changes the answer to a future question before validating the current question. ∀i 1 2 3 change(i, q) [ord(q)=1] accept(i, q) [ord(q)=1]

c:ˆ =2

change(i, q) [ord(q)<c] accept(i, q) [ord(q)<c] accept(i, q) [ord(q)=c]

c++

change(i, q) [ord(q)=c] change(i, q) [ord(q)≤c] accept(i, q) [ord(q)<c] accept(i, q) [ord(q)=c]

c++

Motivation: developers did not log anything related to the event get(i, q) (needed for Question Ordering).

41 / 30

slide-69
SLIDE 69

Acceptance Order

A candidate has to validate the questions in order, but he can skip some questions. ∀i, c : ˆ =1 1 accept(i, q) [ordq≥c]

c:=ordq

Motivation: allows us to check if candidates answer the question in lexicographic order when Question Ordering ∗ fails. It is the case when a candidate able to skip some questions.

42 / 30