Improving Automated Feedback Building a Rule Feedback Generator - - PowerPoint PPT Presentation

▶
improving automated feedback
SMART_READER_LITE
LIVE PREVIEW

Improving Automated Feedback Building a Rule Feedback Generator - - PowerPoint PPT Presentation

Dec 16, 2022 320 likes •1.11k views

Improving Automated Feedback Improving Automated Feedback Building a Rule Feedback Generator Eric Bouwers September 27, 2007 Eric Bouwers Improving Automated Feedback (Generic) Outline The problem 1 Our solution 2 Evaluation 3

slide-1
SLIDE 1

Improving Automated Feedback

Improving Automated Feedback

Building a Rule Feedback Generator Eric Bouwers September 27, 2007

Eric Bouwers

slide-2
SLIDE 2

Improving Automated Feedback

(Generic) Outline

1

The problem

2

Our solution

3

Evaluation

4

Conclusion

Eric Bouwers

slide-3
SLIDE 3

Improving Automated Feedback > The problem

Procedural skills

Eric Bouwers

slide-4
SLIDE 4

Improving Automated Feedback > The problem

Feedback

Eric Bouwers

slide-5
SLIDE 5

Improving Automated Feedback > The problem

Feedback in Education

Eric Bouwers

slide-6
SLIDE 6

Improving Automated Feedback > The problem

Feedback in Education

Eric Bouwers

slide-7
SLIDE 7

Improving Automated Feedback > The problem

Current solutions

Eric Bouwers

slide-8
SLIDE 8

Improving Automated Feedback > The problem

Current solutions

Eric Bouwers

slide-9
SLIDE 9

Improving Automated Feedback > The problem

Current solutions

Eric Bouwers

slide-10
SLIDE 10

Improving Automated Feedback > The problem

Current solutions

Eric Bouwers

slide-11
SLIDE 11

Improving Automated Feedback > The problem

Current solutions

Eric Bouwers

slide-12
SLIDE 12

Improving Automated Feedback > The problem

Current solutions

Eric Bouwers

slide-13
SLIDE 13

Improving Automated Feedback > The problem

Current solutions

Eric Bouwers

slide-14
SLIDE 14

Improving Automated Feedback > The problem

Current solutions

Eric Bouwers

slide-15
SLIDE 15

Improving Automated Feedback > The problem

Synopsis

Feedback is important in learning Personal feedback is not feasible without (computerized) help Current solutions are either:

Only Correct/Incorrect solutions Result of extensive research

Eric Bouwers

slide-16
SLIDE 16

Improving Automated Feedback > Our solution

Goal

Research question How can we make the generation of high-quality, domain-specific feedback easier?

Eric Bouwers

slide-17
SLIDE 17

Improving Automated Feedback > Our solution

Goal

Research question How can we make the generation of high-quality, domain-specific feedback easier? Basic idea Create a generic framework which separates the knowledge of rule-feedback generation from knowledge about the domain.

Eric Bouwers

slide-18
SLIDE 18

Improving Automated Feedback > Our solution

Requirements

Should always be able to produce basic feedback

Eric Bouwers

slide-19
SLIDE 19

Improving Automated Feedback > Our solution

Requirements

Should always be able to produce basic feedback More detailed/complete input leads to better feedback

Eric Bouwers

slide-20
SLIDE 20

Improving Automated Feedback > Our solution

Requirements

Should always be able to produce basic feedback More detailed/complete input leads to better feedback Can be instantiated on different domains (with little effort)

Eric Bouwers

slide-21
SLIDE 21

Improving Automated Feedback > Our solution

Requirements

Should always be able to produce basic feedback More detailed/complete input leads to better feedback Can be instantiated on different domains (with little effort) Adaptable to a single class-room (or student!)

Eric Bouwers

slide-22
SLIDE 22

Improving Automated Feedback > Our solution

Overall design

Eric Bouwers

slide-23
SLIDE 23

Improving Automated Feedback > Our solution

Overall design

Phase 1: Input: CT + PT Output: Correct/Incorrect message

Eric Bouwers

slide-24
SLIDE 24

Improving Automated Feedback > Our solution

Overall design

Phase 2: Input: CT + PT + Student rule Output: Correct/Incorrect message-tuple

Eric Bouwers

slide-25
SLIDE 25

Improving Automated Feedback > Our solution

Overall design

Phase 3: Input: CT + PT + Allowed rules Output: Configurable message, phase level

Eric Bouwers

slide-26
SLIDE 26

Improving Automated Feedback > Our solution

Overall design

Phase 3: Input: CT + PT + Allowed rules Output: Configurable message, phase level Example Message Unfortunately, this step is not correct. I think you wanted to apply this rule: [RULE]. This would result in [PTAPPLIEDRULE]. Can you see the difference with [CT]?

Eric Bouwers

slide-27
SLIDE 27

Improving Automated Feedback > Our solution

Overall design

Phase 4: Input: CT + PT + Allowed rules + Buggy rules Output: Configurable message, rule level

Eric Bouwers

slide-28
SLIDE 28

Improving Automated Feedback > Our solution

First phase

Input: PT, CT Output: Correct or Incorrect Algorithm: Solve both terms and check whether their results are semantically equal.

Eric Bouwers

slide-29
SLIDE 29

Improving Automated Feedback > Our solution

First phase

Input: PT, CT Output: Correct or Incorrect Algorithm: Solve both terms and check whether their results are semantically equal. Implementation firstPhase :: RFG a => RFGSolver a -> RFGEqual a

  • > a -> a -> String

firstPhase solve equal pt ct = let resultCt = solve ct resultPt = solve pt in if equal resultPt resultCt then getConfig Correct else getConfig Incorrect

Eric Bouwers

slide-30
SLIDE 30

Improving Automated Feedback > Our solution

First phase

Input: PT, CT Output: Correct or Incorrect Algorithm: Solve both terms and check whether their results are semantically equal. Implementation firstPhase :: RFG a => RFGSolver a -> RFGEqual a

  • > a -> a -> String

firstPhase solve equal pt ct = let resultCt = solve ct resultPt = solve pt in if equal resultPt resultCt then getConfig Correct else getConfig Incorrect

Eric Bouwers

slide-31
SLIDE 31

Improving Automated Feedback > Our solution

First phase

Input: PT, CT Output: Correct or Incorrect Algorithm: Solve both terms and check whether their results are semantically equal. Implementation firstPhase :: RFG a => RFGSolver a -> RFGEqual a

  • > a -> a -> String

firstPhase solve equal pt ct = let resultCt = solve ct resultPt = solve pt in if equal resultPt resultCt then getConfig Correct else getConfig Incorrect

Eric Bouwers

slide-32
SLIDE 32

Improving Automated Feedback > Our solution

Third phase

Input: PT, CT and allowed rules Output: Configurable message

Eric Bouwers

slide-33
SLIDE 33

Improving Automated Feedback > Our solution

Third phase

Input: PT, CT and allowed rules Output: Configurable message Algorithm Given two terms, calculate the rewrite rule δ between these terms. Determine which allowed rule is closest to δ. Apply rules from the set to the PT if possible and recurse. The result is the rule which is a closest match.

Eric Bouwers

slide-34
SLIDE 34

Improving Automated Feedback > Our solution

Third phase

Input: PT, CT and allowed rules Output: Configurable message Algorithm Given two terms, calculate the rewrite rule δ between these terms. Determine which allowed rule is closest to δ. Apply rules from the set to the PT if possible and recurse. The result is the rule which is a closest match.

Eric Bouwers

slide-35
SLIDE 35

Improving Automated Feedback > Our solution

Third phase

Input: PT, CT and allowed rules Output: Configurable message Algorithm Given two terms, calculate the rewrite rule δ between these terms. Determine which allowed rule is closest to δ. Apply rules from the set to the PT if possible and recurse. The result is the rule which is a closest match.

Eric Bouwers

slide-36
SLIDE 36

Improving Automated Feedback > Our solution

Calculating rewrite rules

PT = 1 + 5 + 3 CT = 3 + 6

Eric Bouwers

slide-37
SLIDE 37

Improving Automated Feedback > Our solution

Calculating rewrite rules

PT = 1 + 5 + 3 CT = 3 + 6

Eric Bouwers

slide-38
SLIDE 38

Improving Automated Feedback > Our solution

Calculating rewrite rules

PT = 1 + 5 + 3 CT = 3 + 6

Eric Bouwers

slide-39
SLIDE 39

Improving Automated Feedback > Our solution

Calculating rewrite rules

PT = 1 + 5 + 3 CT = 3 + 6

Eric Bouwers

slide-40
SLIDE 40

Improving Automated Feedback > Our solution

Calculating rewrite rules

PT = 1 + 5 + 3 CT = 3 + 6 Non-Equal ([1,5,3],[3,6, ])

Eric Bouwers

slide-41
SLIDE 41

Improving Automated Feedback > Our solution

Calculating rewrite rules

PT = 1 + 5 + 3 CT = 3 + 6 Equal, Associative ([1,5,3],[3,6, ]) , ([1,5,3],[ ,3,6])

Eric Bouwers

slide-42
SLIDE 42

Improving Automated Feedback > Our solution

Calculating rewrite rules

PT = 1 + 5 + 3 CT = 3 + 6 Equal, Associative and Commutative ([1,5,3],[3,6, ]), ([1,5,3],[ ,3,6]), ([1,5,3],[6,3, ]), ([1,5,3],[ ,6,3]), ([1,3,5],[3,6, ]), ([1,3,5],[ ,3,6]), ([1,3,5],[6,3, ]), ([1,3,5],[ ,6,3]), ([3,1,5],[3,6, ]), ([3,1,5],[ ,3,6]), ([3,1,5],[6,3, ]), ([3,1,5],[ ,6,3]), ([3,5,1],[3,6, ]), ([3,5,1],[ ,3,6]), ([3,5,1],[6,3, ]), ([3,5,1],[ ,6,3]), ([5,3,1],[3,6, ]), ([5,3,1],[ ,3,6]), ([5,3,1],[6,3, ]), ([5,3,1],[ ,6,3]), ([5,1,3],[3,6, ]), ([5,1,3],[ ,3,6]), ([5,1,3],[6,3, ]), ([5,1,3],[ ,6,3])

Eric Bouwers

slide-43
SLIDE 43

Improving Automated Feedback > Our solution

Calculating rewrite rules

PT = 1 + 5 + 3 CT = 3 + 6 Equal, Associative and Commutative ([1,5,3],[3,6, ]), ([1,5,3],[ ,3,6]), ([1,5,3],[6,3, ]), ([1,5,3],[ ,6,3]), ([1,3,5],[3,6, ]), ([1,3,5],[ ,3,6]), ([1,3,5],[6,3, ]), ([1,3,5],[ ,6,3]), ([3,1,5],[3,6, ]), ([3,1,5],[ ,3,6]), ([3,1,5],[6,3, ]), ([3,1,5],[ ,6,3]), ([3,5,1],[3,6, ]), ([3,5,1],[ ,3,6]), ([3,5,1],[6,3, ]), ([3,5,1],[ ,6,3]), ([5,3,1],[3,6, ]), ([5,3,1],[ ,3,6]), ([5,3,1],[6,3, ]), ([5,3,1],[ ,6,3]), ([5,1,3],[3,6, ]), ([5,1,3],[ ,3,6]), ([5,1,3],[6,3, ]), ([5,1,3],[ ,6,3])

Eric Bouwers

slide-44
SLIDE 44

Improving Automated Feedback > Our solution

Calculating rewrite rules

PT = 1 + 5 + 3 CT = 3 + 6 Result: ([1,5],[6])

Eric Bouwers

slide-45
SLIDE 45

Improving Automated Feedback > Our solution

Calculating rewrite rules

PT = 1 + 5 + 3 CT = 3 + 6 Result: δ = 1 + 5 ⇒ +6

Eric Bouwers

slide-46
SLIDE 46

Improving Automated Feedback > Our solution

Calculating rewrite rules

PT = 1 + 5 + 3 CT = 3 + 6 Result: δ = 1 + 5 ⇒ 6

Eric Bouwers

slide-47
SLIDE 47

Improving Automated Feedback > Our solution

Defining the distance

Distance between rules:

Eric Bouwers

slide-48
SLIDE 48

Improving Automated Feedback > Our solution

Defining the distance

Distance between terms:

Eric Bouwers

slide-49
SLIDE 49

Improving Automated Feedback > Our solution

Defining the distance

Distance between terms: Environment = {} Distance = 0

Eric Bouwers

slide-50
SLIDE 50

Improving Automated Feedback > Our solution

Defining the distance

Distance between terms: Environment = {} Distance = 0

Eric Bouwers

slide-51
SLIDE 51

Improving Automated Feedback > Our solution

Defining the distance

Distance between terms: Environment = {} Distance = 2

Eric Bouwers

slide-52
SLIDE 52

Improving Automated Feedback > Our solution

Defining the distance

Distance between terms: Environment = {} Distance = 2

Eric Bouwers

slide-53
SLIDE 53

Improving Automated Feedback > Our solution

Defining the distance

Distance between terms: Environment = {(A, 1)} Distance = 2

Eric Bouwers

slide-54
SLIDE 54

Improving Automated Feedback > Our solution

Defining the distance

Distance between terms: Environment = {(A, 1)} Distance = 2

Eric Bouwers

slide-55
SLIDE 55

Improving Automated Feedback > Our solution

Defining the distance

Distance between terms: Environment = {(A, 1)} Distance = 4

Eric Bouwers

slide-56
SLIDE 56

Improving Automated Feedback > Our solution

Example

PT = (a ∨ b) → c CT = (¬a ∧ b) ∨ c Rules = { ImpElimination : A → B ⇒ ¬(A) ∨ B , MorganOr : ¬(A ∨ B) ⇒ ¬(A) ∧ ¬(B) }

Eric Bouwers

slide-57
SLIDE 57

Improving Automated Feedback > Our solution

Example

PT = (a ∨ b) → c CT = (¬a ∧ b) ∨ c Rules = { ImpElimination : A → B ⇒ ¬(A) ∨ B , MorganOr : ¬(A ∨ B) ⇒ ¬(A) ∧ ¬(B) } Calculated Rule δ = (a ∨ b) → c ⇒ (¬a ∧ b) ∨ c

Eric Bouwers

slide-58
SLIDE 58

Improving Automated Feedback > Our solution

Example

PT = (a ∨ b) → c CT = (¬a ∧ b) ∨ c Rules = { ImpElimination : A → B ⇒ ¬(A) ∨ B , MorganOr : ¬(A ∨ B) ⇒ ¬(A) ∧ ¬(B) } Calculated Rule δ = (a ∨ b) → c ⇒ (¬a ∧ b) ∨ c Distances dist(δ, ImpElimination) = 9 dist(δ, MorganOr) = 15

Eric Bouwers

slide-59
SLIDE 59

Improving Automated Feedback > Our solution

Example

PT = (a ∨ b) → c CT = (¬a ∧ b) ∨ c Rules = { ImpElimination : A → B ⇒ ¬(A) ∨ B , MorganOr : ¬(A ∨ B) ⇒ ¬(A) ∧ ¬(B) } Calculated Rule δ = (a ∨ b) → c ⇒ (¬a ∧ b) ∨ c Distances dist(δ, ImpElimination) = 9 dist(δ, MorganOr) = 15

Eric Bouwers

slide-60
SLIDE 60

Improving Automated Feedback > Our solution

Example

PT’ = ¬(a ∨ b) ∨ c CT = (¬a ∧ b) ∨ c Rules = { ImpElimination : A → B ⇒ ¬(A) ∨ B , MorganOr : ¬(A ∨ B) ⇒ ¬(A) ∧ ¬(B) }

Eric Bouwers

slide-61
SLIDE 61

Improving Automated Feedback > Our solution

Example

PT’ = ¬(a ∨ b) ∨ c CT = (¬a ∧ b) ∨ c Rules = { ImpElimination : A → B ⇒ ¬(A) ∨ B , MorganOr : ¬(A ∨ B) ⇒ ¬(A) ∧ ¬(B) } Calculated Rule δ = ¬(a ∨ b) ⇒ (¬a ∧ b)

Eric Bouwers

slide-62
SLIDE 62

Improving Automated Feedback > Our solution

Example

PT’ = ¬(a ∨ b) ∨ c CT = (¬a ∧ b) ∨ c Rules = { ImpElimination : A → B ⇒ ¬(A) ∨ B , MorganOr : ¬(A ∨ B) ⇒ ¬(A) ∧ ¬(B) } Calculated Rule δ = ¬(a ∨ b) ⇒ (¬a ∧ b) Distances dist(δ, ImpElimination) = 9 dist(δ, MorganOr) = 4

Eric Bouwers

slide-63
SLIDE 63

Improving Automated Feedback > Our solution

Example

PT’ = ¬(a ∨ b) ∨ c CT = (¬a ∧ b) ∨ c Rules = { ImpElimination : A → B ⇒ ¬(A) ∨ B , MorganOr : ¬(A ∨ B) ⇒ ¬(A) ∧ ¬(B) } Calculated Rule δ = ¬(a ∨ b) ⇒ (¬a ∧ b) Distances dist(δ, ImpElimination) = 9 dist(δ, MorganOr) = 4

Eric Bouwers

slide-64
SLIDE 64

Improving Automated Feedback > Our solution

Example

PT” = (¬a ∧ ¬b) ∨ c CT = (¬a ∧ b) ∨ c Rules = { ImpElimination : A → B ⇒ ¬(A) ∨ B , MorganOr : ¬(A ∨ B) ⇒ ¬(A) ∧ ¬(B) }

Eric Bouwers

slide-65
SLIDE 65

Improving Automated Feedback > Our solution

Example

PT” = (¬a ∧ ¬b) ∨ c CT = (¬a ∧ b) ∨ c Rules = { ImpElimination : A → B ⇒ ¬(A) ∨ B , MorganOr : ¬(A ∨ B) ⇒ ¬(A) ∧ ¬(B) } Calculated Rule δ = ¬b ⇒ b (

Eric Bouwers

slide-66
SLIDE 66

Improving Automated Feedback > Our solution

Example

PT” = (¬a ∧ ¬b) ∨ c CT = (¬a ∧ b) ∨ c Rules = { ImpElimination : A → B ⇒ ¬(A) ∨ B , MorganOr : ¬(A ∨ B) ⇒ ¬(A) ∧ ¬(B) } Calculated Rule δ = ¬b ⇒ b ( Distances dist(δ, ImpElimination) = 9 dist(δ, MorganOr) = 10

Eric Bouwers

slide-67
SLIDE 67

Improving Automated Feedback > Our solution

Fourth phase

  • M. Hennecke

Online Diagnose in intelligenten mathematischen Lehr-Lern-Systemen. PhD thesis, Hildesheim University, 1999.

Eric Bouwers

slide-68
SLIDE 68

Improving Automated Feedback > Our solution

Fourth phase

  • M. Hennecke

Online Diagnose in intelligenten mathematischen Lehr-Lern-Systemen. PhD thesis, Hildesheim University, 1999. Example

A B + C D ⇒ A+C B+D

: AddError, You can only add fractions with equal denominators : 0.3

Eric Bouwers

slide-69
SLIDE 69

Improving Automated Feedback > Our solution

Overview

Eric Bouwers

slide-70
SLIDE 70

Improving Automated Feedback > Evaluation

Receiving feedback

Eric Bouwers

slide-71
SLIDE 71

Improving Automated Feedback > Evaluation

Feedback message

Eric Bouwers

slide-72
SLIDE 72

Improving Automated Feedback > Evaluation

Wrong Guesses

EqElem : A ↔ B → (A ∧ B) ∨ (¬(A) ∧ ¬(B)) ImpElem : A → B → ¬(A) ∨ B MorganA : ¬(A ∧ B) → ¬(A) ∨ ¬(B) NotNot : ¬¬A → A DistAO : A ∧ (B ∨C) → (A ∧ B) ∨ (A ∧ C) IdemA : A ∧ A → A

Eric Bouwers

slide-73
SLIDE 73

Improving Automated Feedback > Evaluation

Wrong Guesses

EqElem : A ↔ B → (A ∧ B) ∨ (¬(A) ∧ ¬(B)) ImpElem : A → B → ¬(A) ∨ B MorganA : ¬(A ∧ B) → ¬(A) ∨ ¬(B) NotNot : ¬¬A → A DistAO : A ∧ (B ∨C) → (A ∧ B) ∨ (A ∧ C) IdemA : A ∧ A → A PT CT Expected rule Actual output a→(b∧c) a∨(b∧c) ImpElem NotNot ¬¬a↔b (¬a∧b)∨(¬¬¬a∨¬b) EqElem MorganA a∨c↔b (a∨c∧b)∨¬(a∨c)∧¬b EqElem IdemA a↔b∧(c∨d) (a↔b∧c)∨(a↔b∧d) EqElem DistAO

Eric Bouwers

slide-74
SLIDE 74

Improving Automated Feedback > Evaluation

Definition of Rules

Multiplication : (A/B) ∗ (C/D) ⇒ (A ∗ C) / (B ∗ D) Division : (A/B) / (C/D) ⇒ (A/B) ∗ (D/C) SameMixed : A|B/B ⇒ A + 1 AddFractions : (A/B) + (C/B) ⇒ (A + C) / B SubFractions : (A/B) − (C/B) ⇒ (A − C) / B

Eric Bouwers

slide-75
SLIDE 75

Improving Automated Feedback > Evaluation

Definition of Rules

Multiplication : (A/B) ∗ (C/D) ⇒ (A ∗ C) / (B ∗ D) Division : (A/B) / (C/D) ⇒ (A/B) ∗ (D/C) SameMixed : A|B/B ⇒ A + 1 AddFractions : (A/B) + (C/B) ⇒ (A + C) / B SubFractions : (A/B) − (C/B) ⇒ (A − C) / B SolveMin : A − B ⇒ C where C := solve(A − B) SolveAdd : A + B ⇒ C where C := solve(A + B) SolveMul : A ∗ B ⇒ C where C := solve(A ∗ B)

Eric Bouwers

slide-76
SLIDE 76

Improving Automated Feedback > Conclusion

Conclusion

Partial/Configurable approach is possible Adding an extra domain takes little effort Combining techniques leads to surprisingly good results Already useful, both for research as well as practical

Eric Bouwers

slide-77
SLIDE 77

Improving Automated Feedback > Conclusion

Resources

Thesis page: http://www.cs.uu.nl/wiki/Students/EricBouwersThesisPage Prototype: http://ideas.cs.uu.nl/∼eric/ Contact me: EricBouwers@gmail.com

Eric Bouwers