Teaching Multiple Concepts to Forgetful Learners Yuxin Chen - - PowerPoint PPT Presentation

β–Ά
teaching multiple concepts to forgetful learners
SMART_READER_LITE
LIVE PREVIEW

Teaching Multiple Concepts to Forgetful Learners Yuxin Chen - - PowerPoint PPT Presentation

Jul 02, 2023 210 likes β€’420 views

Teaching Multiple Concepts to Forgetful Learners Yuxin Chen chenyuxin@uchicago.edu Oisin Mac Aodha Manuel Gomez Anette Hunziker Yuxin Chen Andreas Krause Pietro Perona Yisong Yue Adish Singla Rodriguez Applications: Language learning

slide-1
SLIDE 1

Teaching Multiple Concepts to Forgetful Learners

Yuxin Chen chenyuxin@uchicago.edu

Yisong Yue Oisin Mac Aodha Anette Hunziker Manuel Gomez Rodriguez Pietro Perona Adish Singla Andreas Krause Yuxin Chen

slide-2
SLIDE 2

Applications: Language learning

2

  • Over 300+ million students
  • Based on spaced repetition of flash cards
  • Can we compute optimal personalized schedule of repetition?
slide-3
SLIDE 3

Teaching Interaction Using Flashcards

3

Interaction at time 𝒖 = 𝟐, πŸ‘, … 𝑼

  • 1. Teacher displays a flashcard 𝑦𝑒 ∈ {1,2, . . , π‘œ}
  • 2. Learner’s recall is 𝑧𝑒 ∈ 0, 1
  • 3. Teacher provides the correct answer

1 3 2

jouet Submit Answer: Spielzeug x jouet

1 3 2

jouet Submit

Learning Phase (1)

Answer: Spielzeug x jouet Submit

Learning Phase (3)

Answer: Nachtisch x Buch Submit

Learning Phase (4)

Answer: Buch βœ“ Buch nachs Submit

Learning Phase (5)

Answer: Nachtisch x nachs Nachtisch Submit

Learning Phase (6)

Answer: Nachtisch βœ“ Nachtisch Spielzeug Submit

Learning Phase (2)

Answer: Spielzeug βœ“ Spielzeug

slide-4
SLIDE 4

Background on Teaching Policies

4

Example setup

  • π‘ˆ = 20 and π‘œ = 5 concepts given by 𝑏, 𝑐, 𝑑, 𝑒, 𝑓

NaΓ―ve teaching policies

  • Random:
  • Round-robin: 𝑏 β†’ 𝑐 β†’ 𝑑 β†’ 𝑒 β†’ 𝑓 β†’ 𝑏 β†’ 𝑐 β†’ 𝑑 β†’ 𝑒 β†’ 𝑓 β†’ 𝑏 β†’ 𝑐 β†’ 𝑑 β†’ 𝑒 β†’ 𝑓 β†’ 𝑏 β†’

𝑏 β†’ 𝑐 β†’ 𝑏 β†’ 𝑓 β†’ 𝑑 β†’ 𝑒 β†’ 𝑏 β†’ 𝑒 β†’ 𝑑 β†’ 𝑏 β†’ 𝑐 β†’ 𝑓 β†’ 𝑏 β†’ 𝑐 β†’ 𝑒 β†’ 𝑓 β†’

Key limitation: Schedule agnostic to learning process

slide-5
SLIDE 5

Background: Pimsieur Method (1967)

5

Used in mainstream language learning platforms Based on spaced repetition ideas

𝑏 β†’ 𝑐 β†’ 𝑏 β†’ 𝑐 β†’ 𝑑 β†’ 𝑏 β†’ 𝑑 β†’ 𝑐 β†’ 𝑒 β†’ 𝑑 β†’ 𝑒 β†’ 𝑏 β†’ 𝑐 β†’ 𝑒 β†’ 𝑑 β†’ 𝑓 β†’ 𝑏 β†’ 𝑐 β†’ 𝑏 β†’ 𝑐 β†’ 𝑑 β†’ 𝑏 β†’ 𝑑 β†’ 𝑐 β†’ 𝑒 β†’ 𝑑 β†’ 𝑒 β†’ 𝑏 β†’ 𝑐 β†’ 𝑒 β†’ 𝑑 β†’ 𝑓 β†’

slide-6
SLIDE 6

Background: Leitner System (1972)

6

Adaptive spacing intervals

𝑏 β†’ 𝑐 β†’ 𝑏 β†’ 𝑐 β†’ 𝑑 β†’ 𝑏 β†’ 𝑑 β†’ 𝑐 β†’ 𝑒 β†’ 𝑑 β†’ 𝑒 β†’ 𝑏 β†’ 𝑐 β†’ 𝑒 β†’ 𝑑 β†’ 𝑓 β†’

Student 1:

𝑏 β†’ 𝑏 β†’ 𝑐 β†’ 𝑏 β†’ 𝑐 β†’ 𝑑 β†’ 𝑏 β†’ 𝑑 β†’ 𝑏 β†’ 𝑐 β†’ 𝑑 β†’ 𝑏 β†’ 𝑐 β†’ 𝑏 β†’ 𝑒 β†’ 𝑑 β†’

Student 2:

Key limitation: No guarantees on the optimality of the schedule

slide-7
SLIDE 7

Modeling Forgetfulness pi(t | history) = 2βˆ’ βˆ†ti

hi

Recall Probability

  • f Concept i:

Time t Time since last teaching concept Half-life estimate (depends on feedback)

hi += ai hi += bi

Half-life Regression (HRL) model [Settles & Meeder, ACL 2016]

slide-8
SLIDE 8

Interactive Teaching Protocol

  • For t = 1…T
  • Teacher chooses concept π‘—πœ— 1, … , 𝑛 (e.g., a flashcard)
  • Learner tries to recall concept (success or fail)
  • Teacher reveals answer (e.g., β€œSpielzug”)
  • Goal: maximize

𝑔 history = 1 𝑛 1 π‘ˆ B

"#$ %

B

&#$ '

π‘žπ‘— 𝑒 | history$:&)$

β€œArea Under Curve”

slide-9
SLIDE 9

Naive Approaches

  • Round Robin
  • Doesn’t adapt to new estimates of learner recall probabilities
  • Over-teaches easy concepts
  • Under-teaches hard concepts
  • Lowest Recall Probability
  • Generalization of Pimsleur method and Leitner system
  • Doesn’t consider change to recall probability
slide-10
SLIDE 10

Greedy Teaching Algorithm (interactive)

  • Choose concept i to maximize

Ξ” 𝑗 history = 𝐹*! 𝑔 history⨁ 𝑗, 𝑧& βˆ’ 𝑔(history)

yt: success or failure of recall at time t

pi(t | history) = 2βˆ’ βˆ†ti

hi

(randomness over model estimate) (hi updated after observing yt)

slide-11
SLIDE 11

Characteristics of the Optimization Problem

11

  • Non-submodular
  • Gain of a concept 𝑦 can increase given longer history
  • Captured by submodularity ratio 𝛿 over sequences
slide-12
SLIDE 12

Characteristics of the Optimization Problem (cont.)

12

  • Post-fix non-monotone
  • 𝑔 orange ⨁ blue < 𝑔 blue
  • Captured by curvature Ο‰
slide-13
SLIDE 13

Theoretical Guarantees: General Case

13

  • Guarantees for the general case (any memory model)
  • Utility of 𝜌gr (greedy policy) compared to 𝜌opt is given by

Theorem 1 Corollary 2 𝐺 𝜌!" β‰₯ 𝐺 𝜌#$%

&'( )

𝛿)*& π‘ˆ 2

+', &*(

1 βˆ’ πœ•+ 5 𝛿+ π‘ˆ β‰₯ 𝐺 𝜌#$% 1 πœ•-./ 1 βˆ’ 𝑓*0!"# 1 2!$%

slide-14
SLIDE 14

Theoretical Guarantees: HLR Model

14

  • Consider the task of teaching π‘œ concepts where each concept is

following an independent HLR model with the same parameters 𝑏0 = 𝑨, 𝑐0 = 𝑨 βˆ€ 𝑦 ∈ {1,2, . . , π‘œ}. A sufficient condition for the algorithm to achieve (1 βˆ’ πœ—) high utility is z β‰₯ max {log π‘ˆ, log 3π‘œ , log

12" 3' }

slide-15
SLIDE 15

Illustration: Simulation Results

Greedy Round Robin Optimal Objective

slide-16
SLIDE 16

User Study

150 participants from Mechanical Turk platform T=40, m=15, total study time is about 25 mins

slide-17
SLIDE 17

17

German

GR LR RR RD

  • Avg. gain

0.572 0.487 0.462 0.467

p-value

  • 0.0652

0.0197 0.0151

Figure 6: Samples from the German dataset

slide-18
SLIDE 18

18

Biodiversity (all species) Biodiversity (rare species)

GR LR RR RD

  • Avg. gain

0.475 0.411 0.390 0.251

p-value

  • 0.0017

0.0001 0.0001

GR LR RR RD

  • Avg. gain

0.766 0.668 0.601 0.396

p-value

  • 0.0001

0.0001 0.0001 (a) Common: Owl, Cat, Horse, Elephant, Lion, Tiger, Bear (b) Rare: Angwantibo, Olinguito, Axolotl, Ptarmigan, Patrijshond, Coelacanth, Pyrrhuloxia

slide-19
SLIDE 19

Summary: Teaching Concepts to People

19

  • Teaching forgetful learners
  • Limited memory (modeling forgetfulness)
  • Engagement (interface design)
  • Challenges not covered in this talk:
  • Limited inference power and noise
  • Mismatch in representation
  • Interpretability (e.g., teaching via labels vs. rich feedback)
  • Safety (e.g., when teaching physical tasks)
  • Fairness (e.g., when teaching a class)
  • …

Questions?