Online Meta-Learning Chelsea Finn*, Aravind Rajeswaran*, Sham - - PowerPoint PPT Presentation

Online Meta-Learning

Chelsea Finn*, Aravind Rajeswaran*, Sham Kakade, Sergey Levine

Deep networks + large datasets =

In many prac9cal situa9ons: Learn new task with only a few datapoints Deep networks + large datasets =

In many prac9cal situa9ons: Learn new task with only a few datapoints Deep networks + large datasets = Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

In many prac9cal situa9ons: Learn new task with only a few datapoints Deep networks + large datasets = Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

In many prac9cal situa9ons: Learn new task with only a few datapoints Deep networks + large datasets = Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently learn

In many prac9cal situa9ons: Learn new task with only a few datapoints Deep networks + large datasets = Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

More realis9cally:

learn

In many prac9cal situa9ons: Learn new task with only a few datapoints Deep networks + large datasets = Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

More realis9cally:

learn learn

Nme

In many prac9cal situa9ons: Learn new task with only a few datapoints Deep networks + large datasets = Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

More realis9cally:

learn learn learn

Nme

In many prac9cal situa9ons: Learn new task with only a few datapoints Deep networks + large datasets = Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

More realis9cally:

learn learn learn learn

Nme

In many prac9cal situa9ons: Learn new task with only a few datapoints Deep networks + large datasets = Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

More realis9cally:

learn learn learn learn learn

Nme

In many prac9cal situa9ons: Learn new task with only a few datapoints Deep networks + large datasets = Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

More realis9cally:

learn learn learn learn learn learn

Nme

In many prac9cal situa9ons: Learn new task with only a few datapoints Deep networks + large datasets = Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

More realis9cally:

learn learn learn learn learn learn learn

Nme

In many prac9cal situa9ons: Learn new task with only a few datapoints Deep networks + large datasets = Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

More realis9cally:

learn learn learn learn learn learn learn learn

Nme

In many prac9cal situa9ons: Learn new task with only a few datapoints Deep networks + large datasets = Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

More realis9cally:

learn learn learn learn learn learn learn slow learning rapid learning learn

Nme

Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

Online Learning

(Hannan ’57, Zinkevich ’03)

Perform sequence of tasks while minimizing staNc regret.

learn Nme

perform perform perform perform perform perform perform

Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

Online Learning

(Hannan ’57, Zinkevich ’03)

Perform sequence of tasks while minimizing staNc regret.

learn Nme

perform perform perform perform perform perform perform

zero-shot performance

Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

Online Learning

(Hannan ’57, Zinkevich ’03)

Perform sequence of tasks while minimizing staNc regret.

Online Meta-Learning

(this work)

Efficiently learn a sequence of tasks from a non-staNonary distribuNon.

learn Nme

perform perform perform perform perform perform perform

zero-shot performance

Meta-Learning

(Schmidhuber et al. ’87, Bengio et al. ’92)

Given i.i.d. task distribuNon, learn a new task efficiently

Online Learning

(Hannan ’57, Zinkevich ’03)

Perform sequence of tasks while minimizing staNc regret.

Online Meta-Learning

(this work)

Efficiently learn a sequence of tasks from a non-staNonary distribuNon.

learn Nme

learn learn learn learn learn learn learn

Nme

perform perform perform perform perform perform perform

zero-shot performance performance a?er seeing a small amount of data

The Online Meta-Learning Se6ng

The Online Meta-Learning Se6ng

Space of parameters ✓ ∈ Θ ⊆ Rd and loss functions ` : Θ → R For round t ∈ {1, 2, . . . ∞}:

The Online Meta-Learning Se6ng

Space of parameters ✓ ∈ Θ ⊆ Rd and loss functions ` : Θ → R For round t ∈ {1, 2, . . . ∞}:

• 1. World picks a loss function `t(·)
• 2. Agent should pick ✓t without knowledge of `t

round : t

`t(·)

θt

The Online Meta-Learning Se6ng

Space of parameters ✓ ∈ Θ ⊆ Rd and loss functions ` : Θ → R For round t ∈ {1, 2, . . . ∞}:

• 1. World picks a loss function `t(·)
• 2. Agent should pick ✓t without knowledge of `t
• 3. Agent uses update procedure Φt : Θ → Θ, and obtains ˜

✓t = Φt(✓t)

round : t

`t(·)

θt

˜ θt = Φt(θt)

The Online Meta-Learning Se6ng

Space of parameters ✓ ∈ Θ ⊆ Rd and loss functions ` : Θ → R For round t ∈ {1, 2, . . . ∞}:

• 1. World picks a loss function `t(·)
• 2. Agent should pick ✓t without knowledge of `t
• 3. Agent uses update procedure Φt : Θ → Θ, and obtains ˜

✓t = Φt(✓t)

round : t

`t(·)

θt

˜ θt = Φt(θt)

˜ ✓t = ✓t ↵rˆ `t(✓t)

The Online Meta-Learning Se6ng

Space of parameters ✓ ∈ Θ ⊆ Rd and loss functions ` : Θ → R For round t ∈ {1, 2, . . . ∞}:

• 1. World picks a loss function `t(·)
• 2. Agent should pick ✓t without knowledge of `t
• 3. Agent uses update procedure Φt : Θ → Θ, and obtains ˜

✓t = Φt(✓t)

• 4. Agent suffers `t(˜

✓t) for the round

round : t

`t(·)

θt

˜ ✓t = ✓t ↵rˆ `t(✓t)

˜ θt = Φt(θt)

The Online Meta-Learning Se6ng

Space of parameters ✓ ∈ Θ ⊆ Rd and loss functions ` : Θ → R For round t ∈ {1, 2, . . . ∞}:

• 1. World picks a loss function `t(·)
• 2. Agent should pick ✓t without knowledge of `t
• 3. Agent uses update procedure Φt : Θ → Θ, and obtains ˜

✓t = Φt(✓t)

• 4. Agent suffers `t(˜

✓t) for the round

round : t

`t(·)

θt

˜ ✓t = ✓t ↵rˆ `t(✓t)

˜ θt = Φt(θt)

RegretT :=

T

X

t=1

`t(Φt(✓t)) − min

θ∈Θ T

X

t=1

`t(Φt(✓))

Goal: Learning algorithm with sub-linear

Loss of algorithm Loss of best algorithm
 in hindsight

✓t+1 = arg min

θ T

X

t=1

`t(Φt(✓))

Follow the Meta-Leader (FTML) :

Can be implemented with MAML

✓t+1 = arg min

θ T

X

t=1

`t(Φt(✓))

Follow the Meta-Leader (FTML) :

Can be implemented with MAML

Theorem (Informal): If {`t(·), ˆ `t(·)} ∀t are C2-smooth and strongly convex, the sequence of models {✓1, ✓2, . . . , ✓T } returned by FTML has the property: RegretT :=

T

X

t=1

`t(Φt(✓t)) − min

θ∈Θ T

X

t=1

`t(Φt(✓)) = O(log T)

= ⇒ Avg. Regret = RegretT T → 0 as T → ∞

Learning in a sequen9al non-sta9onary seDng, but s9ll compe99ve
 with best meta-learner in hindsight!

✓t+1 = arg min

θ T

X

t=1

`t(Φt(✓))

Follow the Meta-Leader (FTML) :

Can be implemented with MAML

Theorem (Informal): If {`t(·), ˆ `t(·)} ∀t are C2-smooth and strongly convex, the sequence of models {✓1, ✓2, . . . , ✓T } returned by FTML has the property: RegretT :=

T

X

t=1

`t(Φt(✓t)) − min

θ∈Θ T

X

t=1

`t(Φt(✓)) = O(log T)

FTML: prac9cal instan9a9on of our approach, extending MAML1 meta-train on all data so far, fine-tune on current task

[1] Finn et al. ICML ’17

Experiment with sequences of tasks: FTML: prac9cal instan9a9on of our approach, extending MAML1 meta-train on all data so far, fine-tune on current task

[1] Finn et al. ICML ’17

Experiment with sequences of tasks:

• Colored, rotated, scaled MNIST

FTML: prac9cal instan9a9on of our approach, extending MAML1 meta-train on all data so far, fine-tune on current task

[1] Finn et al. ICML ’17

Experiment with sequences of tasks:

• Colored, rotated, scaled MNIST
• 3D object pose predicHon

FTML: prac9cal instan9a9on of our approach, extending MAML1

Example pose predicNon tasks plane car chair

meta-train on all data so far, fine-tune on current task

[1] Finn et al. ICML ’17

Experiment with sequences of tasks:

• Colored, rotated, scaled MNIST
• 3D object pose predicHon
• CIFAR-100 classificaNon

FTML: prac9cal instan9a9on of our approach, extending MAML1

Example pose predicNon tasks plane car chair

meta-train on all data so far, fine-tune on current task

[1] Finn et al. ICML ’17

Experiments

task index

Experiments

Learning efficiency


(# datapoints)

task index Learning proficiency


(error)

task index

Experiments

Learning efficiency


(# datapoints)

Rainbow MNIST Pose Predic9on task index Learning proficiency


(error)

FTML (ours) Rainbow MNIST Pose Predic9on

task index

Experiments

Learning efficiency


(# datapoints)

Rainbow MNIST Pose Predic9on task index Learning proficiency


(error)

FTML (ours) learns each new task faster & with greater proficiency, Rainbow MNIST Pose Predic9on

task index

Experiments

Learning efficiency


(# datapoints)

Rainbow MNIST Pose Predic9on task index Learning proficiency


(error)

FTML (ours) learns each new task faster & with greater proficiency, approaches few-shot learning regime Rainbow MNIST Pose Predic9on

Takeaways

Introduced online meta-learning problem formulaNon Meta-learning is effecNve in non-staHonary se6ngs Similar guarantees to online learning, but beQer empirical performance

For more, come see us at poster #5!

Nme