[PPT] - Bundles, Lenses & Machine Learning Motivation Backprop as PowerPoint Presentation

SLIDE 1

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Bundles, Lenses & Machine Learning

Jules Hedges1 joint work with Brendan Fong2 Eliana Lorch3 David Spivak2

1Max Planck Institute for Mathematics in the Sciences 2MIT 3University of Oxford

SYCO 5, Birmingham

SLIDE 2

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Bundles, Lenses & Machine Learning

Jules Hedges1 joint work with Brendan Fong2 Eliana Lorch3 David Spivak2

1Max Planck Institute for Mathematics in the Sciences 2MIT 3University of Oxford

SYCO 5, Birmingham

Featuring zero string diagrams :(

SLIDE 3

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Motivation

Machine learning is categorical in 2 different ways: Backprop As Functor (compositional description of ML with monoidal categories) + ML as differential geometry

SLIDE 4

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Motivation

Machine learning is categorical in 2 different ways: Backprop As Functor (compositional description of ML with monoidal categories) + ML as differential geometry In this talk: smoosh them together (why? Why not)

SLIDE 5

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Motivation

Machine learning is categorical in 2 different ways: Backprop As Functor (compositional description of ML with monoidal categories) + ML as differential geometry In this talk: smoosh them together (why? Why not)

It clarifies Backprop as Functor more than anything else

SLIDE 6

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Open learners

Definition (Fong, Spivak & Tuy´ eras) : An open learner X → Y consists of:

A set P of parameters

SLIDE 7

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Open learners

Definition (Fong, Spivak & Tuy´ eras) : An open learner X → Y consists of:

A set P of parameters
A function I : P × X → Y (the implementation)

SLIDE 8

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Open learners

Definition (Fong, Spivak & Tuy´ eras) : An open learner X → Y consists of:

A set P of parameters
A function I : P × X → Y (the implementation)
A function u : P × X × Y → P (the update)

SLIDE 9

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Open learners

Definition (Fong, Spivak & Tuy´ eras) : An open learner X → Y consists of:

A set P of parameters
A function I : P × X → Y (the implementation)
A function u : P × X × Y → P (the update)
A function r : P × X × Y → X (the request)

SLIDE 10

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Open learners

Definition (Fong, Spivak & Tuy´ eras) : An open learner X → Y consists of:

A set P of parameters
A function I : P × X → Y (the implementation)
A function u : P × X × Y → P (the update)
A function r : P × X × Y → X (the request)

Composition of open learners is fiddly

SLIDE 11

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Open learners

Definition (Fong, Spivak & Tuy´ eras) : An open learner X → Y consists of:

A set P of parameters
A function I : P × X → Y (the implementation)
A function u : P × X × Y → P (the update)
A function r : P × X × Y → X (the request)

Composition of open learners is fiddly They form a symmetric monoidal category called Learn

who cares about monoidal bicategories

SLIDE 12

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Lenses

A lens X → Y is a function X → Y and a function X ×Y → X

SLIDE 13

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Lenses

A lens X → Y is a function X → Y and a function X ×Y → X Composition of lenses is also fiddly!

SLIDE 14

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Lenses

A lens X → Y is a function X → Y and a function X ×Y → X Composition of lenses is also fiddly! Theorem (Fong & Johnson): Open learners compose by pullback of lenses: P × Q × X P × X Q × Y X Y Z

π2

ℓ1 π2 ℓ2

SLIDE 15

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The Para construction

Let C be a monoidal category Define a category1 Para(C) by:

1who cares about monoidal bicategories

SLIDE 16

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The Para construction

Let C be a monoidal category Define a category1 Para(C) by:

Objects: objects of C
Morphisms X → Y : pair (A, f ), A object of C,

f : X ⊗ A → Y

1who cares about monoidal bicategories

SLIDE 17

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The Para construction

Let C be a monoidal category Define a category1 Para(C) by:

Objects: objects of C
Morphisms X → Y : pair (A, f ), A object of C,

f : X ⊗ A → Y

Identity on X: (I, X ⊗ I

∼ =

− → X)

1who cares about monoidal bicategories

SLIDE 18

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The Para construction

Let C be a monoidal category Define a category1 Para(C) by:

Objects: objects of C
Morphisms X → Y : pair (A, f ), A object of C,

f : X ⊗ A → Y

Identity on X: (I, X ⊗ I

∼ =

− → X)

Composition of (B, g) ◦ (A, f ):

(A ⊗ B, X ⊗ A ⊗ B

f ⊗B

− − − → Y ⊗ B

g

− → Z)

1who cares about monoidal bicategories

SLIDE 19

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The Para construction

Let C be a monoidal category Define a category1 Para(C) by:

Objects: objects of C
Morphisms X → Y : pair (A, f ), A object of C,

f : X ⊗ A → Y

Identity on X: (I, X ⊗ I

∼ =

− → X)

Composition of (B, g) ◦ (A, f ):

(A ⊗ B, X ⊗ A ⊗ B

f ⊗B

− − − → Y ⊗ B

g

− → Z) ⊗ lifts to a monoidal product on Para(C)

1who cares about monoidal bicategories

SLIDE 20

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The structure of Para(−)

A lax symmetric monoidal functor F : C → D lifts to Para(F) : Para(D) → Para(D) by F(A, f ) : F(X) ⊗ F(A)

ϕ

− → F(X ⊗ A)

F(f )

− − − → F(Y )

SLIDE 21

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The structure of Para(−)

A lax symmetric monoidal functor F : C → D lifts to Para(F) : Para(D) → Para(D) by F(A, f ) : F(X) ⊗ F(A)

ϕ

− → F(X ⊗ A)

F(f )

− − − → F(Y ) Proposition (probably): Para(−) defines a monad on [symmetric monoidal categories, lax symmetric monoidal functors]

SLIDE 22

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Backprop as Functor

Theorem (Fong, Spivak & Tuy´ eras): Fix a learning rate ε > 0 and a differentiable cost function2 C : R2 → R.

2such that every ∂ ∂y C(x, y) is invertible

SLIDE 23

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Backprop as Functor

Theorem (Fong, Spivak & Tuy´ eras): Fix a learning rate ε > 0 and a differentiable cost function2 C : R2 → R. Then there is a symmetric monoidal functor Fε,C : Para(Euc) → Learn defined by

On objects X → underlying set of X

2such that every ∂ ∂y C(x, y) is invertible

SLIDE 24

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Backprop as Functor

Theorem (Fong, Spivak & Tuy´ eras): Fix a learning rate ε > 0 and a differentiable cost function2 C : R2 → R. Then there is a symmetric monoidal functor Fε,C : Para(Euc) → Learn defined by

On objects X → underlying set of X
On morphisms f : P × X → Y :
Parameters P
Implementation I = f

2such that every ∂ ∂y C(x, y) is invertible

SLIDE 25

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Backprop as Functor

Theorem (Fong, Spivak & Tuy´ eras): Fix a learning rate ε > 0 and a differentiable cost function2 C : R2 → R. Then there is a symmetric monoidal functor Fε,C : Para(Euc) → Learn defined by

On objects X → underlying set of X
On morphisms f : P × X → Y :
Parameters P
Implementation I = f
Update U(a, x, y) = a − ε∇aE(a, x, y)
Request r(a, x, y) = (too awkward to write down)

where E(a, x, y) = dim(Y )

i=1

C(f (p, x)i, yi) is total error

2such that every ∂ ∂y C(x, y) is invertible

SLIDE 26

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Backprop as Functor

Theorem (Fong, Spivak & Tuy´ eras): Fix a learning rate ε > 0 and a differentiable cost function2 C : R2 → R. Then there is a symmetric monoidal functor Fε,C : Para(Euc) → Learn defined by

On objects X → underlying set of X
On morphisms f : P × X → Y :
Parameters P
Implementation I = f
Update U(a, x, y) = a − ε∇aE(a, x, y)
Request r(a, x, y) = (too awkward to write down)

where E(a, x, y) = dim(Y )

i=1

C(f (p, x)i, yi) is total error Update is gradient descent, and request is backpropagation

2such that every ∂ ∂y C(x, y) is invertible

SLIDE 27

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

ML doesn’t work like that

Actual backpropagation backpropagates gradients

SLIDE 28

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

ML doesn’t work like that

Actual backpropagation backpropagates gradients Request backpropagates a finite step in the gradient direction

SLIDE 29

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

ML doesn’t work like that

Actual backpropagation backpropagates gradients Request backpropagates a finite step in the gradient direction This is a hack because objects of Learn doesn’t have differentiable structure

SLIDE 30

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

ML doesn’t work like that

Actual backpropagation backpropagates gradients Request backpropagates a finite step in the gradient direction This is a hack because objects of Learn doesn’t have differentiable structure (The benefit is Learn is more general than just ML)

SLIDE 31

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Bundles

Work in a category with finite limits A bundle over X is a morphism E X

p

SLIDE 32

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Bundles

Work in a category with finite limits A bundle over X is a morphism E X

p

Examples:

1 Trivial bundle

X × Y X

π1

SLIDE 33

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Bundles

Work in a category with finite limits A bundle over X is a morphism E X

p

Examples:

1 Trivial bundle

X × Y X

π1 2 Tangent bundle over a differentiable manifold

TM M

π

SLIDE 34

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Bundles

Work in a category with finite limits A bundle over X is a morphism E X

p

Examples:

1 Trivial bundle

X × Y X

π1 2 Tangent bundle over a differentiable manifold

TM M

π 3 Cotangent bundle

T ∗M M

π∗

SLIDE 35

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Bundles over Euclidean spaces

If X = Rn is a Euclidean space then

every Tx(X) ∼

= X

SLIDE 36

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Bundles over Euclidean spaces

If X = Rn is a Euclidean space then

every Tx(X) ∼

= X

so, T(X) ∼

= X × X

SLIDE 37

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Bundles over Euclidean spaces

If X = Rn is a Euclidean space then

every Tx(X) ∼

= X

so, T(X) ∼

= X × X

so, the tangent bundle is trivial:

T(X) X

π2

SLIDE 38

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Bundles over Euclidean spaces

If X = Rn is a Euclidean space then

every Tx(X) ∼

= X

so, T(X) ∼

= X × X

so, the tangent bundle is trivial:

T(X) X

π2

Moreover:

every T ∗

x (X) ∼

= X unnaturally (since X ∗ ∼ = X)

SLIDE 39

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Bundles over Euclidean spaces

If X = Rn is a Euclidean space then

every Tx(X) ∼

= X

so, T(X) ∼

= X × X

so, the tangent bundle is trivial:

T(X) X

π2

Moreover:

every T ∗

x (X) ∼

= X unnaturally (since X ∗ ∼ = X)

so, T ∗(X) ∼

= X × X unnaturally

SLIDE 40

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Bundles over Euclidean spaces

If X = Rn is a Euclidean space then

every Tx(X) ∼

= X

so, T(X) ∼

= X × X

so, the tangent bundle is trivial:

T(X) X

π2

Moreover:

every T ∗

x (X) ∼

= X unnaturally (since X ∗ ∼ = X)

so, T ∗(X) ∼

= X × X unnaturally

elements of X × X are called dual numbers

SLIDE 41

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Bundles over Euclidean spaces

If X = Rn is a Euclidean space then

every Tx(X) ∼

= X

so, T(X) ∼

= X × X

so, the tangent bundle is trivial:

T(X) X

π2

Moreover:

every T ∗

x (X) ∼

= X unnaturally (since X ∗ ∼ = X)

so, T ∗(X) ∼

= X × X unnaturally

elements of X × X are called dual numbers
the cotangent bundle is unnaturally equivalent to a trivial

bundle

SLIDE 42

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Bundles over Euclidean spaces

If X = Rn is a Euclidean space then

every Tx(X) ∼

= X

so, T(X) ∼

= X × X

so, the tangent bundle is trivial:

T(X) X

π2

Moreover:

every T ∗

x (X) ∼

= X unnaturally (since X ∗ ∼ = X)

so, T ∗(X) ∼

= X × X unnaturally

elements of X × X are called dual numbers
the cotangent bundle is unnaturally equivalent to a trivial

bundle

Nb. Euc doesn’t have finite limits, so we work in Top

SLIDE 43

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Morphisms of bundles

A bundle morphism f : E X

p →

F Y

q is:

SLIDE 44

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Morphisms of bundles

A bundle morphism f : E X

p →

F Y

q is:

Morphisms f : X → Y and f # : X ×Y F → E

SLIDE 45

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Morphisms of bundles

A bundle morphism f : E X

p →

F Y

q is:

Morphisms f : X → Y and f # : X ×Y F → E
such that

X ×Y F F E X Y

f #

q

p f

is a pullback

SLIDE 46

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Morphisms of bundles

A bundle morphism f : E X

p →

F Y

q is:

Morphisms f : X → Y and f # : X ×Y F → E
such that

X ×Y F F E X Y

f #

q

p f

is a pullback

Equivalently: f such that X ×Y F → X factors through p

SLIDE 47

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Morphisms of bundles

A bundle morphism f : E X

p →

F Y

q is:

Morphisms f : X → Y and f # : X ×Y F → E
such that

X ×Y F F E X Y

f #

q

p f

is a pullback

Equivalently: f such that X ×Y F → X factors through p

“Every algebraic geometer knows this definition” – David Spivak

SLIDE 48

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The category of bundles

Identity morphism: X ×X E ∼ = E E E X X

p

p

SLIDE 49

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The category of bundles

Identity morphism: X ×X E ∼ = E E E X X

p

p

Composition of morphisms: X ×Z G Y ×Z G G X ×Y F F E X Y Z

f ∗(g#)

g#
r

f #

q

p f g

SLIDE 50

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Where does this come from?

From the Grothendieck construction: Bund(C) =

X∈C

(C/X)op

SLIDE 51

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Where does this come from?

From the Grothendieck construction: Bund(C) =

X∈C

(C/X)op This buys us (conjecture) a monoidal structure: E X

p ⊗

F Y

q =

E × F X × Y

p×q

(this might not be the right one!)

SLIDE 52

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Lenses

A (bimorphic) lens λ : (S, T) → (A, B) consists of:

a morphism λv : S → A called view
a morphism λu : S × B → T called update

SLIDE 53

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Lenses

A (bimorphic) lens λ : (S, T) → (A, B) consists of:

a morphism λv : S → A called view
a morphism λu : S × B → T called update

Composition of lenses is fiddly

SLIDE 54

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Lenses

A (bimorphic) lens λ : (S, T) → (A, B) consists of:

a morphism λv : S → A called view
a morphism λu : S × B → T called update

Composition of lenses is fiddly Where does this come from? The Grothendieck construction: Lens(C) =

X∈C

coKl(X × −)op

SLIDE 55

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Lenses

A (bimorphic) lens λ : (S, T) → (A, B) consists of:

a morphism λv : S → A called view
a morphism λu : S × B → T called update

Composition of lenses is fiddly Where does this come from? The Grothendieck construction: Lens(C) =

X∈C

coKl(X × −)op Theorem (Lambek): coKl(X × −) ∼ = C[x], where C[x] is the polynomial category formed by freely adjoining x : 1 → X and closing under finite products

SLIDE 56

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Lenses are bundle morphisms

Another theorem (Lambek): coEM(X × −) ∼ = C/X

SLIDE 57

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Lenses are bundle morphisms

Another theorem (Lambek): coEM(X × −) ∼ = C/X So there is a canonoical embedding C[x] ֒ → C/X

SLIDE 58

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Lenses are bundle morphisms

Another theorem (Lambek): coEM(X × −) ∼ = C/X So there is a canonoical embedding C[x] ֒ → C/X Grothendieck them all together: Lens(C) → Bund(C) It takes a lens λ : (S, T) → (A, B) to the bundle morphism S × B A × B S × T S A

π1,λu

π2

π2 λv

SLIDE 59

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Lenses are bundle morphisms

Another theorem (Lambek): coEM(X × −) ∼ = C/X So there is a canonoical embedding C[x] ֒ → C/X Grothendieck them all together: Lens(C) → Bund(C) It takes a lens λ : (S, T) → (A, B) to the bundle morphism S × B A × B S × T S A

π1,λu

π2

π2 λv

SLIDE 60

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Morphisms of contangent bundles

There is a functor Cot(−) : DiffMfd → Bund(Top)

SLIDE 61

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Morphisms of contangent bundles

There is a functor Cot(−) : DiffMfd → Bund(Top) It takes f : X → Y to X ×Y T ∗(Y ) T ∗(Y ) T ∗(X) X Y

f ′

π∗

π∗ f

where f ′ : (x, c) → (x, c ◦ Jx(f ))

SLIDE 62

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Morphisms of contangent bundles

There is a functor Cot(−) : DiffMfd → Bund(Top) It takes f : X → Y to X ×Y T ∗(Y ) T ∗(Y ) T ∗(X) X Y

f ′

π∗

π∗ f

where f ′ : (x, c) → (x, c ◦ Jx(f )) Jx(f ) is the Jacobian (matrix of partial derivatives) of f at x

SLIDE 63

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The chain rule

Functorality of Cot(−): X ×Z T ∗(Z) Y ×Z T ∗(Z) T ∗(Z) X ×Y T ∗(Y ) T ∗(Y ) T ∗(X) X Y Z

f ∗(g′)

g′
π∗

f ′

π∗

π∗ f g

SLIDE 64

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The chain rule

Functorality of Cot(−): X ×Z T ∗(Z) Y ×Z T ∗(Z) T ∗(Z) X ×Y T ∗(Y ) T ∗(Y ) T ∗(X) X Y Z

f ∗(g′)

g′
π∗

f ′

π∗

π∗ f g

(g ◦ f )′ = f ′ ◦ f ∗(g′) is the chain rule in differential geometry

SLIDE 65

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

From Para(Bund(Top)) to Learn

Consider a morphism of Para(Bund(Top)) in the image of Para(Cot) : Para(Euc) → Para(Bund(Top)) It looks like (X × A) ×Y T ∗(Y ) Y T ∗(X × A) X × A Y

f ′

π∗

π∗ f

SLIDE 66

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

From Para(Bund(Top)) to Learn

Consider a morphism of Para(Bund(Top)) in the image of Para(Cot) : Para(Euc) → Para(Bund(Top)) It looks like (X × A) ×Y T ∗(Y ) Y T ∗(X × A) X × A Y

f ′

π∗

π∗ f

We’re going to turn it into an open learner, given ε > 0 and differentiable C : R2 → R

SLIDE 67

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The setup

Obviously, parameters are A and implementation is f

SLIDE 68

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The setup

Obviously, parameters are A and implementation is f We need to define U, r : A × X × Y → A × X so, fix a ∈ A, x ∈ X and y ∈ Y and fix the total error Cy(y′) = dim(Y )

i=1

C(yi, y′

i )

SLIDE 69

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The setup

Obviously, parameters are A and implementation is f We need to define U, r : A × X × Y → A × X so, fix a ∈ A, x ∈ X and y ∈ Y and fix the total error Cy(y′) = dim(Y )

i=1

C(yi, y′

i )

Consider the diagram. . .

SLIDE 70

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The brain exploding part

R ∼ = T ∗

Cy(f (x,a))(R)

(X × A) ×R T ∗(R) Y ×R T ∗(R) T ∗(R) T ∗

f (x,a)(Y )

(X × A) ×Y T ∗(Y ) T ∗(Y ) T ∗

(x,a)(X × A)

T ∗(X × A) 1 ∼ = T ∗(1) 1 X × A Y R

f ∗(C ′

y)

C ′

y

π∗
f ′
π∗

!

π∗

(x,a) f Cy

SLIDE 71

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The part we don’t understand

Now: Chase 1 ∈ R to T ∗(X × A) and then apply µε : T ∗(X × A) → X × A The result is r, U (a, x, y)

SLIDE 72

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The part we don’t understand

Now: Chase 1 ∈ R to T ∗(X × A) and then apply µε : T ∗(X × A) → X × A The result is r, U (a, x, y) µε takes a finite step in the gradient direction: µε((x, a), (v, w)) = (x + v, a + εw)

SLIDE 73

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The part we don’t understand

Now: Chase 1 ∈ R to T ∗(X × A) and then apply µε : T ∗(X × A) → X × A The result is r, U (a, x, y) µε takes a finite step in the gradient direction: µε((x, a), (v, w)) = (x + v, a + εw) What is µε? We couldn’t find any nice properties

SLIDE 74

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The part we don’t understand

Now: Chase 1 ∈ R to T ∗(X × A) and then apply µε : T ∗(X × A) → X × A The result is r, U (a, x, y) µε takes a finite step in the gradient direction: µε((x, a), (v, w)) = (x + v, a + εw) What is µε? We couldn’t find any nice properties It looks a bit like a thing called an exponential map

SLIDE 75

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The catch

Conjecture: This defines a symmetric monoidal functor Para(Bund(Top)) ⊇ Im(Para(Cot)) → Learn

SLIDE 76

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The catch

Conjecture: This defines a symmetric monoidal functor Para(Bund(Top)) ⊇ Im(Para(Cot)) → Learn Another conjecture: This commutes: Para(Euc) Im(Para(Cot)) Learn

Para(Cot)

SLIDE 77

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The catch

Conjecture: This defines a symmetric monoidal functor Para(Bund(Top)) ⊇ Im(Para(Cot)) → Learn Another conjecture: This commutes: Para(Euc) Im(Para(Cot)) Learn

Para(Cot)

The catch: We think Para(Cot) is an equivalence of categories

nto its image

SLIDE 78

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

The catch

Conjecture: This defines a symmetric monoidal functor Para(Bund(Top)) ⊇ Im(Para(Cot)) → Learn Another conjecture: This commutes: Para(Euc) Im(Para(Cot)) Learn

Para(Cot)

The catch: We think Para(Cot) is an equivalence of categories

nto its image

So, we’ve just rewritten Backprop as Functor in a different way!

SLIDE 79

Bundles, Lenses & Machine Learning Motivation Backprop as Functor Bundles Putting it together

Even more hard questions

What happens if we extend the functor to the whole of Para(Bund(Top))? We have no idea! Optimistic hope: This allows defining general “ML-like” systems, not necessarily involving gradients (eg. “discrete ML”

n Bayesian networks)