Bundles, Lenses & Machine Learning Motivation Backprop as - PowerPoint PPT Presentation

Bundles, Lenses & Machine Learning Bundles, Lenses & Machine Learning Motivation Backprop as Functor Jules Hedges 1 Bundles Putting it joint work with together Brendan Fong 2 Eliana Lorch 3 David Spivak 2 1 Max Planck Institute for Mathematics in the Sciences 2 MIT 3 University of Oxford SYCO 5, Birmingham

Bundles, Lenses & Machine Learning Bundles, Lenses & Machine Learning Motivation Backprop as Functor Jules Hedges 1 Bundles Putting it joint work with together Brendan Fong 2 Eliana Lorch 3 David Spivak 2 1 Max Planck Institute for Mathematics in the Sciences 2 MIT 3 University of Oxford SYCO 5, Birmingham Featuring zero string diagrams :(

Bundles, Lenses & Motivation Machine Learning Motivation Backprop as Machine learning is categorical in 2 different ways: Functor Bundles Backprop As Functor Putting it together (compositional description of ML with monoidal categories) + ML as differential geometry

Bundles, Lenses & Motivation Machine Learning Motivation Backprop as Machine learning is categorical in 2 different ways: Functor Bundles Backprop As Functor Putting it together (compositional description of ML with monoidal categories) + ML as differential geometry In this talk: smoosh them together (why? Why not)

Bundles, Lenses & Motivation Machine Learning Motivation Backprop as Machine learning is categorical in 2 different ways: Functor Bundles Backprop As Functor Putting it together (compositional description of ML with monoidal categories) + ML as differential geometry In this talk: smoosh them together (why? Why not) It clarifies Backprop as Functor more than anything else

Bundles, Lenses & Open learners Machine Learning Motivation Definition (Fong, Spivak & Tuy´ eras) : An open learner X → Y Backprop as Functor consists of: Bundles • A set P of parameters Putting it together

Bundles, Lenses & Open learners Machine Learning Motivation Definition (Fong, Spivak & Tuy´ eras) : An open learner X → Y Backprop as Functor consists of: Bundles • A set P of parameters Putting it together • A function I : P × X → Y (the implementation)

Bundles, Lenses & Open learners Machine Learning Motivation Definition (Fong, Spivak & Tuy´ eras) : An open learner X → Y Backprop as Functor consists of: Bundles • A set P of parameters Putting it together • A function I : P × X → Y (the implementation) • A function u : P × X × Y → P (the update)

Bundles, Lenses & Open learners Machine Learning Motivation Definition (Fong, Spivak & Tuy´ eras) : An open learner X → Y Backprop as Functor consists of: Bundles • A set P of parameters Putting it together • A function I : P × X → Y (the implementation) • A function u : P × X × Y → P (the update) • A function r : P × X × Y → X (the request)

Bundles, Lenses & Open learners Machine Learning Motivation Definition (Fong, Spivak & Tuy´ eras) : An open learner X → Y Backprop as Functor consists of: Bundles • A set P of parameters Putting it together • A function I : P × X → Y (the implementation) • A function u : P × X × Y → P (the update) • A function r : P × X × Y → X (the request) Composition of open learners is fiddly

Bundles, Lenses & Open learners Machine Learning Motivation Definition (Fong, Spivak & Tuy´ eras) : An open learner X → Y Backprop as Functor consists of: Bundles • A set P of parameters Putting it together • A function I : P × X → Y (the implementation) • A function u : P × X × Y → P (the update) • A function r : P × X × Y → X (the request) Composition of open learners is fiddly They form a symmetric monoidal category called Learn who cares about monoidal bicategories

Bundles, Lenses & Lenses Machine Learning A lens X → Y is a function X → Y and a function X × Y → X Motivation Backprop as Functor Bundles Putting it together

Bundles, Lenses & Lenses Machine Learning A lens X → Y is a function X → Y and a function X × Y → X Motivation Backprop as Functor Composition of lenses is also fiddly! Bundles Putting it together

Bundles, Lenses & Lenses Machine Learning A lens X → Y is a function X → Y and a function X × Y → X Motivation Backprop as Functor Composition of lenses is also fiddly! Bundles Putting it together Theorem (Fong & Johnson): Open learners compose by pullback of lenses: P × Q × X � P × X Q × Y π 2 ℓ 2 ℓ 1 π 2 X Y Z

Bundles, Lenses & The Para construction Machine Learning Let C be a monoidal category Motivation Backprop as Functor Define a category 1 Para ( C ) by: Bundles Putting it together 1 who cares about monoidal bicategories

Bundles, Lenses & The Para construction Machine Learning Let C be a monoidal category Motivation Backprop as Functor Define a category 1 Para ( C ) by: Bundles • Objects: objects of C Putting it together • Morphisms X → Y : pair ( A , f ), A object of C , f : X ⊗ A → Y 1 who cares about monoidal bicategories

Bundles, Lenses & The Para construction Machine Learning Let C be a monoidal category Motivation Backprop as Functor Define a category 1 Para ( C ) by: Bundles • Objects: objects of C Putting it together • Morphisms X → Y : pair ( A , f ), A object of C , f : X ⊗ A → Y ∼ = • Identity on X : ( I , X ⊗ I − → X ) 1 who cares about monoidal bicategories

Bundles, Lenses & The Para construction Machine Learning Let C be a monoidal category Motivation Backprop as Functor Define a category 1 Para ( C ) by: Bundles • Objects: objects of C Putting it together • Morphisms X → Y : pair ( A , f ), A object of C , f : X ⊗ A → Y ∼ = • Identity on X : ( I , X ⊗ I − → X ) • Composition of ( B , g ) ◦ ( A , f ): g f ⊗ B ( A ⊗ B , X ⊗ A ⊗ B − − − → Y ⊗ B − → Z ) 1 who cares about monoidal bicategories

Bundles, Lenses & The Para construction Machine Learning Let C be a monoidal category Motivation Backprop as Functor Define a category 1 Para ( C ) by: Bundles • Objects: objects of C Putting it together • Morphisms X → Y : pair ( A , f ), A object of C , f : X ⊗ A → Y ∼ = • Identity on X : ( I , X ⊗ I − → X ) • Composition of ( B , g ) ◦ ( A , f ): g f ⊗ B ( A ⊗ B , X ⊗ A ⊗ B − − − → Y ⊗ B − → Z ) ⊗ lifts to a monoidal product on Para ( C ) 1 who cares about monoidal bicategories

Bundles, Lenses & The structure of Para ( − ) Machine Learning Motivation Backprop as A lax symmetric monoidal functor F : C → D lifts to Functor Bundles Para ( F ) : Para ( D ) → Para ( D ) Putting it together by F ( f ) ϕ F ( A , f ) : F ( X ) ⊗ F ( A ) − → F ( X ⊗ A ) − − − → F ( Y )

Bundles, Lenses & The structure of Para ( − ) Machine Learning Motivation Backprop as A lax symmetric monoidal functor F : C → D lifts to Functor Bundles Para ( F ) : Para ( D ) → Para ( D ) Putting it together by F ( f ) ϕ F ( A , f ) : F ( X ) ⊗ F ( A ) − → F ( X ⊗ A ) − − − → F ( Y ) Proposition (probably): Para ( − ) defines a monad on [symmetric monoidal categories, lax symmetric monoidal functors]

Bundles, Lenses & Backprop as Functor Machine Learning Theorem (Fong, Spivak & Tuy´ eras): Fix a learning rate ε > 0 Motivation and a differentiable cost function 2 C : R 2 → R . Backprop as Functor Bundles Putting it together 2 such that every ∂ ∂ y C ( x , y ) is invertible

Bundles, Lenses & Backprop as Functor Machine Learning Theorem (Fong, Spivak & Tuy´ eras): Fix a learning rate ε > 0 Motivation and a differentiable cost function 2 C : R 2 → R . Backprop as Functor Bundles Then there is a symmetric monoidal functor Putting it F ε, C : Para ( Euc ) → Learn defined by together • On objects X �→ underlying set of X 2 such that every ∂ ∂ y C ( x , y ) is invertible

Bundles, Lenses & Backprop as Functor Machine Learning Theorem (Fong, Spivak & Tuy´ eras): Fix a learning rate ε > 0 Motivation and a differentiable cost function 2 C : R 2 → R . Backprop as Functor Bundles Then there is a symmetric monoidal functor Putting it F ε, C : Para ( Euc ) → Learn defined by together • On objects X �→ underlying set of X • On morphisms f : P × X → Y : • Parameters P • Implementation I = f 2 such that every ∂ ∂ y C ( x , y ) is invertible

Bundles, Lenses & Backprop as Functor Machine Learning Theorem (Fong, Spivak & Tuy´ eras): Fix a learning rate ε > 0 Motivation and a differentiable cost function 2 C : R 2 → R . Backprop as Functor Bundles Then there is a symmetric monoidal functor Putting it F ε, C : Para ( Euc ) → Learn defined by together • On objects X �→ underlying set of X • On morphisms f : P × X → Y : • Parameters P • Implementation I = f • Update U ( a , x , y ) = a − ε ∇ a E ( a , x , y ) • Request r ( a , x , y ) = (too awkward to write down) where E ( a , x , y ) = � dim ( Y ) C ( f ( p , x ) i , y i ) is total error i =1 2 such that every ∂ ∂ y C ( x , y ) is invertible