Differen'able Func'onal Programming Noel Welsh @noelwelsh - - PowerPoint PPT Presentation

Differen'able Func'onal Programming

Noel Welsh @noelwelsh

underscore

Goals

Deep learning isn’t deep

Deep learning is func'onal programming

You can implement it yourself

Deep Learning

Deep learning is supervised learning of parameterised func'ons by gradient descent

Supervised Learning via Gradient Descent

Parametrised func'ons
 Supervised learning
 Gradient descent

Parameterised func'ons

x → sin(x)

θ → x → sin(x/θ)

Supervised learning

Choose parameters to fit data

Gradient Descent

Algorithmically choose parameters

Calculate the deriva've of the loss func)on with respect to the parameters

n

X

i=1

(datay

i − sin(datax i /θ))2

n

X

i=1

2(datay

i − sin(datax i /θ)) cos(datax i /θ)datax i

θ2

Calculate gradient for current parameters

Change parameters by a small amount in downhill direc'on

Deep Learning

Deep learning is supervised learning of parameterised func'ons by gradient descent

Choice of func'ons that is expressive

Choice of func'ons that is suited to GPU accelera'on

Tensor mul'plica'on and non-linearity

Tensors: mul'dimensional arrays

Generalisa'on of vectors and matrices

× =

Vector[3] Matrix[3,5] Vector[5]

Matrix “weights” are parameters

Non-linearity: just about any non-linear func'on

ReLU(x) = max(0, x)

f(x) = ReLU(x × Weights)

Deep = composing many func'ons

z(x) = k(j(i(h(g(f(x)))))

Differen'able Programming

SoQware that automa'cally calculates deriva'ves and compiles to GPU code

Increasing complexity

• f deep learning models

Condi'onals and loops

“Differen'able programming”

Automa'c Differen'a'on

Calcula'ng gradients

Deriva'ves are composi'onal

Algorithms for calcula'ng gradients

Forward-mode AD

Reverse-mode AD

Composi'on of Deriva'ves

Func'ons compose

g f = g(f(x))

Deriva'ves compose

(g f)0 = (g0 f)f 0

Calcula'ng Gradients

Mathema'cian’s approach

Symbolic differen'a'on

(af)0 = af 0 (f + g)0 = f 0 + g0 (f(x)g(x))0 = f 0(x)g(x) + f(x)g0(x) (g f)0 = (g0 f)f 0 (xn)0 = nxn1 (eax)0 = aeax (ln(x))0 = 1 x (sin(x))0 = cos(x) (cos(x))0 = sin(x) (tan(x))0 = 1 + tan2(x)

Exact solu'on

No loops or condi'onals

Size of deriva've grows exponen'ally

Programmer’s approach

f(x + ✏) − f(x) ✏

Any func'on

• Inexact. Choosing ε is

hard

Automa'c differen'a'on approach

Two main algorithms: forward and reverse

Exact solu'on

Any func'on

Forward-mode AD

Calculate with dual numbers

a + b✏ ✏2 = 0

final case class Dual(v: Double, d: Double)

Add rules for basic

• pera'ons and func'ons

(a + b✏) + (c + d✏) = (a + c) + (b + d)✏

final case class Dual(v: Double, d: Double) { def +(that: Dual): Dual = Dual(this.v + that.v, this.d + that.d) }

(a + b✏)(c + d✏) = ac + bc✏ + ad✏ + bd✏2 = ac + (bc + ad)✏

final case class Dual(v: Double, d: Double) { def +(that: Dual): Dual = Dual(this.v + that.v, this.d + that.d) 
 def *(that: Dual): Dual = Dual(this.v * that.v, 
 (this.v * that.d) + (this.d * that.v)) }

f(a + b✏) = f(a) + f 0(a)b✏

final case class Dual(v: Double, d: Double) { def +(that: Dual): Dual = Dual(this.v + that.v, this.d + that.d) 
 def *(that: Dual): Dual = Dual(this.v * that.v, 
 (this.v * that.d) + (this.d * that.v)) def sin: Dual = Dual(Math.sin(this.v), Math.cos(this.v) * this.d) }

f(x) = sin2(x)

x

f(x) = sin2(x)

sin sin ×

x

f(x) = sin2(x)

sin sin ×

Dual(1.0, 1)

x

f(x) = sin2(x)

sin sin ×

Dual(0.84, 0.54) Dual(0.84, 0.54)

x

f(x) = sin2(x)

sin sin ×

Dual(0.71, 0.91)

val f = (x: Dual) => x.sin * x.sin f(Dual(1.0, 1)) // res: Dual = Dual(0.7080734182735712,0.9092974268256818)

Manipula'on of the chain rule

f(a + b✏) = f(a) + f 0(a)b✏

f(a + b✏) = f(a) + f 0(a)b✏

a → f(a) + f 0(a)

a → f(a)

a ??? ⇣ a → f(a) ⌘ = f(a) + f 0(a)b✏

a ??? ⇣ a → f(a) ⌘ = c

a flatMap ⇣ a → f(a) ⌘ = c

There is a monad for Dual

This expresses the chain rule

final case class Dual[A](v: A, d: Double) { def flatMap[B](f: A => Dual[B]): Dual[B] = { val next = f(v) val nextD = this.d * next.d Dual(next.v, nextD) } }

final case class Dual[A](v: A, d: Double) { def flatMap[B](f: A => Dual[B]): Dual[B] = { val next = f(v) val nextD = this.d * next.d Dual(next.v, nextD) } def sin: Dual = this.flatMap(a => Dual(Math.sin(a), Math.cos(a))) }

For deep learning, generalise to tensors

Forward-mode scales in the size of the input dimension

Reverse-mode AD

Chain rule doesn’t care about order

(g f)0 = (g0 f)f 0

(g f)0(x) = g0(f(x))f 0(x)

(g f)0(x) = g0(f(x))f 0(x)

(g f)0(x) = g0(f(x))f 0(x)

x

f(x) = sin2(x)

sin sin ×

x

f(x) = sin2(x)

sin sin ×

1.0

x

f(x) = sin2(x)

sin sin ×

0.84 0.84 1.0

x

f(x) = sin2(x)

sin sin ×

0.84 0.84 1.0 0.71

x

f(x) = sin2(x)

sin sin ×

0.84 0.84 1.0 0.71 1

x

f(x) = sin2(x)

sin sin ×

0.84 0.84 1.0 0.71 0.54 0.54

x

f(x) = sin2(x)

sin sin ×

0.84 0.84 1.0 0.71 0.91

Use a monad (or con'nua'ons)

flatMap is the chain rule

final case class AD[A]
 (v: A, k: Double => Double) { // Chain rule def flatMap[B](f: A => AD[B]): AD[B] = { val self = this val next = f(v) AD(next.v, 
 (d: Double) => next.k(d) * self.k(d)) } }

• bject AD {

def pure(x: Double): AD[Double] = AD(x, d => 1.0) }

final case class AD[A](v: A, k: Double => Double) { // Chain rule def flatMap[B](f: A => AD[B]): AD[B] = { val self = this val next = f(v) AD(next.v, (d: Double) => next.k(d) * self.k(d)) } def +(that: AD[Double])(implicit ev: A =:= Double): AD[Double] = AD(this.v + that.v, (d: Double) => this.k(d) + that.k(d)) def *(that: AD[Double])(implicit ev: A =:= Double): AD[Double] = AD(this.v * that.v, (d: Double) => (this.k(d) * that.v) + (that.k(d) * this.v)) def sin(implicit ev: A =:= Double): AD[Double] = this.flatMap(x => AD(Math.sin(x), (d) => Math.cos(x) * d)) def gradient: Double = this.k(1.0) }

• bject AD {

def pure(x: Double): AD[Double] = AD(x, d => 1.0) }

val f = (x: AD[Double]) => 
 x.sin * x.sin f(AD.pure(1.0)).gradient // res: Double = 0.9092974268256818

Reverse-mode scales in the size of the output dimension

|input| |output|

h[ps:/ /github.com/noelwelsh/ fdl

Differen'able Programming

Differen'able programming = automa'c differen'a'on + composi'onal code

Treat differen'able programming as programming

Ergonomics

Correctness

Performance

Ergonomics

val f = x => sin(x) * sin(x)

# Tensorflow (Python) X = tf.placeholder(tf.float32)
 Y = tf.sin(X) Z = tf.multiply(Y, Y)

Solu'on: be[er DSLs

Correctness

Tensor dimensions must agree

× =

Vector[3] Matrix[3,5] Vector[5]

Axis have meaning

// What is axis 1?

tf.reduce_sum(tensor, 1)

Solu'on: expressive type systems

h[ps:/ /github.com/ ctongfei/nexus

Linear logic and differen'al λ-calculus

Performance

Need compila'on (to GPU) for performance

Now: embedded DSLs

Future: languages (with staging)

h[ps:/ /github.com/ feiwang3311/Lantern.git

Conclusions

Deep learning is about func'on composi'on and deriva'ves

FP is about func'on composi'on

And we can easily express deriva'ves in FP

Scala is well posi'oned for the next genera'on of DL systems

But it’s a lot of engineering work…

Thank You! github.com/noelwelsh/fdl

Noel Welsh @noelwelsh

underscore