Connection Management: FP Style! Jed Wesley-Smith @jedws Jed - - PowerPoint PPT Presentation
Connection Management: FP Style! Jed Wesley-Smith @jedws Jed - - PowerPoint PPT Presentation
Connection Management: FP Style! Jed Wesley-Smith @jedws Jed Wesley-Smith @jedws Jed Wesley-Smith @jedws Jed Wesley-Smith @jedws our problem our problem statistical analysis our problem statistical analysis heavy lifting in
Jed Wesley-Smith @jedws
Jed Wesley-Smith @jedws
Jed Wesley-Smith @jedws
Jed Wesley-Smith @jedws
- ur problem
- ur problem
- statistical analysis
- ur problem
- statistical analysis
- heavy lifting in R
- ur problem
- statistical analysis
- heavy lifting in R
- manage external process
- ur problem
- statistical analysis
- heavy lifting in R
- manage external process
- connect to external process
- ur problem
- statistical analysis
- heavy lifting in R
- manage external process
- connect to external process
- connection is stateful
first pass: imperative
first pass: imperative
trait Connection {
first pass: imperative
trait Connection { def eval(expr: String): REXP
first pass: imperative
trait Connection { def eval(expr: String): REXP def close() }
first pass: imperative
trait Connection { def eval(expr: String): REXP def close() } trait Server { // AKA ConnectionFactory def connection: Connection }
first pass: imperative
first pass: imperative
- manual
- Server management
- Connection management
- Connection closing (in finally)
first pass: imperative
- manual
- Server management
- Connection management
- Connection closing (in finally)
- hard to implement pooling
second pass: loaner
second pass: loaner
trait Connection { def eval(expr: String): REXP }
second pass: loaner
trait Connection { def eval(expr: String): REXP } trait Server { def withConnection[A](f: Connection => A): A }
second pass: loaner
second pass: loaner
- better
- don’t manage Connection
- don’t need to close Connection
second pass: loaner
- better
- don’t manage Connection
- don’t need to close Connection
- manual
- Server management
- composition
second pass: loaner
- better
- don’t manage Connection
- don’t need to close Connection
- manual
- Server management
- composition
- hard to implement pooling
third pass, implicits
third pass, implicits
trait Exec[A] {
third pass, implicits
trait Exec[A] { def map[B](f: A => B)(implicit c: Conn): Exec[B]
third pass, implicits
trait Exec[A] { def map[B](f: A => B)(implicit c: Conn): Exec[B] def flatMap[B](f: A => Exec[B])(implicit c: Conn) : Exec[B] }
third pass, implicits
third pass, implicits
- it’s Scala, implicits are the thing
third pass, implicits
- it’s Scala, implicits are the thing
- requires
- client code to manage the instance
- often done poorly
recap: requirements
recap: requirements
- execute code with an externally managed and
provided Connection (a program)
recap: requirements
- execute code with an externally managed and
provided Connection (a program)
- compose small programs into larger programs
recap: requirements
- execute code with an externally managed and
provided Connection (a program)
- compose small programs into larger programs
- programs should not manage object lifecycles
recap: requirements
- execute code with an externally managed and
provided Connection (a program)
- compose small programs into larger programs
- programs should not manage object lifecycles
- clients should not manage object lifecycles
recap: requirements
- execute code with an externally managed and
provided Connection (a program)
- compose small programs into larger programs
- programs should not manage object lifecycles
- clients should not manage object lifecycles
- provide central management
so, let’s build some kit!
so, let’s build some kit!
- we’ll invest some time creating some library that
will suit our needs
so, let’s build some kit!
- we’ll invest some time creating some library that
will suit our needs
- what would a functional programmer do?
- f course it is!
- f course it is!
what we want is a Reader monad
recap: Reader monad
recap: Reader monad
class Reader[C, A](f: C => A) extends (C => A) { def apply(c: C) = f(c) }
recap: Reader monad
class Reader[C, A](f: C => A) extends (C => A) { def apply(c: C) = f(c) } class MonadReader[C, A](read: Reader[C, A]) extends Monad[Reader[C, _]] { // <- not really Scala
recap: Reader monad
class Reader[C, A](f: C => A) extends (C => A) { def apply(c: C) = f(c) } class MonadReader[C, A](read: Reader[C, A]) extends Monad[Reader[C, _]] { // <- not really Scala def point(a: A) =
recap: Reader monad
class Reader[C, A](f: C => A) extends (C => A) { def apply(c: C) = f(c) } class MonadReader[C, A](read: Reader[C, A]) extends Monad[Reader[C, _]] { // <- not really Scala def point(a: A) = new Reader[C, A](_ => a)
recap: Reader monad
class Reader[C, A](f: C => A) extends (C => A) { def apply(c: C) = f(c) } class MonadReader[C, A](read: Reader[C, A]) extends Monad[Reader[C, _]] { // <- not really Scala def point(a: A) = new Reader[C, A](_ => a) def map[B](f: A => B): Reader[C, B] =
recap: Reader monad
class Reader[C, A](f: C => A) extends (C => A) { def apply(c: C) = f(c) } class MonadReader[C, A](read: Reader[C, A]) extends Monad[Reader[C, _]] { // <- not really Scala def point(a: A) = new Reader[C, A](_ => a) def map[B](f: A => B): Reader[C, B] = new Reader[C, B](read andThen f)
recap: Reader monad
class Reader[C, A](f: C => A) extends (C => A) { def apply(c: C) = f(c) } class MonadReader[C, A](read: Reader[C, A]) extends Monad[Reader[C, _]] { // <- not really Scala def point(a: A) = new Reader[C, A](_ => a) def map[B](f: A => B): Reader[C, B] = new Reader[C, B](read andThen f) def flatMap[B](f: A => Reader[C, B]): Reader[C, B] =
recap: Reader monad
class Reader[C, A](f: C => A) extends (C => A) { def apply(c: C) = f(c) } class MonadReader[C, A](read: Reader[C, A]) extends Monad[Reader[C, _]] { // <- not really Scala def point(a: A) = new Reader[C, A](_ => a) def map[B](f: A => B): Reader[C, B] = new Reader[C, B](read andThen f) def flatMap[B](f: A => Reader[C, B]): Reader[C, B] = new Reader[C, B](c => f(read(c))(c)) }
recap: Reader monad
recap: Reader monad
class Reader[C, A](f: C => A) extends (C => A) { def apply(c: C) = f(c) } class MonadReader[C, A](read: Reader[C, A]) extends Monad[Reader[C, _]] { // <- not really Scala def flatMap[B](f: A => Reader[C, B]): Reader[C, B] =
recap: Reader monad
class Reader[C, A](f: C => A) extends (C => A) { def apply(c: C) = f(c) } class MonadReader[C, A](read: Reader[C, A]) extends Monad[Reader[C, _]] { // <- not really Scala def flatMap[B](f: A => Reader[C, B]): Reader[C, B] = new Reader[C, B]({ c: C =>
recap: Reader monad
class Reader[C, A](f: C => A) extends (C => A) { def apply(c: C) = f(c) } class MonadReader[C, A](read: Reader[C, A]) extends Monad[Reader[C, _]] { // <- not really Scala def flatMap[B](f: A => Reader[C, B]): Reader[C, B] = new Reader[C, B]({ c: C => val a: A = read(c)
recap: Reader monad
class Reader[C, A](f: C => A) extends (C => A) { def apply(c: C) = f(c) } class MonadReader[C, A](read: Reader[C, A]) extends Monad[Reader[C, _]] { // <- not really Scala def flatMap[B](f: A => Reader[C, B]): Reader[C, B] = new Reader[C, B]({ c: C => val a: A = read(c) val readB: Reader[C, B] = f(a)
recap: Reader monad
class Reader[C, A](f: C => A) extends (C => A) { def apply(c: C) = f(c) } class MonadReader[C, A](read: Reader[C, A]) extends Monad[Reader[C, _]] { // <- not really Scala def flatMap[B](f: A => Reader[C, B]): Reader[C, B] = new Reader[C, B]({ c: C => val a: A = read(c) val readB: Reader[C, B] = f(a) val b: B = readB(c)
recap: Reader monad
class Reader[C, A](f: C => A) extends (C => A) { def apply(c: C) = f(c) } class MonadReader[C, A](read: Reader[C, A]) extends Monad[Reader[C, _]] { // <- not really Scala def flatMap[B](f: A => Reader[C, B]): Reader[C, B] = new Reader[C, B]({ c: C => val a: A = read(c) val readB: Reader[C, B] = f(a) val b: B = readB(c) b }) }
recap: Reader monad
recap: Reader monad
- the Monad on function application
- f an environment
recap: Reader monad
- the Monad on function application
- f an environment
- sub-computations may be fed
an altered environment
connections
connections
- ur environment is a Connection:
connections
- ur environment is a Connection:
type Program[+A] = Reader[Connection, A]
programs
programs
/** Apply the native function to the given arguments */ def native(name: String, args: Arg*): Program[REXP] = …
programs
/** Apply the native function to the given arguments */ def native(name: String, args: Arg*): Program[REXP] = … def mean(x: Seq[Double]): Program[Double] = native("mean", x) flatMap toDouble
programs
/** Apply the native function to the given arguments */ def native(name: String, args: Arg*): Program[REXP] = … def mean(x: Seq[Double]): Program[Double] = native("mean", x) flatMap toDouble def stdDev(x: Seq[Double]): Program[Double] = if (x.length < 2) const(0D) // < 2 datas will NaN else native("sd", x) flatMap toDouble
programs
/** Apply the native function to the given arguments */ def native(name: String, args: Arg*): Program[REXP] = … def mean(x: Seq[Double]): Program[Double] = native("mean", x) flatMap toDouble def stdDev(x: Seq[Double]): Program[Double] = if (x.length < 2) const(0D) // < 2 datas will NaN else native("sd", x) flatMap toDouble def meanAndStdDev(values: Seq[Double]) : Program[(Double, Double, Int)] =
programs
/** Apply the native function to the given arguments */ def native(name: String, args: Arg*): Program[REXP] = … def mean(x: Seq[Double]): Program[Double] = native("mean", x) flatMap toDouble def stdDev(x: Seq[Double]): Program[Double] = if (x.length < 2) const(0D) // < 2 datas will NaN else native("sd", x) flatMap toDouble def meanAndStdDev(values: Seq[Double]) : Program[(Double, Double, Int)] = for { m <- mean(values) sd <- stdDev(values) } yield (m, sd, values.length)
now what?
now what?
- getting a value of type Program[A] is not the same
as getting a value of type A, a program still needs to be executed.
now what?
- getting a value of type Program[A] is not the same
as getting a value of type A, a program still needs to be executed.
- how do we execute a program?
now what?
- getting a value of type Program[A] is not the same
as getting a value of type A, a program still needs to be executed.
- how do we execute a program?
- first we need to compile it
compile
compile
def compile[A](program: Program[A]): IO[A] =
compile
def compile[A](program: Program[A]): IO[A] = IO {
compile
def compile[A](program: Program[A]): IO[A] = IO { for { server <- Server.start()
compile
def compile[A](program: Program[A]): IO[A] = IO { for { server <- Server.start() result <- try server execute program
compile
def compile[A](program: Program[A]): IO[A] = IO { for { server <- Server.start() result <- try server execute program finally server.stop()
compile
def compile[A](program: Program[A]): IO[A] = IO { for { server <- Server.start() result <- try server execute program finally server.stop() } yield result }
compile
def compile[A](program: Program[A]): IO[A] = IO { for { server <- Server.start() result <- try server execute program finally server.stop() } yield result } def main(args: Array[String]) { batchedReduction.run.unsafePerformIO }
pooled
pooled
def compilePooled[A](program: Program[A]): IO[A] =
pooled
def compilePooled[A](program: Program[A]): IO[A] = IO {
pooled
def compilePooled[A](program: Program[A]): IO[A] = IO { for { server <- pool.borrow
pooled
def compilePooled[A](program: Program[A]): IO[A] = IO { for { server <- pool.borrow result <- try server execute program finally pool + server
pooled
def compilePooled[A](program: Program[A]): IO[A] = IO { for { server <- pool.borrow result <- try server execute program finally pool + server } yield result }
problems
problems
- error handling
problems
- error handling
- tail recursion/stack overflow
error handling
error handling
sealed trait Invalid
error handling
sealed trait Invalid case class Message(s: String) extends Invalid
error handling
sealed trait Invalid case class Message(s: String) extends Invalid case class Err(x: Throwable) extends Invalid { … // define custom eq as Throwable doesn’t define eq }
error handling
sealed trait Invalid case class Message(s: String) extends Invalid case class Err(x: Throwable) extends Invalid { … // define custom eq as Throwable doesn’t define eq } case class Composite(left: Invalid, right: Invalid) extends Invalid
error handling
sealed trait Invalid case class Message(s: String) extends Invalid case class Err(x: Throwable) extends Invalid { … // define custom eq as Throwable doesn’t define eq } case class Composite(left: Invalid, right: Invalid) extends Invalid // scalaz.\/ is a far better Either[Invalid, A]
error handling
sealed trait Invalid case class Message(s: String) extends Invalid case class Err(x: Throwable) extends Invalid { … // define custom eq as Throwable doesn’t define eq } case class Composite(left: Invalid, right: Invalid) extends Invalid // scalaz.\/ is a far better Either[Invalid, A] type Result[+A] = Invalid \/ A
tail recursion
tail recursion
- to turn a tail recursive program into a non-tail-
recursive one we need trampolining
tail recursion
- to turn a tail recursive program into a non-tail-
recursive one we need trampolining
- essentially turns recursive function application into
continuation objects, replacing stack with heap
tail recursion
- to turn a tail recursive program into a non-tail-
recursive one we need trampolining
- essentially turns recursive function application into
continuation objects, replacing stack with heap
- fortunately there’s an excellent implementation of
this already built: scalaz.Free monad[1]
- n noes, too many Monads!
- n noes, too many Monads!
- lots of Monads
- Free
- Result – Either or \/
- Program
- IO
- n noes, too many Monads!
- lots of Monads
- Free
- Result – Either or \/
- Program
- IO
- stack them together with MonadTransformers
ReaderT
ReaderT
type ReaderT[F[+_], -E, +A] = Kleisli[F, E, A]
ReaderT
type ReaderT[F[+_], -E, +A] = Kleisli[F, E, A] /** Represents a function `A => M[B]`. */ sealed trait Kleisli[M[+_], -A, +B] { self =>
ReaderT
type ReaderT[F[+_], -E, +A] = Kleisli[F, E, A] /** Represents a function `A => M[B]`. */ sealed trait Kleisli[M[+_], -A, +B] { self => def run(a: A): M[B]
ReaderT
type ReaderT[F[+_], -E, +A] = Kleisli[F, E, A] /** Represents a function `A => M[B]`. */ sealed trait Kleisli[M[+_], -A, +B] { self => def run(a: A): M[B] def map[C](f: B => C)(implicit M: Functor[M]) : Kleisli[M, A, C] = kleisli(a => M.map(run(a))(f))
ReaderT
type ReaderT[F[+_], -E, +A] = Kleisli[F, E, A] /** Represents a function `A => M[B]`. */ sealed trait Kleisli[M[+_], -A, +B] { self => def run(a: A): M[B] def map[C](f: B => C)(implicit M: Functor[M]) : Kleisli[M, A, C] = kleisli(a => M.map(run(a))(f)) /**Construct a Kleisli from a Function1 */
ReaderT
type ReaderT[F[+_], -E, +A] = Kleisli[F, E, A] /** Represents a function `A => M[B]`. */ sealed trait Kleisli[M[+_], -A, +B] { self => def run(a: A): M[B] def map[C](f: B => C)(implicit M: Functor[M]) : Kleisli[M, A, C] = kleisli(a => M.map(run(a))(f)) /**Construct a Kleisli from a Function1 */ def kleisli[M[+_], A, B](f: A => M[B]) = new Kleisli[M, A, B] { def run(a: A) = f(a) } }
monad stack
monad stack
type Id[+X] = X
monad stack
type Id[+X] = X type ResultT[F[+_], +A] = scalaz.EitherT[F, Invalid, A]
monad stack
type Id[+X] = X type ResultT[F[+_], +A] = scalaz.EitherT[F, Invalid, A] type Result[+A] = ResultT[Id, A]
monad stack
type Id[+X] = X type ResultT[F[+_], +A] = scalaz.EitherT[F, Invalid, A] type Result[+A] = ResultT[Id, A] // type alias for Free partially applied to Id type FreeId[+A] = Free[Id, A]
monad stack
type Id[+X] = X type ResultT[F[+_], +A] = scalaz.EitherT[F, Invalid, A] type Result[+A] = ResultT[Id, A] // type alias for Free partially applied to Id type FreeId[+A] = Free[Id, A] type Connected[+A] = ReaderT[FreeId, Connection, A]
monad stack
type Id[+X] = X type ResultT[F[+_], +A] = scalaz.EitherT[F, Invalid, A] type Result[+A] = ResultT[Id, A] // type alias for Free partially applied to Id type FreeId[+A] = Free[Id, A] type Connected[+A] = ReaderT[FreeId, Connection, A] type Program[+A] = ResultT[Connected, A]
monad stack
type Id[+X] = X type ResultT[F[+_], +A] = scalaz.EitherT[F, Invalid, A] type Result[+A] = ResultT[Id, A] // type alias for Free partially applied to Id type FreeId[+A] = Free[Id, A] type Connected[+A] = ReaderT[FreeId, Connection, A] type Program[+A] = ResultT[Connected, A] type IOResult[+A] = ResultT[IO, A]
which allows us to
which allows us to
// remember this? def mean(x: Seq[Double]): Program[Double] = native("mean", x) flatMap toDouble
which allows us to
// remember this? def mean(x: Seq[Double]): Program[Double] = native("mean", x) flatMap toDouble val toDouble: REXP => Program[Double] = rexp => { val d = rexp.asDouble if (d.isNaN) error("Computed NaN from REXP".invalid) else d }
which allows us to
// remember this? def mean(x: Seq[Double]): Program[Double] = native("mean", x) flatMap toDouble val toDouble: REXP => Program[Double] = rexp => { val d = rexp.asDouble if (d.isNaN) error("Computed NaN from REXP".invalid) else d } type Program[+A] = EitherT[ReaderT[Free[Id,A], Connection,A], Invalid,A]
and it looks like
and it looks like
def incrementalStats(key: ActionKey) = { debug(key) for { current <- fetchBuild(key).toProgram _ = debug("Stats for" + key.toString) values = current.map { _.value } (mean, sd, samples) <- computeStats(values) correlation <- cor(values, current.map { _.testtime }) (ks, lastmeandiff) <- { for { previd <- catchingToResult { findBuilds(key.product, key.series).filter { getId(_) < getId(current.head.testid) }.maxBy { getId } }.leftMap { _ => "No previous build".invalid }.toProgram prev <- fetchBuild(ActionKey(key.product, key.series, previd, key.action)).toProgram stats <- computeLastStats(values, prev.map { _.value }) } yield stats }.leftMap { x => { debug(x); x } }.orElse { const(1D, 1D) } } yield (key.product, key.build, key.series, key.action, mean, sd, ks, lastmeandiff, samples, correlation) }why?
why?
So, what was the point of all this? Why did we put things inside those pesky monads again?
why?
So, what was the point of all this? Why did we put things inside those pesky monads again? Well, by putting everything inside very carefully labelled boxes we have been able to abstract and generalise away all the mess in dealing with IO & Rserv processes and compose programs using them into larger ones that all Do the Right Thing™ automatically.
was it hard?
was it hard?
yes!
was it hard?
yes!
it took hours to get everything compiling,
was it hard?
yes!
it took hours to get everything compiling, and quite a lot of research
was it worth it?
was it worth it?
after all those hours (days?) getting it all to compile, we ran it
was it worth it?
after all those hours (days?) getting it all to compile, we ran it it worked perfectly first time
was it worth it?
after all those hours (days?) getting it all to compile, we ran it it worked perfectly first time it worked so well, MPJ though it wasn’t running; he was used to it opening and closing connections all the time, executing our whole program used just the one
was it worth it?
after all those hours (days?) getting it all to compile, we ran it it worked perfectly first time it worked so well, MPJ though it wasn’t running; he was used to it opening and closing connections all the time, executing our whole program used just the one and I got a trip to LambdaJam!
Stackless Scala With Free Monads Runar Óli Bjarnason http://days2012.scala-lang.org/sites/days2012/files/bjarnason_trampolines.pdf Using the Free Monad to Avoid Stack Overflows in Scala Michael Peyton Jones http://termsandtruthconditions.herokuapp.com//blog/2013/03/16/free-monad/ Free: Interpreters and Little Languages Mark Hibberd http://mth.io/talks/free/ scalaz http://github.com/scalaz/scalaz/
references