Build Systems Neil Mitchell @ndm_haskell https://ndmitchell.com A - - PowerPoint PPT Presentation

▶
build systems
SMART_READER_LITE
LIVE PREVIEW

Build Systems Neil Mitchell @ndm_haskell https://ndmitchell.com A - - PowerPoint PPT Presentation

Sep 13, 2022 15 likes •433 views

Distributed Build Systems Neil Mitchell @ndm_haskell https://ndmitchell.com A simple build system main.exe : main.o gcc -o main.exe main.o main.o : main.c gcc -c main.c Make, 1976 (42 years ago, 12BG) Build system definition A build

slide-1
SLIDE 1

Distributed Build Systems

Neil Mitchell @ndm_haskell https://ndmitchell.com

slide-2
SLIDE 2

main.exe : main.o gcc -o main.exe main.o main.o : main.c gcc -c main.c Make, 1976

(42 years ago, 12BG)

A simple build system

slide-3
SLIDE 3

We focus on general-purpose build systems A build system performs necessary actions, respecting dependencies

Build system definition

slide-4
SLIDE 4

Excel Shake Ninja Nix Buck Bazel Pants Make Hippo

Build systems

slide-5
SLIDE 5

Excel Shake Ninja Nix Buck Bazel Pants Make

necessary actions respecting dependencies suspend restart topological dirty verifying trace constructive trace deterministic constructive trace Engineering +

Build Systems à la Carte

slide-6
SLIDE 6

The order in which to execute tasks

  • Topological
  • Restart
  • Suspend

RESPECTING DEPENDENCIES

slide-7
SLIDE 7
  • When do I tell you my dependencies?

– Applicative: Before doing anything, in advance – Monadic: Before I use them

“Monadic” dependencies

main.o : need main.c need $(includes_of main.c) gcc -c main.c main.c : …

slide-8
SLIDE 8
  • Only works for Applicative dependencies
  • Build a graph, traverse graph

main.exe main.o main.c util.h util.o util.c

Topological

slide-9
SLIDE 9
  • Build a rule
  • If it depends on a rule not yet built

– Restart: Cancel this rule, schedule it last, build dep – Suspend: Pause this rule, build dep, resume

  • Can you cancel or pause your rules?
  • Pause requires more memory, but less work

Restart/Suspend

slide-10
SLIDE 10
  • Bazel

– Use the applicative dependencies to part order – Doesn’t really allow user written monadic deps

  • Excel

– Keep a list of the order that worked last time – Consequence: Your sheet calcs faster over time!

Tricks for restarting

slide-11
SLIDE 11
  • Topological – Applicative only, easy
  • Restart – May duplicate work
  • Suspend – May be hard to orchestrate

Shake

  • Shake’s raison d'être is monadic deps
  • Uses continuations to efficiently suspend

– First version used green threads

Respecting dependencies

slide-12
SLIDE 12

I rebuilt this rule last time, should I do so again?

  • Dirty
  • Verifying trace
  • Constructive trace
  • Deterministic constructive trace

NECESSARY ACTIONS

slide-13
SLIDE 13

A rule is dirty if anything it depends on is dirty

  • Excel records it directly
  • Make encodes dirty bit with relative modtimes

– modtime(in) > modtime(out) = dirty – Cute trick: outputting a new result clears the bit, and propagates dirty bits upstream

  • You need to know your deps, ~Applicative only

Dirty bit

slide-14
SLIDE 14

A trace records the relevant bit of the state

  • What did I depend on last time?
  • What were the values of those things?

main.o depends on main.c, which had hash 0x12

  • If the trace matches, don’t rerun

Verifying trace

slide-15
SLIDE 15
  • What if I build but don’t change?
  • Possible with Dirty? Possible with Verifying?

main.exe main.o main.c util.h util.o util.c

Early cut-off

slide-16
SLIDE 16

Aka “Cloud build” or “Distributed build systems”

  • Record the output with the trace
  • Shove all the traces on the server
  • Now you can download already built stuff

Lots of engineering involved…

Constructive traces

slide-17
SLIDE 17

Imagine the output of a rule depends only on its inputs (deterministic)

  • Given the inputs, I can predict the value of any
  • utput, download the final answer
  • Less round-trips to the server
  • Doesn’t support cut-off

Deterministic constructive traces

slide-18
SLIDE 18
  • Dirty – ~Applicative only
  • Verifying trace – local only
  • Constructive trace
  • Deterministic constructive trace – no cut-off

Shake

  • Uses optimised verifying trace (two versions)

Necessary actions

slide-19
SLIDE 19

Excel Shake HEAD Ninja Nix Buck Bazel Pants

necessary actions respecting dependencies suspend restart topological dirty verifying trace constructive trace deterministic constructive trace

Accepted to ICFP 2018 with Andrey Mokhov, Simon Peyton Jones

Build Systems à la Carte

Make

Engineering +

slide-20
SLIDE 20

Engineering: Shake

Neil Mitchell @ndm_haskell https://shakebuild.com

slide-21
SLIDE 21

PhD build system Haskell EDSL Standard Chartered Replace Make with Shake Academic paper Monadic dependencies Open source

Papers with Andrey Mokhov, Simon Peyton Jones, Simon Marlow

Engineering GHC build system Commercial users Comparative academic paper Distributed

Rewind the clock

Academic paper

slide-22
SLIDE 22
  • ut : in

cp in out "out" %> \out -> do need ["in"] cmd "cp in out"

:: Rule () Monad Rule :: Action () Monad Action (%>) :: FilePattern -> (FilePath -> Action ()) -> Rule ()

Simple Shake

slide-23
SLIDE 23

result.tar notes.txt talk.pdf pic.jpg

import Development.Shake import Development.Shake.FilePath main = shakeArgs shakeOptions $ do want ["result.tar"] "*.tar" %> \out -> do need [out -<.> "lst"] contents <- readFileLines $ out -<.> "lst" need contents cmd "tar -cf" [out] contents

result.lst notes.txt talk.pdf pic.jpg

Longer example

slide-24
SLIDE 24

MyGen.hs MySource.xml MySource.c MySource.o

What does MySource.o depend on?

Generated files

slide-25
SLIDE 25
  • Hardcode it?

– Very fragile.

  • Hack an approximation of MyGen?

– Slow, somewhat fragile, a lot of effort.

  • Build in stages?

– Non-compositional

  • Run MyGen.hs and look at MySource.c

– Easy, fast, precise. Use monadic dependencies.

Generated approaches

slide-26
SLIDE 26
  • If any rule needs monadic, you need it

– Even if “rare” in your system

  • Workarounds are not compositional
  • Generated files cry out for monadic

– Generated code is common in large projects

  • Advice: Don’t use a non-monadic system

Monadic is necessary

slide-27
SLIDE 27

Build system Monadic + suspend Modern engineering + Haskell

Shake

Syntax Types Abstraction Libraries Monads Profiling Lint Analysis Parallelism Robustness Efficient

slide-28
SLIDE 28
  • In use for three nine years:

– 1M+ build runs, 30K+ build objects, 1M+ lines source, 1M+ lines generated

  • Replaced 10,000 lines of Makefile

with 1,000 lines of Shake scripts

– Twice as fast to compile from scratch – Massively more robust

Disclaimer: I used to be employed by Standard Chartered Bank. These slides do not represent the views of Standard Chartered.

Shake at Standard Chartered (2012)

slide-29
SLIDE 29

Ready for primetime!

  • Standard Chartered have been using Shake since 2009,

1000’s of compiles per day.

  • factis research GmbH use Shake to compile their

Checkpad MED application.

  • Samplecount have been using Shake since 2012,

producing several open-source projects for working with Shake.

  • CovenantEyes use Shake to build their Windows client.
  • Keystone Tower Systems has a robotic welder with a

Shake build system.

  • FP Complete use Shake to build Docker images.

Don’t write a build system unless you have to!

slide-30
SLIDE 30
  • Syntax, reasonable DSLs
  • Some use of the type system (not heavy)
  • Abstraction, functions/modules/packages
  • Profiling the Haskell functions

Stealing from Haskell

slide-31
SLIDE 31
  • HTML profile reports
  • Very multithreaded
  • Progress reporting
  • Reports of live files
  • Lint reports
  • …

Extra features

slide-32
SLIDE 32

Why is Shake fast?

  • What does fast even mean?

– Everything changed? Rebuild from scratch. – Nothing changed? Rebuild nothing.

  • In practice, a blend, but optimise both

extremes and you win

slide-33
SLIDE 33

Fast when nothing changes

  • Don’t run users rules if you can avoid it
  • Shake records a verifying trace, [(k, v, …)]
  • Avoid lots of locking/parallelism

– Take a lock, check storedValue a lot

  • Binary serialisation is a bottleneck

unchanged journal = flip allM journal $ \(k,v) -> (== Just v) <$> storedValue k

slide-34
SLIDE 34

Fast when everything changes

  • If everything changes, rule dominate (you hope)
  • One rule: Start things as soon as you can

– Dependencies should be fine grained – Start spawning before checking everything – Make use of multiple cores – Randomise the order of dependencies (~15% faster)

  • Expressive dependencies, Continuation monad,

cheap threads, immutable values (easy in Haskell)

slide-35
SLIDE 35

State changes

Ready Error Running Loaded Missing

slide-36
SLIDE 36

Inside “Running”

  • Build all my dependencies from last time

– If any changed, then dirty

  • Look at my result from last time

– If it has changed, then dirty

  • If dirty, see if I’m in the constructive trace

– If I am, copy the result into my trace

  • If still dirty

– Run the user supplied action

slide-37
SLIDE 37

Efficient suspend

  • Continuations are mind-blowing (still)
  • a = I get given ‘a’ now
  • (a -> r) -> r = I get given ‘a’ later
  • Covariant/contravariant equivalence
  • Efficiently pause a running computation

a (a -> r) -> r

slide-38
SLIDE 38

Efficient resume

  • Resumption is restarting suspended things
  • Resume everything when changing status

– Resumption is required to be “quick” – Therefore most resumption adds to the Pool...

data Status = Running [Either Error Ready -> IO ()] | …

slide-39
SLIDE 39

Efficient parallelism

  • A thread pool
  • Not to reduce thread overhead

– Haskell threads are super cheap

  • To limit parallelism, and cleanup/finish

addPool :: Pool -> PoolPriority -> IO () -> IO ()

slide-40
SLIDE 40

Efficient journaling

  • Shake needs to record the verifying traces

– Recorded in .shake.database

  • A linear record of traces

– Append to the end – Size prefixed to detect corruption – Compact if < ½ the values still useful – Flush every 5s

slide-41
SLIDE 41

Conclusions

  • Build systems make three choices:

– Respecting dependencies – Necessary actions – Engineering choices

  • Shake occupies an interesting spot

– Plenty of engineering required to make it work