SLIDE 1
High ! Availability ! at ! Heroku Mark ! McGranaghan - - PowerPoint PPT Presentation
High ! Availability ! at ! Heroku Mark ! McGranaghan - - PowerPoint PPT Presentation
High ! Availability ! at ! Heroku Mark ! McGranaghan
SLIDE 2
SLIDE 3
SLIDE 4
SLIDE 5
SLIDE 6
SLIDE 7
SLIDE 8
SLIDE 9
SLIDE 10
Scale !& !Scope
SLIDE 11
O(1,000) !instances O(1,000,000) !apps
SLIDE 12
Success !& !Failure
SLIDE 13
Architecture Execution
SLIDE 14
Architecture Execution
SLIDE 15
SLIDE 16
Platform-Enabled HA !Routing
SLIDE 17
SLIDE 18
SLIDE 19
SLIDE 20
SLIDE 21
Crashes ! & Supervision
SLIDE 22
SLIDE 23
SLIDE 24
SLIDE 25
Crashes !as !the !
- nly !code !path
SLIDE 26
SLIDE 27
SLIDE 28
Crashes !as !a !hot ! code !path
SLIDE 29
SLIDE 30
SLIDE 31
Error !Kernel
SLIDE 32
SLIDE 33
Layered !design
SLIDE 34
SLIDE 35
Message ! passing...
SLIDE 36
SLIDE 37
{slug: “https://aws...”, cmd: “java ...”, env: {“JAVA_OPTS”: ..., “DATABASE_URL”: ..., “SESSION_SECRET”: ...}}
SLIDE 38
...of !narrow, ! versioned !values
SLIDE 39
{slug: “https://s3...”, cmd: “java -cp ...”, env: {“JAVA_OPTS”: ..., “DATABASE_URL”: ..., “SESSION_SECRET”: ...}, flag: “extra_cpu”}
SLIDE 40
SLIDE 41
No !Stopping !the ! World
SLIDE 42
SLIDE 43
Load !balancing Supervision Crash-only Error !kernels Layered !design Message-passing
SLIDE 44
SLIDE 45
Erlang
Designed !for granular !failure
SLIDE 46
Distributed ! Systems
Defined !as granular !failure
SLIDE 47
Brokered ! Queueing
SLIDE 48
SLIDE 49
SLIDE 50
SLIDE 51
SLIDE 52
Publish !one !/ Subscribe !many
SLIDE 53
SLIDE 54
SLIDE 55
SLIDE 56
Distributed !call ! graphs
SLIDE 57
SLIDE 58
SLIDE 59
SLIDE 60
SLIDE 61
Read !call !graph Partial !failure
SLIDE 62
SLIDE 63
Write !call !graph de-synchronizing
SLIDE 64
SLIDE 65
SLIDE 66
Architecture Execution
SLIDE 67
Architecture Execution
SLIDE 68
“...we !deployed !a !code !change... ...visible !under !unusual...conditions... ...introduced !a...problem... ...engineers !noticed !a !deviation... ...began !to !escalate... ...system...entered !into !a !feedback !loop... ...engineers...deactivated !the !feedback...
SLIDE 69
Evolving Socio-Technical ! Systems
SLIDE 70
SLIDE 71
SLIDE 72
Availability !>> !Architecture
SLIDE 73
Failed !deploys Bad !visibility Cascading !feedback
SLIDE 74
Evolving Socio-Technical ! Systems
SLIDE 75
Failed !deploys Bad !visibility Cascading !feedback
SLIDE 76
Deploy !tooling Visibility !services Feedback !controls
SLIDE 77
bin/ship
SLIDE 78
bin/ship \
- -component api \
- -version v408
SLIDE 79
Incremental !deploys
SLIDE 80
SLIDE 81
SLIDE 82
Incremental !rollouts
SLIDE 83
prep_launch
- launch_without_lxc
+ launch-with_lxc monitor_launch
SLIDE 84
if flag_on?(“lxc”) launch_with_lxc else launch_without_lxc
SLIDE 85
app.flag_on(“lxc”) app.flag_off(“lxc”)
SLIDE 86
SLIDE 87
SLIDE 88
Real-time !visibility
SLIDE 89
SLIDE 90
Service-level !assertions
SLIDE 91
SLIDE 92
assert(index > 0)
SLIDE 93
assert(index > 0)
- bjects[index]
SLIDE 94
assert(p99_latency < 50)
SLIDE 95
assert(p99_latency < 50)
SLIDE 96
assert(active_cons > 10)
SLIDE 97
assert(active_cons > 10)
SLIDE 98
Flow !control & !Backpressure
SLIDE 99
SLIDE 100
SLIDE 101
SLIDE 102
SLIDE 103
limit(“publish”, 10) do Bus.publish(msg) end
SLIDE 104
limit(“publish”, prate) do publish(msg) end echo 0 >/etc/rates/publish
SLIDE 105
Evolving Socio-Technical ! Systems
SLIDE 106
Architecture Execution
SLIDE 107
SLIDE 108
Thanks
SLIDE 109