Without Resilience Nothing Else Matters Jonas Bonr CTO TypEsafe - - PowerPoint PPT Presentation

▶
without resilience nothing else matters
SMART_READER_LITE
LIVE PREVIEW

Without Resilience Nothing Else Matters Jonas Bonr CTO TypEsafe - - PowerPoint PPT Presentation

Oct 11, 2022 167 likes •1.5k views

Without Resilience Nothing Else Matters Jonas Bonr CTO TypEsafe @jboner Without Resilience Nothing Else Matters Jonas Bonr CTO TypEsafe @jboner But it aint how hard youre hit; its about how hard you can get hit, and keep

slide-1
SLIDE 1

Without Resilience Nothing Else Matters

Jonas Bonér

CTO TypEsafe @jboner
slide-2
SLIDE 2

Without Resilience Nothing Else Matters

Jonas Bonér

CTO TypEsafe @jboner
slide-3
SLIDE 3

“But it ain’t how hard you’re hit; it’s about how hard you can get hit, and keep moving forward. How much you can take, and keep moving forward. That’s how winning is done.”

  • Rocky Balboa
slide-4
SLIDE 4

“But it ain’t how hard you’re hit; it’s about how hard you can get hit, and keep moving forward. How much you can take, and keep moving forward. That’s how winning is done.”

  • Rocky Balboa

This is Fault Tolerance

slide-5
SLIDE 5

Resilience is Beyond Fault Tolerance

slide-6
SLIDE 6

Resilience

“The ability of a substance or

  • bject to spring back into shape.

The capacity to recover quickly from difficulties.”

  • Merriam Webster
slide-7
SLIDE 7

Antifragility

“Antifragility is beyond resilience and

  • robustness. The resilient resists shock and

stays the same; the antifragile gets better.”

  • Nassem Nicholas Taleb
Antifragile: Things That Gain from Disorder - Nassim Nicholas Taleb
slide-8
SLIDE 8

“We can model and understand in isolation. 
 But, when released into competitive nominally regulated societies, their connections proliferate, 
 their interactions and interdependencies multiply, 
 their complexities mushroom. 
 And we are caught short.”

  • Sidney Dekker
Drift into Failure - Sidney Dekker
slide-9
SLIDE 9

We Need to Study Resilience in Complex Systems

slide-10
SLIDE 10

Complicated System

slide-11
SLIDE 11

Complicated System

slide-12
SLIDE 12

Complex System

slide-13
SLIDE 13

Complex System

slide-14
SLIDE 14

Complex System

slide-15
SLIDE 15

Complicated ≠ Complex

slide-16
SLIDE 16

“Counterintuitive. That’s [Jay] Forrester’s word to describe complex systems. Leverage points are not intuitive. Or if they are, we intuitively use them backward, systematically worsening whatever problems we are trying to solve.”

  • Donella Meadows
Leverage Points: Places to Intervene in a System - Donella Meadows
slide-17
SLIDE 17 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013

Operating at the Edge of Failure

slide-18
SLIDE 18 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Economic Failure Boundary

Operating at the Edge of Failure

slide-19
SLIDE 19 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Economic Failure Boundary Unacceptable Workload Boundary

Operating at the Edge of Failure

slide-20
SLIDE 20 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Economic Failure Boundary Unacceptable Workload Boundary Accident Boundary

Operating at the Edge of Failure

slide-21
SLIDE 21 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Economic Failure Boundary Unacceptable Workload Boundary Operating Point Accident Boundary

Operating at the Edge of Failure

slide-22
SLIDE 22 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Economic Failure Boundary Unacceptable Workload Boundary Accident Boundary

Operating at the Edge of Failure

slide-23
SLIDE 23 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Economic Failure Boundary Unacceptable Workload Boundary FAILURE Accident Boundary

Operating at the Edge of Failure

slide-24
SLIDE 24 Economic Failure Boundary Unacceptable Workload Boundary Accident Boundary ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013

Operating at the Edge of Failure

slide-25
SLIDE 25 Economic Failure Boundary Unacceptable Workload Boundary Accident Boundary Management Pressure Towards Economic Efficiency ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013

Operating at the Edge of Failure

slide-26
SLIDE 26 Economic Failure Boundary Unacceptable Workload Boundary Accident Boundary Management Pressure Towards Economic Efficiency Gradient Towards Least Effort ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013

Operating at the Edge of Failure

slide-27
SLIDE 27 Economic Failure Boundary Unacceptable Workload Boundary Accident Boundary Management Pressure Towards Economic Efficiency Gradient Towards Least Effort ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013

Operating at the Edge of Failure

slide-28
SLIDE 28 Economic Failure Boundary Unacceptable Workload Boundary Accident Boundary Management Pressure Towards Economic Efficiency Gradient Towards Least Effort Counter Gradient For More Resilience ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013

Operating at the Edge of Failure

slide-29
SLIDE 29 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Economic Failure Boundary Unacceptable Workload Boundary Accident Boundary

Operating at the Edge of Failure

slide-30
SLIDE 30 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Economic Failure Boundary Unacceptable Workload Boundary Accident Boundary Error Margin Marginal Boundary

Operating at the Edge of Failure

slide-31
SLIDE 31 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Economic Failure Boundary Unacceptable Workload Boundary Accident Boundary Error Margin Marginal Boundary

Operating at the Edge of Failure

slide-32
SLIDE 32 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Economic Failure Boundary Unacceptable Workload Boundary Accident Boundary Error Margin Marginal Boundary

Operating at the Edge of Failure

slide-33
SLIDE 33 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Accident Boundary Marginal Boundary

Operating at the Edge of Failure

slide-34
SLIDE 34 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Marginal Boundary

?

Operating at the Edge of Failure

slide-35
SLIDE 35 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013

Operating at the Edge of Failure

Accident Boundary Marginal Boundary
slide-36
SLIDE 36 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013

Operating at the Edge of Failure

Accident Boundary Marginal Boundary
slide-37
SLIDE 37 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013

Operating at the Edge of Failure

Accident Boundary Marginal Boundary
slide-38
SLIDE 38 ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013

Operating at the Edge of Failure

Accident Boundary Marginal Boundary
slide-39
SLIDE 39

Embrace Failure

slide-40
SLIDE 40

Resilience in Social Systems

slide-41
SLIDE 41

Dealing in Security

Understanding vital services, and how they keep you safe

1 INDIVIDUAL 6 ways to die 3 sets of essential services 7 layers of PROTECTION

Dealing in Security - Mike Bennet, Vinay Gupta
slide-42
SLIDE 42

7 Principles for Building Resilience in Social Systems

  • 1. Maintain diversity & Redundancy
  • 2. Manage connectivity
  • 3. Manage slow variables & feedback
  • 4. Foster complex adaptive systems thinking
  • 5. Encourage learning
  • 6. Broaden participation
  • 7. Promote polycentric governance
Applying resilience thinking: Seven principles for building resilience in social-ecological systems - Reinette Biggs et. al.
slide-43
SLIDE 43

Resilience in Biological Systems

slide-44
SLIDE 44

Meerkats

Puppies! Now that I’ve got your attention, complexity theory - Nicolas Perony, TED talk
slide-45
SLIDE 45

What We Can Learn From Biological Systems

  • 1. Feature Diversity and redundancy
  • 2. Inter-Connected network structure
  • 3. Wide distribution across all scales
  • 4. Capacity to self-adapt & self-organize
Toward Resilient Architectures 1: Biology Lessons - Michael Mehaffy, Nikos A. Salingaros
slide-46
SLIDE 46

“Animals show extraordinary social complexity, and this allows them to adapt and 
 respond to changes in their environment. In three words, in the animal kingdom, simplicity leads to complexity 
 which leads to resilience.”

  • Nicolas Perony
Puppies! Now that I’ve got your attention, complexity theory - Nicolas Perony, TED talk
slide-47
SLIDE 47

Resilience in Computer Systems

slide-48
SLIDE 48
slide-49
SLIDE 49

“Complex systems run in degraded mode.” “Complex systems run as broken systems.”

  • richard Cook
How Complex Systems Fail - Richard Cook
slide-50
SLIDE 50

Resilience is by Design

Photo courtesy of FEMA/Joselyne Augustino
slide-51
SLIDE 51

We Need to Manage Failure

slide-52
SLIDE 52

“Post-accident attribution to a 
 ‘root cause’ is fundamentally wrong: 
 Because overt failure requires multiple faults, there is no isolated ‘cause’ of an accident.”

  • richard Cook
How Complex Systems Fail - Richard Cook
slide-53
SLIDE 53

There is No Root Cause

slide-54
SLIDE 54

Crash Only Software

Crash-Only Software - George Candea, Armando Fox

Stop = Crash Safely Start = Recover Fast

slide-55
SLIDE 55

Recursive Restartability

Turning the Crash-Only Sledgehammer into a Scalpel

Recursive Restartability: Turning the Reboot Sledgehammer into a Scalpel - George Candea, Armando Fox
slide-56
SLIDE 56

Services need to accept NO for an answer

slide-57
SLIDE 57

Classification

  • f State
  • Static Data
  • Scratch Data
  • Dynamic Data
  • Recomputable
  • not recomputable
slide-58
SLIDE 58

Classification

  • f State
  • Static Data
  • Scratch Data
  • Dynamic Data
  • Recomputable
  • not recomputable

Critical

slide-59
SLIDE 59

Traditional State Management

Object Critical state that needs protection Client Thread boundary
slide-60
SLIDE 60

Traditional State Management

Object Critical state that needs protection Client Thread boundary
slide-61
SLIDE 61

Traditional State Management

Object Critical state that needs protection Client Thread boundary
slide-62
SLIDE 62

Traditional State Management

Object Critical state that needs protection Client Thread boundary Synchronous dispatch Thread boundary
slide-63
SLIDE 63

Traditional State Management

Object Critical state that needs protection Client Thread boundary Synchronous dispatch Thread boundary
slide-64
SLIDE 64

Traditional State Management

Object Critical state that needs protection Client Thread boundary Synchronous dispatch Thread boundary ?
slide-65
SLIDE 65

Traditional State Management

Object Critical state that needs protection Client Thread boundary Synchronous dispatch Thread boundary ?

Utterly broken

slide-66
SLIDE 66

“Accidents come from relationships not broken parts.”

  • Sidney dekker
Drift into Failure - Sidney Dekker
slide-67
SLIDE 67

Requirements for a

Sane Failure Mode

  • 1. Contained
  • 2. Reified—as messages
  • 3. Signalled—Asynchronously
  • 4. Observed—by 1-N
  • 5. Managed

Failures need to be

slide-68
SLIDE 68

Bulkhead Pattern

slide-69
SLIDE 69

Bulkhead Pattern

slide-70
SLIDE 70

Bulkhead Pattern

slide-71
SLIDE 71

Enter Supervision

slide-72
SLIDE 72

Enter Supervision

slide-73
SLIDE 73

The Vending Machine Pattern

slide-74
SLIDE 74

Think Vending Machine

Coffee Machine Programmer
slide-75
SLIDE 75

Think Vending Machine

Coffee Machine Programmer Inserts coins
slide-76
SLIDE 76

Think Vending Machine

Coffee Machine Programmer Inserts coins Add more coins
slide-77
SLIDE 77

Think Vending Machine

Coffee Machine Programmer Inserts coins Gets coffee Add more coins
slide-78
SLIDE 78

Think Vending Machine

Programmer Coffee Machine
slide-79
SLIDE 79

Think Vending Machine

Programmer Inserts coins Coffee Machine
slide-80
SLIDE 80

Think Vending Machine

Programmer Inserts coins Out of coffee beans error Coffee Machine
slide-81
SLIDE 81

Think Vending Machine

Programmer Inserts coins Out of coffee beans error

WRONG

Coffee Machine
slide-82
SLIDE 82

Think Vending Machine

Programmer Inserts coins Coffee Machine
slide-83
SLIDE 83

Think Vending Machine

Programmer Inserts coins Out of coffee beans failure Coffee Machine
slide-84
SLIDE 84

Think Vending Machine

Programmer Service Guy Inserts coins Out of coffee beans failure Coffee Machine
slide-85
SLIDE 85

Think Vending Machine

Programmer Service Guy Inserts coins Out of coffee beans failure Adds more beans Coffee Machine
slide-86
SLIDE 86

Think Vending Machine

Programmer Service Guy Inserts coins Gets coffee Out of coffee beans failure Adds more beans Coffee Machine
slide-87
SLIDE 87

Think Vending Machine

Service Client
slide-88
SLIDE 88

Think Vending Machine

Service Client Request
slide-89
SLIDE 89

Think Vending Machine

Service Client Request Response
slide-90
SLIDE 90

Think Vending Machine

Service Client Request Response Validation Error
slide-91
SLIDE 91

Think Vending Machine

Service Client Request Response Validation Error Application Failure
slide-92
SLIDE 92

Think Vending Machine

Service Client Supervisor Request Response Validation Error Application Failure
slide-93
SLIDE 93

Think Vending Machine

Service Client Supervisor Request Response Validation Error Application Failure Manages Failure
slide-94
SLIDE 94
slide-95
SLIDE 95

Error Kernel Pattern

Onion-layered state & Failure management

Making reliable distributed systems in the presence of software errors - Joe Armstrong On Erlang, State and Crashes - Jesper Louis Andersen
slide-96
SLIDE 96

Onion Layered State Management

Object Critical state that needs protection Client Thread boundary
slide-97
SLIDE 97

Onion Layered State Management

Object Critical state that needs protection Client Thread boundary
slide-98
SLIDE 98

Onion Layered State Management

Error Kernel Object Critical state that needs protection Client Thread boundary
slide-99
SLIDE 99

Onion Layered State Management

Error Kernel Object Critical state that needs protection Client Thread boundary
slide-100
SLIDE 100

Onion Layered State Management

Error Kernel Object Critical state that needs protection Client Supervision Thread boundary
slide-101
SLIDE 101

Onion Layered State Management

Error Kernel Object Critical state that needs protection Client Supervision Supervision Thread boundary
slide-102
SLIDE 102

Onion Layered State Management

Error Kernel Object Critical state that needs protection Client Supervision Supervision Thread boundary
slide-103
SLIDE 103

Onion Layered State Management

Error Kernel Object Critical state that needs protection Client Supervision Supervision Thread boundary
slide-104
SLIDE 104

Onion Layered State Management

Error Kernel Object Critical state that needs protection Client Supervision Supervision Thread boundary
slide-105
SLIDE 105

Onion Layered State Management

Error Kernel Object Critical state that needs protection Client Supervision Supervision Thread boundary
slide-106
SLIDE 106

Onion Layered State Management

Error Kernel Object Critical state that needs protection Client Supervision Supervision Thread boundary
slide-107
SLIDE 107

Onion Layered State Management

Error Kernel Object Critical state that needs protection Client Supervision Supervision Thread boundary
slide-108
SLIDE 108

Onion Layered State Management

Error Kernel Object Critical state that needs protection Client Supervision Supervision Thread boundary
slide-109
SLIDE 109

Onion Layered State Management

Error Kernel Object Critical state that needs protection Client Supervision Supervision Thread boundary
slide-110
SLIDE 110

Maintain Diversity and Redundancy

slide-111
SLIDE 111

Maintain Diversity and Redundancy

slide-112
SLIDE 112

The Network is Reliable

slide-113
SLIDE 113

The Network is Reliable

NAT

slide-114
SLIDE 114

Strong Consistency

Is the wrong default

slide-115
SLIDE 115

We need Systems that are Decoupled IN

Time and Space

slide-116
SLIDE 116

Resilient Protocols

Depend on

Asynchronous Communication Eventual Consistency

slide-117
SLIDE 117

Resilient Protocols

  • are tolerant to
  • Message loss
  • Message reordering
  • Message duplication

Depend on

Asynchronous Communication Eventual Consistency

slide-118
SLIDE 118

Resilient Protocols

  • are tolerant to
  • Message loss
  • Message reordering
  • Message duplication
  • Embrace ACID 2.0
  • Associative
  • Commutative
  • Idempotent
  • Distributed

Depend on

Asynchronous Communication Eventual Consistency

slide-119
SLIDE 119

Testing

slide-120
SLIDE 120

What can we learn from Arnold?

slide-121
SLIDE 121

What can we learn from Arnold?

slide-122
SLIDE 122

What can we learn from Arnold? Blow things up

slide-123
SLIDE 123

Shoot Your App Down

slide-124
SLIDE 124

Pull the Plug

…and see what happens

slide-125
SLIDE 125
slide-126
SLIDE 126

Executive Summary

slide-127
SLIDE 127

“Complex systems run in degraded mode.” “Complex systems run as broken systems.”

  • richard Cook
How Complex Systems Fail - Richard Cook
slide-128
SLIDE 128

Resilience is by Design

Photo courtesy of FEMA/Joselyne Augustino
slide-129
SLIDE 129

Thank

You

slide-130
SLIDE 130

Thank

You

slide-131
SLIDE 131

References

Antifragile: Things That Gain from Disorder - http://www.amazon.com/Antifragile-Things-that-Gain-Disorder-ebook/dp/B009K6DKTS Drift into Failure - http://www.amazon.com/Drift-into-Failure-Components-Understanding-ebook/dp/B009KOKXKY How Complex Systems Fail - http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf Leverage Points: Places to Intervene in a System - http://www.donellameadows.org/archives/leverage-points-places-to-intervene-in-a-system/ Going Solid: A Model of System Dynamics and Consequences for Patient Safety - http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1743994/ Resilience in Complex Adaptive Systems: Operating at the Edge of Failure - https://www.youtube.com/watch?v=PGLYEDpNu60 Dealing in Security - http://resiliencemaps.org/files/Dealing_in_Security.July2010.en.pdf What is resilience? An introduction to social-ecological research - http://www.stockholmresilience.org/download/ 18.10119fc11455d3c557d6d21/1398172490555/SU_SRC_whatisresilience_sidaApril2014.pdf Applying resilience thinking: Seven principles for building resilience in social-ecological systems - http://www.stockholmresilience.org/download/ 18.10119fc11455d3c557d6928/1398150799790/SRC+Applying+Resilience+final.pdf Puppies! Now that I’ve got your attention, Complexity Theory - https://www.ted.com/talks/ nicolas_perony_puppies_now_that_i_ve_got_your_attention_complexity_theory How Bacteria Becomes Resistant - http://www.abc.net.au/science/slab/antibiotics/resistance.htm Towards Resilient Architectures: Biology Lessons - http://www.metropolismag.com/Point-of-View/March-2013/Toward-Resilient-Architectures-1-Biology- Lessons/ Crash-Only Software - https://www.usenix.org/legacy/events/hotos03/tech/full_papers/candea/candea.pdf Recursive Restartability: Turning the Reboot Sledgehammer into a Scalpel - http://roc.cs.berkeley.edu/papers/recursive_restartability.pdf Out of the Tar Pit - http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.93.8928 Bulkhead Pattern - http://skife.org/architecture/fault-tolerance/2009/12/31/bulkheads.html Making Reliable Distributed Systems in the Presence of Software Errors - http://www.erlang.org/download/armstrong_thesis_2003.pdf On Erlang, State and Crashes - http://jlouisramblings.blogspot.be/2010/11/on-erlang-state-and-crashes.html Akka Supervision - http://doc.akka.io/docs/akka/snapshot/general/supervision.html Release It!: Design and Deploy Production-Ready Software - https://pragprog.com/book/mnee/release-it Hystrix - https://github.com/Netflix/Hystrix Akka Circuit Breaker - http://doc.akka.io/docs/akka/snapshot/common/circuitbreaker.html Reactive Streams - http://reactive-streams.org Akka Streams - http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0/scala/stream-introduction.html RxJava - https://github.com/ReactiveX/RxJava Feedback Control for Computer Systems - http://www.amazon.com/Feedback-Control-Computer-Systems-Philipp/dp/1449361692 Simian Army - https://github.com/Netflix/SimianArmy Gatling - http://gatling.io Akka MultiNode Testing - http://doc.akka.io/docs/akka/snapshot/dev/multi-node-testing.html
slide-132
SLIDE 132

Q & A