Betting on Performance a note on hypothesis-driven performance - - PowerPoint PPT Presentation

▶
betting on performance
SMART_READER_LITE
LIVE PREVIEW

Betting on Performance a note on hypothesis-driven performance - - PowerPoint PPT Presentation

Jun 21, 2023 359 likes •1.2k views

Betting on Performance a note on hypothesis-driven performance testing James Lewis : @boicy 2 How do we usually make decisions about architecture, design and performance? based on experience based on gut feel and often times up front 3 Can we

slide-1
SLIDE 1

Betting on Performance

a note on hypothesis-driven performance testing

James Lewis : @boicy

slide-2
SLIDE 2 2
slide-3
SLIDE 3 3

How do we usually make decisions about architecture, design and performance?

based on experience based on gut feel and often times up front

slide-4
SLIDE 4 4

Can we make design decisions more systematically?

and what would that look like?

slide-5
SLIDE 5
slide-6
SLIDE 6
slide-7
SLIDE 7 7

History

The lawful good product owners of the publishing house had long lived in awe and fear of their publishing systems. In awe, for they had made a tremendous amount of Gold, but in fear of the time taken to change them, their slowness and their fragility. A messenger was sent to fetch help from a distant land famed for it’s mighty wizards. You have taken up the challenge…

link to close link hello close

slide-8
SLIDE 8 8

1.

You must save the product owners by rebuilding their website. You start off the project. In the course of discussions you discover that your goals are three fold:

  • 1. improve availability
  • 2. improve performance
  • 3. reduce the cost of delay

An Enterprise Architect approaches and addresses you. You may use:

Summon Walking Skeleton

?

turn to 4 Analysis Paralysis turn to 3

If you have none of these you will have to draw your sword and fight (turn to 178)

link to close link hello close

slide-9
SLIDE 9 9

1.

You must save the product owners by rebuilding their website. You start off the project. In the course of discussions you discover that your goals are three fold:

  • 1. improve availability
  • 2. improve performance
  • 3. reduce the cost of delay

An Enterprise Architect approaches and addresses you. You may use:

Summon Walking Skeleton

?

turn to 4 Analysis Paralysis turn to 3

If you have none of these you will have to draw your sword and fight (turn to 178)

link to close link hello close

slide-10
SLIDE 10 10

3.

You cast Analysis Paralysis at the Enterprise Architect. “Foolish young adventurer” says the architect, “we follow the evolutionary school of architecture and we shall have none of the lawful-evil ways of waterfall”. The last thing you see before everything goes dark is the architect incanting in a strange voice. You have died. Turn to page 1.

link to close link hello close

slide-11
SLIDE 11 11

1.

You must save the product owners by rebuilding their website. You start off the project. In the course of discussions you discover that your goals are three fold:

  • 1. improve availability
  • 2. improve performance
  • 3. reduce the cost of delay

An Enterprise Architect approaches and addresses you. You may use:

Summon Walking Skeleton

?

turn to 4 Analysis Paralysis turn to 3

If you have none of these you will have to draw your sword and fight (turn to 178)

link to close link hello close

slide-12
SLIDE 12 12

4.

Your walking skeleton coalesces in a cloud of noxious gasses and solidifies as a java dropwizard application. You reach into your backpack and deploy the content store. Your walking skeleton reaches out it’s skeletal arms and grabs armfuls of raw xml. Would you like to:

S3

Transform the xml inside the skeleton turn to 6 Use a magic box turn to 5

link to close link hello close

slide-13
SLIDE 13 13

monolithic MVC

  • app. All logic,

transformations and rendering happen in process distributed system Transform content within the walking skeleton create another service to transform content

slide-14
SLIDE 14 14

So how do you decide which path to follow?

slide-15
SLIDE 15 15

Betting on Performance

slide-16
SLIDE 16 16

Betting on Performance

slide-17
SLIDE 17 17

Option Option Option $ $ $ Evolutionary Architecture allows us to place small bets and then re- evaluate our decisions based on outcomes Option Option Option $ $$$$$$$$$ $ Up front design “bets the house”. It collapses the options by presupposing the answer (and that there is an answer)

slide-18
SLIDE 18 18

back to the story

slide-19
SLIDE 19 19

5.

You throw the magic box in between the walking skeleton and the content store. A villager approaches and exclaims: “this beautiful content I see in front

  • f me seems to take an awful long

time to get here” You must somehow make the content arrive faster. If you have a http cache in your inventory, you may use it now.

S3

Cache in between S3 and content turn to 10

content

Cache in between skeleton and content turn to 33

link to close link hello close

slide-20
SLIDE 20 20

ESI / Templating

Computationally expensive service

  • ther service
  • ther service

XML in S3 Xslt

slide-21
SLIDE 21

21

0.8 seconds for first byte 1.5 seconds for page load

slide-22
SLIDE 22 22

Page Load (sec) 10 20 30 40 Initial observations 1 35

0.8 seconds for first byte 1.5 seconds for page load

slide-23
SLIDE 23 23

ESI / Templating

Computationally expensive service

  • ther service
  • ther service

XML in S3 Xslt

slide-24
SLIDE 24 24

We’ve detected performance problems

slide-25
SLIDE 25 25

lets just add a cache

slide-26
SLIDE 26 26

ESI / Templating

Computationally expensive service

  • ther service
  • ther service

XML in S3 Xslt

slide-27
SLIDE 27 27

ESI / Templating

Computationally expensive service

  • ther service
  • ther service

XML in S3 Xslt

slide-28
SLIDE 28 28

Page Load (secs)

Cache misses predominate

Page Load (secs)

Cache hits predominate

slide-29
SLIDE 29 29

There are only two hard things in Computer Science: cache invalidation and naming things.

  • - Phil Karlton

Bonus Joke

There are only two hard things in messaging:

  • 2. Exactly Once Delivery
  • 1. Delivery Order
  • 2. Exactly Once Delivery
slide-30
SLIDE 30 30

lets just add a cache

said no one ever

slide-31
SLIDE 31 31

a note on hypothesis-driven performance testing

Betting on Performance

slide-32
SLIDE 32 32

a note on hypothesis-driven performance testing

slide-33
SLIDE 33 33

a note on hypothesis-driven performance testing

“In general, we look for a new law by the following process: First we guess it; then we compute the consequences of the guess to see what would be implied if this law that we guessed is right; then we compare the result of the computation to nature, with experiment or experience, compare it directly with

  • bservation, to see if it works. If it disagrees with experiment,

it is wrong. In that simple statement is the key to science. It does not make any difference how beautiful your guess is, it does not make any difference how smart you are, who made the guess, or what his name is — if it disagrees with experiment, it is wrong”

slide-34
SLIDE 34

Compute implications Compare with Nature Draw conclusion Observe nature Make a guess

slide-35
SLIDE 35

Compute implications Compare with Nature Draw conclusion Observe nature Make a guess

Observe metrics Measure results Was I right? Make a small change Guess

slide-36
SLIDE 36

Compute implications Compare with Nature Draw conclusion Observe nature Make a guess

Lightweight probes Ability to quickly deploy small changes Observable systems A brain

slide-37
SLIDE 37 37

practices that enable small bets* *not exhaustive

slide-38
SLIDE 38 38

Good Monitoring

slide-39
SLIDE 39 39

What do we mean when we say “monitoring”?

slide-40
SLIDE 40

What do we mean when we say “monitoring”?

Instrumentation Telemetry Visualisation Alerting Predictive alerting

Our code and hardware describes what it’s doing We have a way of gathering the data from our services We can meaningfully visualise our systems behaviour We can take action based on this information We can predict in advance when something may go wrong

credit: Dan North

slide-41
SLIDE 41 41

Service Service request latency downstream dependencies OS metrics health request tracing

slide-42
SLIDE 42

42

Codahale Metrics Dropwizard Tenacity Circuit Breakers Breaker Box Yammer

slide-43
SLIDE 43 43
slide-44
SLIDE 44 44
slide-45
SLIDE 45 45

Continuous Delivery

Safely and sustainably reduce lead time to value

slide-46
SLIDE 46 46
slide-47
SLIDE 47 47
slide-48
SLIDE 48 48

Compute implications Compare with Nature Draw conclusion Observe nature Make a guess

slide-49
SLIDE 49 49
slide-50
SLIDE 50 50
slide-51
SLIDE 51 51
slide-52
SLIDE 52 52

ApacheBench

ab -n 100 -c 10 “http://sciencesite.com/articles/9dfa1ae0-6ba5-4f2f-…”

Siege Vegeta

echo "GET http://sciencesite.com/“ | vegeta attack -duration=60s -rate=10

slide-53
SLIDE 53 53

Cloud native infrastructure as code

slide-54
SLIDE 54 54
slide-55
SLIDE 55

boto

slide-56
SLIDE 56

Compute implications Compare with Nature Draw conclusion Observe nature Make a guess

Lightweight probes Ability to quickly deploy small changes Good monitoring A brain

slide-57
SLIDE 57 57

Putting it all together

slide-58
SLIDE 58 58

Page Load (sec) 10 20 30 40 Initial observations 1 35

0.8 seconds for first byte 1.5 seconds for page load

slide-59
SLIDE 59 59
slide-60
SLIDE 60 60

Option: Add CPU Option: Cache S3 content Option: Cache Service content

slide-61
SLIDE 61 61

Compute implications Compare with Nature Draw conclusion Observe nature Make a guess XSLT can be expensive - maybe we are CPU bound Increasing the # of CPU’s could help

slide-62
SLIDE 62 62

Page load (sec) 10 20 30 40 After adding more CPU’s 1 2 35 6

slide-63
SLIDE 63 63
slide-64
SLIDE 64 64

Second Guess: Moving compute to the data

slide-65
SLIDE 65 65

Page load (sec) 10 20 30 40 Move closer to the data 1 2 3 35 6 4

slide-66
SLIDE 66 66

Third Guess: Transformations are slow and could be optimised

slide-67
SLIDE 67 67

Page load (sec) 10 20 30 40 Tune xslt transforms 1 2 3 4 35 6 4 3.5

slide-68
SLIDE 68 68

Final Guess: And, cache

slide-69
SLIDE 69 69

ESI / Templating

Computationally expensive service

  • ther service
  • ther service

XML in S3 Xslt

slide-70
SLIDE 70 70

Page load (sec) 10 20 30 40 After adding a cache 1 2 3 4 5 35 6 4 3.5 0.2

slide-71
SLIDE 71 71
slide-72
SLIDE 72 72

Do we add a cache? create another service to transform content

slide-73
SLIDE 73 73

saves the fetch from S3 saves the xslt transform time saves the fetch from S3 Cache in front of content service Cache in front of S3

slide-74
SLIDE 74 74

150.

Content trickles into the store. You keep up by listening for the new content and casting “wget” on the cache to keep it refreshed. New types of content appears - content the villagers have never seen

  • before. Content the walking skeleton

is unable to combat. Every time the structure of the content changes, the cache must be

  • refreshed. The cache grows and

grows until it is the size of the Great Wahui of Berathorob. It is no longer possible to refresh it. Latency increases.

link to close link hello close

You have died. Turn to page 1.

slide-75
SLIDE 75 75

Cache in front of S3 Cache in front of content service

slide-76
SLIDE 76 76

10.

The cache causes the content load times to drop from 300ms to 150ms. The villager says “this wonderful content is now arriving more swiftly than even the knight-messengers of the Empress”. The villagers are happy but all too soon, all is not well for the content has a long tail. You must work out how to refresh the content when it changes. You can either:

Refresh the content when it appears from the ether turn to 150 Trust that it will be fast enough on first view turn to 22

link to close link hello close

slide-77
SLIDE 77 77

4.

Your walking skeleton coalesces in a cloud of noxious gasses and solidifies as a java dropwizard application. You reach into your backpack and deploy the S3 content store. Your walking skeleton reaches out it’s skeletal arms and grabs armfuls of raw xml. Would you like to:

Transform the xml inside the skeleton turn to 6 Use a magic box turn to 5

slide-78
SLIDE 78 78

Real Options, Small Bets

slide-79
SLIDE 79 79

rapidly change all the things

Cloud Native, CD for infra

run experiments

performance testing doesn’t have to be heavy weight

  • bserve nature

instrumentation, telemetry, visualisation

slide-80
SLIDE 80 80

Evolutionary Architecture keeps our options open Continuous Delivery, lean product engineering and Infrastructure as Code reduce the amount of money we need to place a bet (so we can place more) Lightweight performance testing early allows us to make guesses and test

  • ur hypotheses
slide-81
SLIDE 81 81

Thanks

James Lewis : @boicy

slide-82
SLIDE 82 82

https://www.thoughtworks.com/radar

slide-83
SLIDE 83 83

Oh Hi!