QuickCheck John Hughes DEMO Registry tests on multiple nodes - - PowerPoint PPT Presentation

▶
quickcheck
SMART_READER_LITE
LIVE PREVIEW

QuickCheck John Hughes DEMO Registry tests on multiple nodes - - PowerPoint PPT Presentation

Jan 06, 2023 989 likes •1.29k views

QuickCheck John Hughes DEMO Registry tests on multiple nodes 40000 35000 30000 Tests per minute 25000 20000 15000 10000 5000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Nodes Four core/8HT SkyLake i7

slide-1
SLIDE 1

QuickCheck

John Hughes

slide-2
SLIDE 2

DEMO

slide-3
SLIDE 3

5000 10000 15000 20000 25000 30000 35000 40000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Tests per minute Nodes

Registry tests on multiple nodes

Four core/8HT SkyLake i7 Dual core/4HT Ivy Bridge i7

slide-4
SLIDE 4

Architecture

Master Worker Worker Worker Worker

Generate and run tests Count tests, print dots, start and stop

slide-5
SLIDE 5

How can the Master count the tests?

Master Worker Worker Worker Worker

slide-6
SLIDE 6

How can the Master stop the workers?

Master Worker Worker Worker Worker

slide-7
SLIDE 7

Benchmark

Bouncer Mirror 1.2 million/second

slide-8
SLIDE 8

Multiple bouncers

Bouncer Mirror Bouncer Mirror Bouncer Mirror

slide-9
SLIDE 9

500000 1000000 1500000 2000000 2500000 3000000 3500000 4000000 4500000

1 2 3 4 5 6 7 8 9 10

Round trips per second

P--number of bouncers

Message round-trips per second Four core/8 thread i7

P bouncers, P mirrors

slide-10
SLIDE 10

Mirror

Multiple bouncers, one mirror

Bouncer Bouncer Bouncer

slide-11
SLIDE 11

500000 1000000 1500000 2000000 2500000 3000000 3500000 4000000 4500000

1 2 3 4 5 6 7 8 9 10

Round trips per second

P--number of bouncers

Message round-trips per second Four core/8 thread i7

P bouncers, P mirrors P bouncers, 1 mirror

slide-12
SLIDE 12

Batch permissions?

Master Worker Worker Worker Worker BUT this may delay termination!

slide-13
SLIDE 13

Alternative Architecture

Master Manager Manager Manager Manager Worker Worker Worker Worker

Another? Yes

slide-14
SLIDE 14
  • Every worker

communicates with its

  • wn manager—

scalable!

  • Stopping can be

slightly delayed

slide-15
SLIDE 15

> eqc:quickcheck(examples:prop_reverse()). .................................................................................................... OK, passed 100 tests true > eqc:quickcheck(examples:prop_reverse()). ....................................................................................................(x10).(x1)......... OK, passed 119 tests true > eqc:quickcheck(eqc:in_parallel(examples:prop_reverse())). ....................................................................................................(x10)..(x1)...... OK, passed 126 tests true > eqc:quickcheck(eqc:on_nodes(examples:prop_reverse())). ....................................................................................................(x10)................................ ........................(x1)......... OK, passed 669 tests

But how bad is it to run a few extra tests?

slide-16
SLIDE 16

What about node placement?

Master Manager Manager Manager Manager Worker Worker Worker Worker

Another? Yes

slide-17
SLIDE 17

What about node placement?

Master Manager Manager Manager Manager Worker Worker Worker Worker

slide-18
SLIDE 18

What about node placement?

Master Manager Manager Manager Manager Worker Worker Worker Worker

Another? Yes

slide-19
SLIDE 19

20000 40000 60000 80000 100000 120000 140000 160000 1 2 3 4 5 6 7 8 9 10

P bouncers, 1 mirror, different nodes Four core/8 HT i7

slide-20
SLIDE 20

200000 400000 600000 800000 1000000 1200000 1400000 1600000 1 2 3 4 5 6 7 8 9 10

P bouncers, 1 mirror Four core/8HT i7

Different nodes Same node

10-30x slower from a different node!

slide-21
SLIDE 21

5000 10000 15000 20000 25000 30000 5 10 15 20 25 30 35

P bouncers, 1 mirror Bouncers on dual core laptop, mirror on quad core

slide-22
SLIDE 22

200000 400000 600000 800000 1000000 1200000 1400000 Same LAN Same Host Same Node

Bounces per second (single bouncer)

slide-23
SLIDE 23

1 2 3 4 5 6 7 Same LAN Same Host Same Node Log scale

Bounces per second (single bouncer)

50x 30x

slide-24
SLIDE 24

What about success messages?

Master Manager Manager Manager Manager Worker Worker Worker Worker

Success

slide-25
SLIDE 25

Two-way vs one-way

Bouncer Mirror 1.2 million/second 5.4 million/second

slide-26
SLIDE 26

1 2 3 4 5 6 7 8 Same LAN Same Host Same Node

Log scale

Bounces/Messages per second

Two way One way

4x 10x 100x

slide-27
SLIDE 27

One more optimization…

Master Manager Manager Manager Manager Worker Worker Worker Worker

Success

There are a LOT of success messages Count them! Send the total every 100ms

slide-28
SLIDE 28
  • Tripled the speed of

quickcheck(true)!

  • Stopping can be even

more delayed

slide-29
SLIDE 29

Lessons

  • There is at least an order of magnitude difference

between communication costs

  • Within a node
  • Between nodes
  • Between hosts
  • Latency is much worse affected than bandwidth
  • This affects design for performance
  • Favours asynchronous over synchronous communication

between nodes

  • Optimising performance may require changes to
  • bservable behaviour
  • …and we didn’t even consider fault tolerance