Coor dinat ed Omission in NoSQ L Dat abase Benchmar king St ef f en - - PowerPoint PPT Presentation

▶
coor dinat ed omission
SMART_READER_LITE
LIVE PREVIEW

Coor dinat ed Omission in NoSQ L Dat abase Benchmar king St ef f en - - PowerPoint PPT Presentation

Sep 18, 2023 261 likes •583 views

Coor dinat ed Omission in NoSQ L Dat abase Benchmar king St ef f en Fr iedr ich, W o lf r am W inger at h, No r ber t R it t er U niver sit y o f Hambur g Depar t ment o f Inf o r mat ics Dat abases and Inf o r mat io n Syst ems M ar ch 6,

slide-1
SLIDE 1

Coor dinat ed Omission

in NoSQ L Dat abase Benchmar king

St ef f en Fr iedr ich, W o lf r am W inger at h, No r ber t R it t er

U niver sit y o f Hambur g Depar t ment o f Inf o r mat ics Dat abases and Inf o r mat io n Syst ems M ar ch 6, 2 0 1 7

slide-2
SLIDE 2

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2

NoSQ L Per f or mance Evaluat ion ?

Co o per et al.: Benchmar king Clo ud Ser ving Syst ems w it h YCSB, So CC‘1 0 , ACM , 2 0 1

slide-3
SLIDE 3

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 3

YCSBs load gener at ion

_targetOpsTickNanos = (long) (1 000 000 000 / target) long overallStartTime = System.nanoTime(); while (_opsdone < _opcount) { long startTime = System.nanoTime(); Status status = _db.read( table, key, fields, result ); long endTime = System.nanoTime(); _measurements.measure("READ", (int)( (endTime - startTime) / 1000)); _opsdone++; long deadline = overallStartTime + _opsdone * _targetOpsTickNanos; long now = System.nanoTime(); while((now = System.nanoTime()) < deadline) { LockSupport.parkNanos( deadline – now ); } }

slide-4
SLIDE 4

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 4

YCSBs load gener at ion

_targetOpsTickNanos = (long) (1 000 000 000 / targetThroughput) long overallStartTime = System.nanoTime(); while (_opsdone < _opcount) { long startTime = System.nanoTime(); Status status = _db.read( table, key, fields, result ); long endTime = System.nanoTime(); _measurements.measure("READ", (int)( (endTime - startTime) / 1000)); _opsdone++; long deadline = overallStartTime + _opsdone * _targetOpsTickNanos; long now = System.nanoTime(); while((now = System.nanoTime()) < deadline) { LockSupport.parkNanos( deadline – now ); } }

slide-5
SLIDE 5

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 5

YCSBs load gener at ion

_targetOpsTickNanos = (long) (1 000 000 000 / targetThroughput) long overallStartTime = System.nanoTime(); while (_opsdone < _opcount) { long startTime = System.nanoTime(); Status status = _db.read( table, key, fields, result ); long endTime = System.nanoTime(); _measurements.measure("READ", (int)( (endTime - startTime) / 1000)); _opsdone++; long deadline = overallStartTime + _opsdone * _targetOpsTickNanos; long now = System.nanoTime(); while((now = System.nanoTime()) < deadline) { LockSupport.parkNanos( deadline – now ); } }

W hat if lat ency > _ t ar get OpsTickNanos ? now > deadline ?

slide-6
SLIDE 6

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 6

The Coor dinat ed Omission Pr oblem

“a conspir acy w e’r e all a par t of ”

Gil Tene, Azul Syst ems: Ho w NOT t o M easur e Lat ency, Q Co n, 2 0 1 3 - 2 0 1 6 infoq.com/presentations/latency-response-time

slide-7
SLIDE 7

The Coor dinat ed Omission Pr oblem

7 Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king

E xample I

slide-8
SLIDE 8

The Coor dinat ed Omission Pr oblem

8 Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king

E xample II

slide-9
SLIDE 9

The Coor dinat ed Omission Pr oblem

9 Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king

E xample III

slide-10
SLIDE 10

The Coor dinat ed Omission Pr oblem

1 Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king

E xample III

Dat abase is able t o inf luence t he r eq uest r at e ! = > coor dinat ed omission of r elevant measur ement s

slide-11
SLIDE 11

The Coor dinat ed Omission Pr oblem

1 1 Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king

E xample

The R esult s:

slide-12
SLIDE 12

The Coor dinat ed Omission Pr oblem

1 2 Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king

R eal W o r ld GC E E vent ent Char t r t

ht t p:/ / t echblog.net f lix.com/ 2 0 1 3/ 0 5 / gar bage-collect ion-visualizat ion.ht ml

80 -1 2 0 seco nds GC pauses @ Net f lix Cassandr a Clust er 2 0 1 2

slide-13
SLIDE 13

The Coor dinat ed Omission Pr oblem

1 3 Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king

R eal W o r ld GC E E vent ent Char t r t

ht t p:/ / t echblog.net f lix.com/ 2 0 1 3/ 0 5 / gar bage-collect ion-visualizat ion.ht ml

Af t er f ixing t he pr o blem: max 5 seco nds GC pauses

slide-14
SLIDE 14

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 4

Closed VS. Open Syst em M odel

Clo sed Syst em M o del Open Syst em M o del

Schr ö der et al. Open Ver sus Clo sed: A Caut io nar y Tale, 2 0 0 6

slide-15
SLIDE 15

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 5

Closed VS. Open Syst em M odel

  • “M ost of t hese gener at or s/ benchmar ks assume a

closed syst em model”

  • “For many of t hese w or kload gener at or s, it

w as q uit e dif f icult t o f igur e out w hich syst em model w as being assumed” Schr ö der et al. sur veyed syst em models in var ious w eb r elat ed w or kload gener at or s:

  • “Pr inciple (i): For a given load, mean r esponse

t imes ar e signif icant ly low er in closed syst ems t han in open syst ems.”

slide-16
SLIDE 16

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 6

Coor dinat ed Omission Cor r ect ion

since YCSB Ver sio n 0 .2 .0 R C 1 , June 2 0 1 5

while (_opsdone < _opcount) { long startTime = System.nanoTime(); Status status = _db.read( table, key, fields, result ); long endTime = System.nanoTime(); _measurements.measure("INTENDED_READ", (int)( (endTime - _deadline) / 1000)); _opsdone++; _deadline = overallStartTime + _opsdone * _targetOpsTickNanos; long now = System.nanoTime(); while((now = System.nanoTime()) < deadline) { LockSupport.parkNanos( deadline – now ); } }

= > int ended measur ement int er val

…

slide-17
SLIDE 17

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 7

Coor dinat ed Omission Cor r ect ion

since YCSB Ver sio n 0 .2 .0 R C 1 , June 2 0 1 5

while (_opsdone < _opcount) { long startTime = System.nanoTime(); Status status = _db.read( table, key, fields, result ); long endTime = System.nanoTime(); _measurements.measure("INTENDED_READ", (int)( (endTime - _deadline) / 1000)); _opsdone++; _deadline = overallStartTime + _opsdone * _targetOpsTickNanos; long now = System.nanoTime(); while((now = System.nanoTime()) < deadline) { LockSupport.parkNanos( deadline – now ); } }

= > but st ill inf luences t he r eq uest r at e ! = > int ended measur ement int er val

slide-18
SLIDE 18

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 8

Scalable NoSQ L- Benchmar king

nosqlmark.informatik.uni-hamburg.de

slide-19
SLIDE 19

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 9

  • Scaling YCSB compat ible w or kloads t o mult iple

benchmar king nodes = > Aut omat ically aggr egat ed r esult s

  • Compat ible t o YCSB dat abase int er f ace layer
  • Closed and Open Syst em M odel

Scalable NoSQ L- Benchmar king

nosqlmark.informatik.uni-hamburg.de

slide-20
SLIDE 20

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2 0

Coor dinat ed Omission Avoidance

in No SQ LM ar k

implicit val ec = context.system.dispatchers.lookup("blocking-io-dispatcher") case DoOperation => { val operation = workload.nextOperation val startTime = System.nanoTime val future = Future { sendRequest(operation) } future.onComplete { case Success(status) => { val endTime = System.nanoTime measurementActor ! Measure(operation.name, (endTime - startTime) / 1000) } case Failure(ex) => { log.error(ex, "Error occured during operation {}", operation.name) } }

asynchr onous load gener at ion !

…

slide-21
SLIDE 21

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2 1

Coor dinat ed Omission Validat ion w it h

Or iginally developed t o validat e consit ency measur ement appr oaches

  • consist ent single-node backend
  • mult i-node behaviour
  • t unable st aleness
  • globaly consist ent logf ile

W inger at h, Fr iedr ich, Gesser , R it t er : W ho W at ches t he W at chmen? BTW 2 0 1 5

Single-node inconsist ent key-value St or e

github.com/steffenfriedrich/SickStore

slide-22
SLIDE 22

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2 2

Coor dinat ed Omission Validat ion w it h

New Feat ur e: Simulat ion of maximum t hr oughput and dat abase hiccups 1 . comput e t heor et ical w ait ing t ime Ti of r eq uest i in t he dat abse syst em 2 . calling client t hr ead has t o sleep f or Ti

Single-node inconsist ent key-value St or e

slide-23
SLIDE 23

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2 3

Exper iment al Validat ion

  • n:

: Sick SickSt or e

Benchmar k: 90 0 0 0 o ps, t ar get = 1 0 0 0 o ps/ sec, SickSt o r e: 1 seco nd hiccup, max t hr o ughput = 1 2 5 0 o ps/ sec, 80 % o f max t hr o ughput

YCSB NoSQ LM ar k YCSB Int ended

AVG.: 2 ms 2 9 ms 1 80 ms

slide-24
SLIDE 24

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2 4

Exper iment al Validat ion

  • n:

: Sick SickSt or e

Benchmar k: 90 0 0 0 o ps, t ar get = 1 0 0 0 o ps/ sec, SickSt o r e: 1 seco nd hiccup, max t hr o ughput = 1 2 5 0 o ps/ sec,

YCSB NoSQ LM ar k YCSB Int ended

AVG.: 6 ms 2 9 ms 4 9ms

slide-25
SLIDE 25

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2 5

Exper iment al Validat ion

  • n:

: Sick SickSt or e

Benchmar k: 90 0 0 0 o ps, t ar get = 1 0 0 0 o ps/ sec, SickSt o r e: 1 seco nd hiccup, max t hr o ughput = 1 2 5 0 o ps/ sec,

YCSB NoSQ LM ar k YCSB Int ended

AVG.: 1 9 ms 2 9 ms 4 4 ms

YCSB NoSQ LM ar k YCSB Int ended

AVG.: 4 9 ms 2 9 ms 5 4 ms

slide-26
SLIDE 26

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2 6

Exper iment al Validat ion

  • n:

: Sick SickSt or e

Dif f er r ent max t hr o ughput s

~ 70% load ~ 10% load ~ 95% load ~ 90% load

slide-27
SLIDE 27

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2 7

Elast icit y Benchmar k w it h Cassandr a

  • ne Cassandr a node loaded w it h 1

0 million r ecor ds

  • af t er 5 min add second node

= > it st ar t s ser ving af t er ~ 5 min = > r oughly t he t ime it t akes lat ency t o st abilize

  • r un each exper iment f or max 1

5 min on a f r esh Clust er

Kuhlenkamp et al.: Benchmar king Scalabilit y and E last icit y o f Dist r ibut ed Dat abase Syst ems, VLDB, 2 0 1 4

YCSB w it ho ut int ended measur ement int er val

slide-28
SLIDE 28

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2 8

Elast icit y Benchmar k w it h Cassandr a

t ar get t hr o ughput = 1 0 0 0 0 o ps / sec

slide-29
SLIDE 29

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2 9

Elast icit y Benchmar k w it h Cassandr a

YCSB: 4 8 t hr eads

slide-30
SLIDE 30

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 30

Summar y

Benchmar king is har d and your lat ency values ar e pr obably lying t o you !

Be aw ar e of t he syst em model under lying your load gener at or s.

slide-31
SLIDE 31

Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 31

Fur t her Impr ovement in NoSQ LM ar k

M or e r ealist ic dist r ibut ions f or r eq uest r at e

  • Expo nent ial int er -r eq uest / ar r ival t ime = > po isso n

dist r ibut ed r eq uest / ar r ival r at e

  • So me aut ho r s co nsider Per at o o r hyper -expo nent ial

dist r ibut ed int er -ar r ival t ime

James F. Br ady & Neil J. Gunt her : Ho w t o E mulat e W eb Tr af f ic U sing St andar d Lo ad Test ing To o ls, Co RR , 2 0 1 6 perfdynamics.blogspot.de/2010/05/load-testing-think-time-distributions.html Neil J. Gunt her : Lo ad Te Test st ing Th Think Ti Time Dist r ibut io n

  • ns, b

blo gpo s po st , 2 2 0 1 1