Performance Chall llenges along the Conti tinuous Deli livery - - PowerPoint PPT Presentation

▶
performance chall llenges along the conti tinuous deli
SMART_READER_LITE
LIVE PREVIEW

Performance Chall llenges along the Conti tinuous Deli livery - - PowerPoint PPT Presentation

Feb 07, 2024 46 likes •460 views

Performance Chall llenges along the Conti tinuous Deli livery Pipelin ine Wolfgang Gottesheim @gottesheim 1 #Dynatrace Example of a Bad Web Deployment 282! Objects 9.68MB Page Size on that page 8.8s Page Load Time Most objects are

slide-1
SLIDE 1 1 #Dynatrace

Wolfgang Gottesheim @gottesheim

Performance Chall llenges along the Conti tinuous Deli livery Pipelin ine

slide-2
SLIDE 2 2 #Dynatrace

Example of a “Bad” Web Deployment

282! Objects

  • n that page

9.68MB Page Size

8.8s Page

Load Time Most objects are images delivered from your main domain Very long Connect time (1.8s) to your CDN

slide-3
SLIDE 3 3 #Dynatrace

Application problems in the wild

526s to render a financial transaction report

1 SQL running

210s!

Debug Logging with log4j on outdate ted log4j library (sync issue)

slide-4
SLIDE 4 4 #Dynatrace

But now we‘re doing Continuous

Delivery

slide-5
SLIDE 5 5 #Dynatrace

700 Deployments / Year 50-60 Deployments / Day 10+ Deployments / Day Every 11.6 seconds

slide-6
SLIDE 6 6 #Dynatrace

It‘s not about the pipeline…

(only)

slide-7
SLIDE 7 7 #Dynatrace

Commit Stage Acceptance Testing Release Developer s Capacity Testing UAT

slide-8
SLIDE 8 8 #Dynatrace

Metrics

to the rescue!

slide-9
SLIDE 9 9 #Dynatrace

We want to replace magic with data!

Commit Stage Acceptance Testing Release Capacity Testing UAT

slide-10
SLIDE 10 10 #Dynatrace

4

examples

slide-11
SLIDE 11 11 #Dynatrace

#1 #1

slide-12
SLIDE 12 12 #Dynatrace

Mobile Landing Page of Super Bowl Ad

434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB

slide-13
SLIDE 13 13 #Dynatrace

8MB background image for STPCon

slide-14
SLIDE 14 14 #Dynatrace

Key Metrics

# Resources

Size of Resources Page Size

slide-15
SLIDE 15 15 #Dynatrace

#2 #2

slide-16
SLIDE 16 16 #Dynatrace

#1: Loading too much data

24889! Calls to the Database API! High CPU and High Memory Usage to keep all data in Memory

slide-17
SLIDE 17 17 #Dynatrace

#2: On individual connections

Classical N+1 Query Problem Individual SQL really <1ms 12444! individual connections (and no

pooling…)

slide-18
SLIDE 18 18 #Dynatrace

#3: Homegrown Caching Framework

Lots of time spent in Hashtable.get Called from their Entity Objects

slide-19
SLIDE 19 19 #Dynatrace

Key Metrics

# Functional Errors

# of SQL Calls # of same SQL Execs (1+N) # of Connections Rows/Data Transferred

slide-20
SLIDE 20 20 #Dynatrace

#3 #3

Si Simply spli litting web servic ice call lls is is not t a mig igratio ion to a mic icroservice architecture

slide-21
SLIDE 21 21 #Dynatrace

3136! Calls to H2 mostly executed on async background threads 40! internal Web Service Calls that do all these DB Updates 21671! Calls to Oracle DB Exceptions on both Databases DB Exceptions on both Databases 33! Different connections used

slide-22
SLIDE 22 22 #Dynatrace

Key Metrics

# Service Calls

# of Threads Timing (execution, wait) # SQL executions # of SAME SQLs

slide-23
SLIDE 23 23 #Dynatrace

#4 #4

slide-24
SLIDE 24 24 #Dynatrace

Is this a successful new Build?

slide-25
SLIDE 25 25 #Dynatrace

Look at Resource Usage: CPU, Memory, …

slide-26
SLIDE 26 26 #Dynatrace

Memory? Look at Heap Generations

slide-27
SLIDE 27 27 #Dynatrace

Root Cause: Dependency Injection

slide-28
SLIDE 28 28 #Dynatrace

Prevent: Monitor Memory Metrics for every Build

slide-29
SLIDE 29 29 #Dynatrace

Key Metrics

# of Objects per Generation

# of GC Runs Total Impact of GC

slide-30
SLIDE 30 30 #Dynatrace
  • 1. # Resources
  • 2. Size of Resources
  • 3. Page Size
  • 4. # Functional

Errors

  • 5. # SQL Executions
  • 6. # of SAME SQLs

Where to sta tart?

slide-31
SLIDE 31 31 #Dynatrace

P A M I ick gree easure ntegrat e

Allow me to introduce…

slide-32
SLIDE 32 32 #Dynatrace

AND MANY MORE Kieker

slide-33
SLIDE 33 33 #Dynatrace

We successfully replaced magic with data!

Commit Stage Acceptance Testing Release Capacity Testing UAT

slide-34
SLIDE 34 34 #Dynatrace

Just one more thing…

slide-35
SLIDE 35 35 #Dynatrace

It‘s al also about the pipeline…

 Define pipeline metrics!

slide-36
SLIDE 36 36 #Dynatrace

Connect your Tests with Quality

12 120ms 3 1 68ms Build 20 testPurchase OK testSearch OK Build 17 testPurchase OK testSearch OK Build 18 testPurchase FAILED testSearch OK Build 19 testPurchase OK testSearch OK Build # Test Case Status # SQL # Excep CPU 12 120ms 3 1 68ms 12 5 60ms 3 1 68ms 75 230ms 3 1 68ms Test Framework Results Architectural Data

We identified a regresesion Problem solved

Exceptions probably reason for failed tests Problem fixed but now we have an architectural regression Problem fixed but now we have an architectural regression

Now we have the functional and architectural confidence Let’s look behind the scenes

slide-37
SLIDE 37 37 #Dynatrace

#1: Analyzing each Test #2: Metrics for each Test #3: Detecting Regression based on Measure

slide-38
SLIDE 38 38 #Dynatrace

Quality-Metrics based Build Status

slide-39
SLIDE 39 39 #Dynatrace

Pull data into Jenkins, Bamboo ...

slide-40
SLIDE 40 40 #Dynatrace

Make Quality a first-class citizen

„Too hard“ „we‘ll get round to this later“ „not cool enough“

slide-41
SLIDE 41 41 @Dynatrace

Wolfgang Gottesheim

Free Tools: http://bit.ly/dttrial Twitter @gottesheim wolfgang.gottesheim@dynatrace.com