Java One 2015 Deep Dive T op Performance Mistakes And other Tips - PowerPoint PPT Presentation

Java One 2015 – Deep Dive T op Performance Mistakes And other Tips & T ricks to make you a “Performance Expert” More on http://blog.dynatrace.com Andreas Grabner - @grabnerandi

Safe Harbor 

AND MANY MORE

0.01ms 0.02ms

15 Years: That’s why I ended up talking about performance

Where do your Stories come from?

#1: Real Life & Real User Stories

#2: http://bit.ly/onlineperfclinic

#3: http://bit.ly/sharepurepath

80% 20%

Frontend Performance We are getting FATer!

Example of a “Bad” Web Deployment 282! Objects 282! Objects 9.68MB Page Size on that page on that page 9.68MB Page Size 8.8s Page Load 8.8s Page Load Time Time Most objects are images Most objects are images delivered from your main delivered from your main domain domain Very long Connect tjme Very long Connect tjme (1.8s) to your CDN (1.8s) to your CDN

Mobile landing page of Super Bowl ad Total size of ~ Total size of ~ 20MB 20MB 434 Resources in total on that page: 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … 230 JPEGs, 75 PNGs, 50 GIFs, …

Fifa.com during Worldcup Source: htup://apmblog.compuware.com/2014/05/21/is-the-fjfa-world-cup-website-ready-for-the-tournament/

8MB of background image for STPCon (Word Press)

Make F12 or Browser Agent your friend!

Compare yourself Online!

Key Metrics # of Resources Size of Resources Total Size of Content

T ooling • Browser Built-In Developer Tools • Extensions such as YSlow, PageSpeed • Online Tools WebPageTest • Google PageSpeed Insights • Dynatrace Performance Center • ... • • Automate!! With Selenium, WebDriver, Cucumber, ...

Frontend Availability Back to Basics Please!

Online Services for you: Is it down right now?

Online Services for you: Outage Analyzer

Tip for handling Spike Load: GO LEAN!! Response tjme improved 4x Response tjme improved 4x 1h afuer 1h before 1h afuer 1h before Game ended SuperBowl KickOf Game ended SuperBowl KickOf

Key Metrics HTTP 3xx, 4xx, 5xx # of Domains

Online Services • Dynatrace Synthetic • Ruxit Synthetic • NewRelic Synthetic • AppDynamics • PingDom • ... Just Google for „Synthetic Monitoring“

Backend Performance The Usual Suspects

Project: Online Room Reservation System • Symptoms HTML takes between 60 and 120s to render • High GC Time • • Developer Assumptions Bad GC Tuning • Probably bad Database Performance as rendering was simple • • Result : 2 Years of Finger pointing between Dev and DBA

Developers built own monitoring void roomreservationReport(int officeId) { long startTime = System.currentTimeMillis(); Object data = loadDataForOffice(officeId); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, officeId); } Result : DB Tool says : Avg. Data Load Time: 45s! Avg. SQL Query: <1ms!

#1: Loading too much data 24889! Calls to the Database 24889! Calls to the Database API! API! High CPU and High Memory Usage High CPU and High Memory Usage to keep all data in Memory to keep all data in Memory

12444! 12444! #2: On individual connections individual individual connectjons connectjons Individual SQL Individual SQL really <1ms really <1ms Classical N+1 Query Classical N+1 Query Problem Problem

#3: Putting all data in temp Hashtable Lots of tjme spent in Lots of tjme spent in Hashtable.get Hashtable.get Called from their Entjty Called from their Entjty Objects Objects

Lessons Learned – Don’t Assume … • … you know what code is doing you inherited!! • … you are not making mistakes like this  • Explore the Right Tools • Built-In Database Analysis Tools • “Logging” options of Frameworks such as Hibernate, … • JMX, Perf Counters, … of your Application Servers • Performance Tracing Tools: Dynatrace, Ruxit, NewRelic, AppDynamics, Your Profjler of Choice …

Key Metrics # of SQL Calls # of same SQL Execs (1+N) # of Connectjons Rows/Data Transferred

Backend Performance Architectural Mistakes with „Migrating“ to (Micro)Services

Architecture Violatjon: Direct access to DB instead Architecture Violatjon: Direct access to DB instead from frontend logic from frontend logic 26.7s Executjon 26.7s Executjon Time Time 171! SQL Queries through LINQ 171! SQL Queries through LINQ 33! Calls to the same 33! Calls to the same by this Web Service – request by this Web Service – request Web Service Web Service similar data for each call similar data for each call

33! Diferent 33! Diferent connectjons used connectjons used 3136! Calls to H2 3136! Calls to H2 mostly executed on mostly executed on async background async background threads threads DB Exceptjons on both DB Exceptjons on both DB Exceptjons on both DB Exceptjons on both Databases Databases Databases Databases 40! internal Web 40! internal Web Service Calls that Service Calls that do all these DB do all these DB 21671! Calls to Oracle Updates 21671! Calls to Oracle Updates

Key Metrics # of Service Calls Payload of Service Calls # of Involved Threads 1+N Service Call Patuern!

T ooling • Dynatrace • Ruxit • NewRelic • AppDynamics • Any Profjler that can trace across tiers • Google for Tracing or APM (Application Performance Management)

Logging WE CAN LOG THIS!! LOG

Log Hotspots in Frameworks! callAppenders clear CPU and I/O Hotspot Excessive logging through Spring Framework

Debug Log and outdated log4j library #1: Top Problem: log4j.callAppenders #1: Top Problem: log4j.callAppenders -> 71% Sync Time -> 71% Sync Time #3: Doing “DEBUG” log #3: Doing “DEBUG” log #2: Most of logging done from #2: Most of logging done from output: Is this necessary? output: Is this necessary? fjllDetail method fjllDetail method

Key Metrics # of Log Entries Size of Logs per Use Case

Response Time is not the only Performance Indicator Look at Resources as well

Is this a successful new Build?

Look at Resource Usage: CPU, Memory, …

Memory? Look at Heap Generations

Root Cause: Dependency Injection

Prevent: Monitor Memory Metrics for every Build

#5: Throughput #5: Throughput #1: Eden Space stays constant. #1: Eden Space stays constant. of Applicatjon of Applicatjon Objects being propagated to Objects being propagated to goes to 0 due to goes to 0 due to Survivor Space Survivor Space no memory no memory available available #2: GC Actjvity in Young #2: GC Actjvity in Young #4: Heavy GC Generatjon ultjmately #4: Heavy GC Generatjon ultjmately kicks in when moves objects into Old kicks in when moves objects into Old Old Generatjon Old Generatjon Generatjon is Generatjon is full! full! #3: Growing #3: Growing “Old Gen” is a “Old Gen” is a good indicator good indicator for a Mem Leak for a Mem Leak

Key Metrics # of Objects per Generatjon # of GC Runs Total Impact of GC

Tips & Tricks And more Metrics of course 

Tip: Layer Breakdown over Time With increasing load: Which LAYER With increasing load: Which LAYER doesn’t SCALE ? doesn’t SCALE ?

Tip: Exceptions and Log Messages How are # of EXCEPTIONS How are # of EXCEPTIONS evolving over tjme? evolving over tjme? How many SEVERE LOG How many SEVERE LOG messages to we write in messages to we write in relatjon to Exceptjons? relatjon to Exceptjons?

Tip: Failed Transactions Are more TRANSACTIONS Are more TRANSACTIONS FAILING (HTTP 5xx, 4xx, …) FAILING (HTTP 5xx, 4xx, …) under heavier load? under heavier load?

Tip: Database Activity Do we see increased in AVG # Do we see increased in AVG # of SQL Executjons over Time? of SQL Executjons over Time? Do TOTAL # of SQL Executjons Do TOTAL # of SQL Executjons increase with load? Shouldn’t increase with load? Shouldn’t it fmatuen due to CACHES ? it fmatuen due to CACHES ?

Tip: Database History Dashboard How many SQL Statements are How many SQL Statements are PREPARED ? PREPARED ? What’s the overall Executjon What’s the overall Executjon Time of diferent SQL Types Time of diferent SQL Types ( SELECT, INSERT, DELETE, …) ( SELECT, INSERT, DELETE, …)

Tip: DB Connection Pool Utilization Do we have enough DB Do we have enough DB CONNECTIONS per pool? CONNECTIONS per pool?

For more Key Metrics htup://blog.dynatrace.com htup://blog.ruxit.com

We want to get from here …

T o here!

Use these applicatjon metrics as additjonal Quality Gates

Quality Metrics What you currently measure # Test Failures in your CI Overall Duration What you should measure Execution Time per test # calls to API # executed SQL statements # Web Service Calls # JMS Messages # Objects Allocated # Exceptions # Log Messages # HTTP 4xx/5xx Request/Response Size Page Load/Rendering Time … 71