Performance tuning for Java applications George Barnett, Atlassian - - PowerPoint PPT Presentation

Performance tuning for Java applications

George Barnett, Atlassian

Friday, 11 March 2011

• How to make your apps run faster
• A few process tips
• Tuning ideas
• What doesn’t work

Topics

Friday, 11 March 2011

• X vs Y will always depend on the context
• Test with your application!

A Note

Friday, 11 March 2011

• Isolate performance issues before they become

headaches in production.

• Make sure these issues get fixed.

Performance Engineering

Friday, 11 March 2011

CODE! CODE! CODE!

Unit & Functional tests

Software artefact

Reporting

commit tests pass

Simple code lifecycle

Friday, 11 March 2011

Performance Testing

Reporting

• Catch regressions on commit
• Create a standard benchmark
• Performance testing will save your Ops team

time

Add to your testing

Friday, 11 March 2011

Getting started

• Define test cases and traffjc levels
• Examine logs and monitoring data
• Use real world data
• Eg, Public Atlassian instances such as

http://jira.atlassian.com

Friday, 11 March 2011

• Apache JMeter
• Howto:
• http://blogs.atlassian.com/developer/2008/10/

performance_testing_with_jmete.html

• Example
• http://confluence.atlassian.com/display/DOC/

Performance+Testing+Scripts

Some useful tools

Friday, 11 March 2011

• Automation
• Maven
• Bamboo
• Getting a repeatable build is critical!
• Automated Performance Testing will save your

Dev team time!

Some useful tools

Friday, 11 March 2011

How often?

• Daily performance tests
• Most products & libraries
• Weekly soak tests
• Run for much longer than daily tests

Friday, 11 March 2011

Reporting

• Put there data where you cant miss it
• Build screens, dashboards, notifications
• JMeter Aggregator Plugin

Friday, 11 March 2011

Reporting

Friday, 11 March 2011

Reporting

Friday, 11 March 2011

You are here

• How I did the testing
• Hardware & Software
• Must have improvements
• Java, Hardware, Virtualization and Garbage Collection
• Worth doing
• Gigabit vs 100mbit, Heap size tuning, use an SSD
• Sadly, not useful
• Tune MySQL, Replace MySQL

Friday, 11 March 2011

The contenders

• Dell PE 850 - “WALL-E”
• Xeon X3220 2.4Ghz Quad Core
• Quad Core, 2 x 4Mb L3
• 8Gb DDR2 RAM
• 2 x 7.2K SAS Disks
• 1 x Intel X25-E 32GB SSD
• ~$1500

Friday, 11 March 2011

• Dell PE 2950 - “Johnny 5”
• 2 x Xeon E5405 2.0Ghz
• Quad Core, 2 x 6M L3
• 32Gb RAM
• 2 x 72Gb 15K Drives
• ~$4000

The contenders

Friday, 11 March 2011

The contenders

• Dell PE R610 - “EVE”
• 2 x Xeon E5520 2.2Ghz
• Quad Core, 8M “SmartCache”
• 32Gb RAM
• 2 x 146G 15K drives
• ~$4000

Friday, 11 March 2011

The Workload

• JIRA 4, MySQL 5, RHEL 5
• Java 1.6, 3G heap, ParNew, ParOld GC
• ~70,000 Issues, ~80 projects
• ~30 reqs/sec, JMeter 2.3.4 (patched, JSR-223)
• Traffjc modelled on http://jira.atlassian.com with

higher writes

Friday, 11 March 2011

About the graphs

• Most show Average Response Time in

Milliseconds

• Includes time to generate AND deliver the response
• Does not include any browser time
• Arrows show if lower or higher is better!

Friday, 11 March 2011

You are here

• How I did the testing
• Hardware & Software
• Must have improvements
• Java, Hardware, Virtualization and Garbage Collection
• Worth doing
• Gigabit vs 100mbit, Heap size tuning, use an SSD
• Sadly, not useful
• Tune MySQL, Replace MySQL

Friday, 11 March 2011

• Must. Have.

Friday, 11 March 2011

Upgrade Java 1.5 to 1.6

• Advances in compiler / VM technology
• Produces better runtime code
• Able to use modern instructions
• Able to use modern hardware

Friday, 11 March 2011

Java 1.5 vs 1.6 (Johnny 5)

375 750 1125 1500 Average Response Time (ms)

Java 1.5 Java 1.6.0

6.3X

Friday, 11 March 2011

Java 1.5 vs 1.6 (Johnny 5)

95 190 285 380 Average CPU (%)

26%

Java 1.5.0 Java 1.6.0

Friday, 11 March 2011

Upgrade Java to 1.6

• HotSpot VM updates in 1.6
• Advancing technology
• Compressed OOPS on 64bit
• Escape Analysis
• NUMA
• http://java.sun.com/performance/reference/

whitepapers/6_performance.html

Friday, 11 March 2011

Java 1.6 vs 1.6 (Eve)

37 74 111 148 Average Response Time (ms)

Java 1.6.0_7 Java 1.6.0_16

7%

Friday, 11 March 2011

Hardware Upgrades

• Hyper-Threading
• QuickPath Interconnect
• Nehalem (Eve) have on die DDR3 Memory Controllers
• Memory throughput improvements - good for Java!

Technology Speed

DDR2-800 (dual channel) 6.4 GB/sec DDR3-1333 (dual channel) 21.3 GB/sec

Friday, 11 March 2011

• Shared bus
• 1333 MT/s = 1333M Transfers / sec
• 4 transfers/clock = 266Mhz.

CPU

North Bridge

RAM

I/O: SATA, USB, etc

FSB

CPU

South Bridge

FSB architecture

Friday, 11 March 2011

Quick Path Interconnect

• 3.2Ghz, 6.4 GT/sec
• 25.6 GB/sec

CPU

QPI

CPU

RAM RAM

IOH

Friday, 11 March 2011

Hyper Threading

CPU

Thread 1 Thread 2

Thread 1 runs until it hits a cache miss. Thread 2 runs on the same CPU while T1 is awaiting data

Friday, 11 March 2011

Hardware Upgrades

500 1000 1500 2000 Average Response Time (ms)

2.4Ghz Xeon (Wall-E) 2.0Ghz Xeon (Johnny 5) 2.2Ghz Xeon (Eve)

8-12X

Friday, 11 March 2011

Improved GC Throughput

7500 15000 22500 30000 GC Throughput (MB/sec)

2.4Ghz Xeon (Wall-E) 2.0Ghz Xeon (Johnny 5) 2.2Ghz Xeon (Eve)

2-3X

Friday, 11 March 2011

Garbage Collection

• Java’s Virtual Machine manages memory
• New objects are continually created during

runtime

• GC is the process of collecting dead objects to

reclaim memory

Friday, 11 March 2011

Tune Garbage Collection

• Getting your GC tuning wrong is bad
• How bad?
• Are the defaults sane?

Friday, 11 March 2011

Tune Garbage Collection

1000 2000 3000 4000 Wall-E (ms) Johnny 5 (ms) Eve (ms)

CMS ParNew ParOld

Average Response Time

2X

Friday, 11 March 2011

Tune Garbage Collection

125 250 375 500 Johnny 5 (ms) Eve (ms)

CMS ParNew ParOld

Average Response Time

2X

Friday, 11 March 2011

Tune Garbage Collection

• Defaults are sane
• Parallel New + Parallel Old is fastest
• CMS prioritises low pauses over CPU usage
• CMS does not compact - can lead to heap

fragmentation

Friday, 11 March 2011

Virtualisation

• Does being in a VM hurt?
• What if the risks are managed?

Friday, 11 March 2011

Virtualisation

42.5 85.0 127.5 170.0 Average Response Time (ms)

Real VMWare ESX 3.5 VMWare ESX 4i

43%

Friday, 11 March 2011

Virtualisation

• On average ~30% slower
• Under extreme resource starvation, up to 100%

slower

• Avoid virtualisation for high traffjc instances
• See the Atlassian docs for recommendations
• http://confluence.atlassian.com/display/DOC/Running+Confluence+in+a+Virtualised

+Environment

• http://confluence.atlassian.com/display/JIRA/Running+JIRA+in+a+Virtualised+Environment

Friday, 11 March 2011

You are here

• How I did the testing
• Hardware & Software
• Must have improvements
• Java, Hardware, Virtulization and Garbage Collection
• Worth doing
• Gigabit vs 100mbit, Heap size tuning, use an SSD
• Sadly, not useful
• Tune MySQL, Replace MySQL

Friday, 11 March 2011

Maybe.

Friday, 11 March 2011

Gigabit Network vs 100Mbit

• Gigabit is faster, sure
• But 100Mb is OK if you’re not doing big

transfers, RIGHT?!

Friday, 11 March 2011

Use Gigabit

475 950 1425 1900 Average Response Time (ms)

100mb/sec 1000mb/sec

10%

Friday, 11 March 2011

Use Gigabit

• Lower RTT on Gigabit
• If you have enough CPU power, keep the DB on

localhost

• Check your development environment

Friday, 11 March 2011

Use Gigabit

50 100 150 200 ReIndex Time (seconds)

100Mbit 1000Mbit

26%

Friday, 11 March 2011

Heap size tuning

• Memory starvation is bad, but can we “drown”

java with too much memory?

Friday, 11 March 2011

Heap size - Wall-E

500 1000 1500 2000 Average Response Time

1.5G 3G 6G

23%

Friday, 11 March 2011

Heap size - Johnny 5

72.5 145.0 217.5 290.0 Average Response Time

1.5G 3G 6G 15G

49%

Friday, 11 March 2011

Heap size - Eve

42.5 85.0 127.5 170.0 Average Response Time

1.5G 3G 6G 15G

28%

Friday, 11 March 2011

Heap size tuning

• Heap too small - bad
• Heap too big - not bad!
• Bigger heaps give the JVM more room to work
• Diminishing returns from really big heaps

Friday, 11 March 2011

Buy an SSD

• SSDs are FAST!
• SSDs are EXPENSIVE!
• SSDs are all the rage!
• DBAs are finally happy (almost)

Friday, 11 March 2011

Buy an SSD

200 400 600 800 Browse Issue Create Issue

SATA SSD

33%

Friday, 11 March 2011

Buy an SSD

• Fast writes
• 0.1ms Avg Service Time
• Most improvement comes from SSD on the DB
• Writing to the Lucene index means re-opening

IndexReaders

• Much faster on SSD
• Lucene Search Term Cache fits in OS bufgers

Friday, 11 March 2011

You are here

• How I did the testing
• Hardware & Software
• Must have improvements
• Java, Hardware, Virtulization and Garbage Collection
• Worth doing
• Gigabit vs 100mbit, Heap size tuning, use an SSD
• Sadly, not useful
• Tune MySQL, Replace MySQL

Friday, 11 March 2011

Meh.

Friday, 11 March 2011

Replace MySQL with MySQL

• MySQL 5.0.45 - RHEL 5
• MySQL 5.0.84 - Percona HighPerf
• Patches to fix contention points
• Improved I/O
• Better scaling

Friday, 11 March 2011

Replace MySQL with MySQL

1000.00 1171.75 1343.50 1515.25 1687.00 Average Response Time (ms)

MySQL 5.0.45 Percona MySQL 5.0.85

0%

Friday, 11 March 2011

Replace MySQL with MySQL

6.25 12.50 18.75 25.00 Average CPU(%)

MySQL 5.0.45 Percona MySQL 5.0.84

0%

Friday, 11 March 2011

Replace MySQL with MySQL

• JIRA doesn’t do enough DB queries
• JIRA uses Lucene for many queries
• Percona Highperf MySQL really shines at high

query rates

• Try this if you share your DB server with other apps

Friday, 11 March 2011

Tune MySQL

• InnoDB-Heavy-4G.conf

sort_bufger_size = 8M join_bufger_size = 8M read_bufger_size = 2M read_rnd_bufger_size = 16M thread_concurrecy = 16

Friday, 11 March 2011

Tune MySQL

1000.0 1177.5 1355.0 1532.5 1710.0 Average Response Time (ms)

Defaults Tuned

1%

Friday, 11 March 2011

Tune MySQL

MyISAM ALL Engines

key_cache_age_threshold, key_cache_block_size, key_cache_division_limit, key_bufger_size read_bufger_size, read_rnd_bufger_size bulk_insert_bufger_size join_bufger_size - Large joins without Indexes sort_bufger_size - Used by ORDER BY / GROUP BY query_cache_size query_cache_limit query_cache_min_res_unit

Friday, 11 March 2011

Tune MySQL

• Be sure to set innodb_bufger_pool_size
• Note! innodb_log_file_size will afgect shutdown

speed

Friday, 11 March 2011

Remember!

• Upgrade!
• Java to 1.6
• Your server
• Use Parallel Garbage Collection with a big heap
• Remove virtualisation if you can
• Use 1000mbit network
• Buy an SSD for write workloads

Friday, 11 March 2011

A final word

• TEST EVERYTHING!

Friday, 11 March 2011

• http://blogs.atlassian.com/developer

Flickr Attributions: Siesta in Denmark - http://www.flickr.com/photos/jakecaptive/27123533/ Day 11 - http://www.flickr.com/photos/ivanomak/4057632458/ Blue Streak - http://www.flickr.com/photos/jettajet/2061715292/

Thanks! Questions?

Friday, 11 March 2011