Performance tuning for Java applications George Barnett, Atlassian - - PowerPoint PPT Presentation

performance tuning for java applications
SMART_READER_LITE
LIVE PREVIEW

Performance tuning for Java applications George Barnett, Atlassian - - PowerPoint PPT Presentation

Performance tuning for Java applications George Barnett, Atlassian Friday, 11 March 2011 Topics How to make your apps run faster A few process tips Tuning ideas What doesnt work Friday, 11 March 2011 A Note X vs Y will


slide-1
SLIDE 1

Performance tuning for Java applications

George Barnett, Atlassian

Friday, 11 March 2011

slide-2
SLIDE 2
  • How to make your apps run faster
  • A few process tips
  • Tuning ideas
  • What doesn’t work

Topics

Friday, 11 March 2011

slide-3
SLIDE 3
  • X vs Y will always depend on the context
  • Test with your application!

A Note

Friday, 11 March 2011

slide-4
SLIDE 4
  • Isolate performance issues before they become

headaches in production.

  • Make sure these issues get fixed.

Performance Engineering

Friday, 11 March 2011

slide-5
SLIDE 5

CODE! CODE! CODE!

Unit & Functional tests

Software artefact

Reporting

commit tests pass

Simple code lifecycle

Friday, 11 March 2011

slide-6
SLIDE 6

Performance Testing

Reporting

  • Catch regressions on commit
  • Create a standard benchmark
  • Performance testing will save your Ops team

time

Add to your testing

Friday, 11 March 2011

slide-7
SLIDE 7

Getting started

  • Define test cases and traffjc levels
  • Examine logs and monitoring data
  • Use real world data
  • Eg, Public Atlassian instances such as

http://jira.atlassian.com

Friday, 11 March 2011

slide-8
SLIDE 8
  • Apache JMeter
  • Howto:
  • http://blogs.atlassian.com/developer/2008/10/

performance_testing_with_jmete.html

  • Example
  • http://confluence.atlassian.com/display/DOC/

Performance+Testing+Scripts

Some useful tools

Friday, 11 March 2011

slide-9
SLIDE 9
  • Automation
  • Maven
  • Bamboo
  • Getting a repeatable build is critical!
  • Automated Performance Testing will save your

Dev team time!

Some useful tools

Friday, 11 March 2011

slide-10
SLIDE 10

How often?

  • Daily performance tests
  • Most products & libraries
  • Weekly soak tests
  • Run for much longer than daily tests

Friday, 11 March 2011

slide-11
SLIDE 11

Reporting

  • Put there data where you cant miss it
  • Build screens, dashboards, notifications
  • JMeter Aggregator Plugin

Friday, 11 March 2011

slide-12
SLIDE 12

Reporting

Friday, 11 March 2011

slide-13
SLIDE 13

Reporting

Friday, 11 March 2011

slide-14
SLIDE 14

You are here

  • How I did the testing
  • Hardware & Software
  • Must have improvements
  • Java, Hardware, Virtualization and Garbage Collection
  • Worth doing
  • Gigabit vs 100mbit, Heap size tuning, use an SSD
  • Sadly, not useful
  • Tune MySQL, Replace MySQL

Friday, 11 March 2011

slide-15
SLIDE 15

The contenders

  • Dell PE 850 - “WALL-E”
  • Xeon X3220 2.4Ghz Quad Core
  • Quad Core, 2 x 4Mb L3
  • 8Gb DDR2 RAM
  • 2 x 7.2K SAS Disks
  • 1 x Intel X25-E 32GB SSD
  • ~$1500

Friday, 11 March 2011

slide-16
SLIDE 16
  • Dell PE 2950 - “Johnny 5”
  • 2 x Xeon E5405 2.0Ghz
  • Quad Core, 2 x 6M L3
  • 32Gb RAM
  • 2 x 72Gb 15K Drives
  • ~$4000

The contenders

Friday, 11 March 2011

slide-17
SLIDE 17

The contenders

  • Dell PE R610 - “EVE”
  • 2 x Xeon E5520 2.2Ghz
  • Quad Core, 8M “SmartCache”
  • 32Gb RAM
  • 2 x 146G 15K drives
  • ~$4000

Friday, 11 March 2011

slide-18
SLIDE 18

The Workload

  • JIRA 4, MySQL 5, RHEL 5
  • Java 1.6, 3G heap, ParNew, ParOld GC
  • ~70,000 Issues, ~80 projects
  • ~30 reqs/sec, JMeter 2.3.4 (patched, JSR-223)
  • Traffjc modelled on http://jira.atlassian.com with

higher writes

Friday, 11 March 2011

slide-19
SLIDE 19

About the graphs

  • Most show Average Response Time in

Milliseconds

  • Includes time to generate AND deliver the response
  • Does not include any browser time
  • Arrows show if lower or higher is better!

Friday, 11 March 2011

slide-20
SLIDE 20

You are here

  • How I did the testing
  • Hardware & Software
  • Must have improvements
  • Java, Hardware, Virtualization and Garbage Collection
  • Worth doing
  • Gigabit vs 100mbit, Heap size tuning, use an SSD
  • Sadly, not useful
  • Tune MySQL, Replace MySQL

Friday, 11 March 2011

slide-21
SLIDE 21
  • Must. Have.

Friday, 11 March 2011

slide-22
SLIDE 22

Upgrade Java 1.5 to 1.6

  • Advances in compiler / VM technology
  • Produces better runtime code
  • Able to use modern instructions
  • Able to use modern hardware

Friday, 11 March 2011

slide-23
SLIDE 23

Java 1.5 vs 1.6 (Johnny 5)

375 750 1125 1500 Average Response Time (ms)

Java 1.5 Java 1.6.0

6.3X

Friday, 11 March 2011

slide-24
SLIDE 24

Java 1.5 vs 1.6 (Johnny 5)

95 190 285 380 Average CPU (%)

26%

Java 1.5.0 Java 1.6.0

Friday, 11 March 2011

slide-25
SLIDE 25

Upgrade Java to 1.6

  • HotSpot VM updates in 1.6
  • Advancing technology
  • Compressed OOPS on 64bit
  • Escape Analysis
  • NUMA
  • http://java.sun.com/performance/reference/

whitepapers/6_performance.html

Friday, 11 March 2011

slide-26
SLIDE 26

Java 1.6 vs 1.6 (Eve)

37 74 111 148 Average Response Time (ms)

Java 1.6.0_7 Java 1.6.0_16

7%

Friday, 11 March 2011

slide-27
SLIDE 27

Hardware Upgrades

  • Hyper-Threading
  • QuickPath Interconnect
  • Nehalem (Eve) have on die DDR3 Memory Controllers
  • Memory throughput improvements - good for Java!

Technology Speed

DDR2-800 (dual channel) 6.4 GB/sec DDR3-1333 (dual channel) 21.3 GB/sec

Friday, 11 March 2011

slide-28
SLIDE 28
  • Shared bus
  • 1333 MT/s = 1333M Transfers / sec
  • 4 transfers/clock = 266Mhz.

CPU

North Bridge

RAM

I/O: SATA, USB, etc

FSB

CPU

South Bridge

FSB architecture

Friday, 11 March 2011

slide-29
SLIDE 29

Quick Path Interconnect

  • 3.2Ghz, 6.4 GT/sec
  • 25.6 GB/sec

CPU

QPI

CPU

RAM RAM

IOH

Friday, 11 March 2011

slide-30
SLIDE 30

Hyper Threading

CPU

Thread 1 Thread 2

Thread 1 runs until it hits a cache miss. Thread 2 runs on the same CPU while T1 is awaiting data

Friday, 11 March 2011

slide-31
SLIDE 31

Hardware Upgrades

500 1000 1500 2000 Average Response Time (ms)

2.4Ghz Xeon (Wall-E) 2.0Ghz Xeon (Johnny 5) 2.2Ghz Xeon (Eve)

8-12X

Friday, 11 March 2011

slide-32
SLIDE 32

Improved GC Throughput

7500 15000 22500 30000 GC Throughput (MB/sec)

2.4Ghz Xeon (Wall-E) 2.0Ghz Xeon (Johnny 5) 2.2Ghz Xeon (Eve)

2-3X

Friday, 11 March 2011

slide-33
SLIDE 33

Garbage Collection

  • Java’s Virtual Machine manages memory
  • New objects are continually created during

runtime

  • GC is the process of collecting dead objects to

reclaim memory

Friday, 11 March 2011

slide-34
SLIDE 34

Tune Garbage Collection

  • Getting your GC tuning wrong is bad
  • How bad?
  • Are the defaults sane?

Friday, 11 March 2011

slide-35
SLIDE 35

Tune Garbage Collection

1000 2000 3000 4000 Wall-E (ms) Johnny 5 (ms) Eve (ms)

CMS ParNew ParOld

Average Response Time

2X

Friday, 11 March 2011

slide-36
SLIDE 36

Tune Garbage Collection

125 250 375 500 Johnny 5 (ms) Eve (ms)

CMS ParNew ParOld

Average Response Time

2X

Friday, 11 March 2011

slide-37
SLIDE 37

Tune Garbage Collection

  • Defaults are sane
  • Parallel New + Parallel Old is fastest
  • CMS prioritises low pauses over CPU usage
  • CMS does not compact - can lead to heap

fragmentation

Friday, 11 March 2011

slide-38
SLIDE 38

Virtualisation

  • Does being in a VM hurt?
  • What if the risks are managed?

Friday, 11 March 2011

slide-39
SLIDE 39

Virtualisation

42.5 85.0 127.5 170.0 Average Response Time (ms)

Real VMWare ESX 3.5 VMWare ESX 4i

43%

Friday, 11 March 2011

slide-40
SLIDE 40

Virtualisation

  • On average ~30% slower
  • Under extreme resource starvation, up to 100%

slower

  • Avoid virtualisation for high traffjc instances
  • See the Atlassian docs for recommendations
  • http://confluence.atlassian.com/display/DOC/Running+Confluence+in+a+Virtualised

+Environment

  • http://confluence.atlassian.com/display/JIRA/Running+JIRA+in+a+Virtualised+Environment

Friday, 11 March 2011

slide-41
SLIDE 41

You are here

  • How I did the testing
  • Hardware & Software
  • Must have improvements
  • Java, Hardware, Virtulization and Garbage Collection
  • Worth doing
  • Gigabit vs 100mbit, Heap size tuning, use an SSD
  • Sadly, not useful
  • Tune MySQL, Replace MySQL

Friday, 11 March 2011

slide-42
SLIDE 42

Maybe.

Friday, 11 March 2011

slide-43
SLIDE 43

Gigabit Network vs 100Mbit

  • Gigabit is faster, sure
  • But 100Mb is OK if you’re not doing big

transfers, RIGHT?!

Friday, 11 March 2011

slide-44
SLIDE 44

Use Gigabit

475 950 1425 1900 Average Response Time (ms)

100mb/sec 1000mb/sec

10%

Friday, 11 March 2011

slide-45
SLIDE 45

Use Gigabit

  • Lower RTT on Gigabit
  • If you have enough CPU power, keep the DB on

localhost

  • Check your development environment

Friday, 11 March 2011

slide-46
SLIDE 46

Use Gigabit

50 100 150 200 ReIndex Time (seconds)

100Mbit 1000Mbit

26%

Friday, 11 March 2011

slide-47
SLIDE 47

Heap size tuning

  • Memory starvation is bad, but can we “drown”

java with too much memory?

Friday, 11 March 2011

slide-48
SLIDE 48

Heap size - Wall-E

500 1000 1500 2000 Average Response Time

1.5G 3G 6G

23%

Friday, 11 March 2011

slide-49
SLIDE 49

Heap size - Johnny 5

72.5 145.0 217.5 290.0 Average Response Time

1.5G 3G 6G 15G

49%

Friday, 11 March 2011

slide-50
SLIDE 50

Heap size - Eve

42.5 85.0 127.5 170.0 Average Response Time

1.5G 3G 6G 15G

28%

Friday, 11 March 2011

slide-51
SLIDE 51

Heap size tuning

  • Heap too small - bad
  • Heap too big - not bad!
  • Bigger heaps give the JVM more room to work
  • Diminishing returns from really big heaps

Friday, 11 March 2011

slide-52
SLIDE 52

Buy an SSD

  • SSDs are FAST!
  • SSDs are EXPENSIVE!
  • SSDs are all the rage!
  • DBAs are finally happy (almost)

Friday, 11 March 2011

slide-53
SLIDE 53

Buy an SSD

200 400 600 800 Browse Issue Create Issue

SATA SSD

33%

Friday, 11 March 2011

slide-54
SLIDE 54

Buy an SSD

  • Fast writes
  • 0.1ms Avg Service Time
  • Most improvement comes from SSD on the DB
  • Writing to the Lucene index means re-opening

IndexReaders

  • Much faster on SSD
  • Lucene Search Term Cache fits in OS bufgers

Friday, 11 March 2011

slide-55
SLIDE 55

You are here

  • How I did the testing
  • Hardware & Software
  • Must have improvements
  • Java, Hardware, Virtulization and Garbage Collection
  • Worth doing
  • Gigabit vs 100mbit, Heap size tuning, use an SSD
  • Sadly, not useful
  • Tune MySQL, Replace MySQL

Friday, 11 March 2011

slide-56
SLIDE 56

Meh.

Friday, 11 March 2011

slide-57
SLIDE 57

Replace MySQL with MySQL

  • MySQL 5.0.45 - RHEL 5
  • MySQL 5.0.84 - Percona HighPerf
  • Patches to fix contention points
  • Improved I/O
  • Better scaling

Friday, 11 March 2011

slide-58
SLIDE 58

Replace MySQL with MySQL

1000.00 1171.75 1343.50 1515.25 1687.00 Average Response Time (ms)

MySQL 5.0.45 Percona MySQL 5.0.85

0%

Friday, 11 March 2011

slide-59
SLIDE 59

Replace MySQL with MySQL

6.25 12.50 18.75 25.00 Average CPU(%)

MySQL 5.0.45 Percona MySQL 5.0.84

0%

Friday, 11 March 2011

slide-60
SLIDE 60

Replace MySQL with MySQL

  • JIRA doesn’t do enough DB queries
  • JIRA uses Lucene for many queries
  • Percona Highperf MySQL really shines at high

query rates

  • Try this if you share your DB server with other apps

Friday, 11 March 2011

slide-61
SLIDE 61

Tune MySQL

  • InnoDB-Heavy-4G.conf

sort_bufger_size = 8M join_bufger_size = 8M read_bufger_size = 2M read_rnd_bufger_size = 16M thread_concurrecy = 16

Friday, 11 March 2011

slide-62
SLIDE 62

Tune MySQL

1000.0 1177.5 1355.0 1532.5 1710.0 Average Response Time (ms)

Defaults Tuned

1%

Friday, 11 March 2011

slide-63
SLIDE 63

Tune MySQL

MyISAM ALL Engines

key_cache_age_threshold, key_cache_block_size, key_cache_division_limit, key_bufger_size read_bufger_size, read_rnd_bufger_size bulk_insert_bufger_size join_bufger_size - Large joins without Indexes sort_bufger_size - Used by ORDER BY / GROUP BY query_cache_size query_cache_limit query_cache_min_res_unit

Friday, 11 March 2011

slide-64
SLIDE 64

Tune MySQL

  • Be sure to set innodb_bufger_pool_size
  • Note! innodb_log_file_size will afgect shutdown

speed

Friday, 11 March 2011

slide-65
SLIDE 65

Remember!

  • Upgrade!
  • Java to 1.6
  • Your server
  • Use Parallel Garbage Collection with a big heap
  • Remove virtualisation if you can
  • Use 1000mbit network
  • Buy an SSD for write workloads

Friday, 11 March 2011

slide-66
SLIDE 66

A final word

  • TEST EVERYTHING!

Friday, 11 March 2011

slide-67
SLIDE 67
  • http://blogs.atlassian.com/developer

Flickr Attributions: Siesta in Denmark - http://www.flickr.com/photos/jakecaptive/27123533/ Day 11 - http://www.flickr.com/photos/ivanomak/4057632458/ Blue Streak - http://www.flickr.com/photos/jettajet/2061715292/

Thanks! Questions?

Friday, 11 March 2011