Unit Testing Performance in Java Projects: Are We There Yet? Petr - - PowerPoint PPT Presentation

▶
unit testing performance in java projects are we there yet
SMART_READER_LITE
LIVE PREVIEW

Unit Testing Performance in Java Projects: Are We There Yet? Petr - - PowerPoint PPT Presentation

Feb 26, 2023 46 likes •511 views

Unit Testing Performance in Java Projects: Are We There Yet? Petr Stefan, Vojtch Hork , Lubomr Bulej and Petr Tma Charles University Unit Testing Performance in Java Projects: Are We There Yet? Unit Testing Performance in Java

slide-1
SLIDE 1

Unit Testing Performance in Java Projects: Are We There Yet?

Petr Stefan, Vojtěch Horký, Lubomír Bulej and Petr Tůma

Charles University

slide-2
SLIDE 2

Unit Testing Performance in Java Projects: Are We There Yet?

slide-3
SLIDE 3

Unit Testing Performance in Java Projects: Are We There Yet?

Short, isolated performance tests of individual components. Suitable to be run during nightly builds.

slide-4
SLIDE 4

Unit Testing Performance in Java Projects: Are We There Yet?

Where else :-)

slide-5
SLIDE 5

Unit Testing Performance in Java Projects: Are We There Yet?

How many projects have such tests? ...

slide-6
SLIDE 6

“Are We There Yet?” Expanded

– How many projects have performance unit tests? – Which framework is used for these tests? – What kind of projects care about performance? – How often are performance unit tests changed (maintained)? – Are the tests short enough for “after every commit” execution? – What (de)motivates developers towards performance tests?

slide-7
SLIDE 7

Setup

slide-8
SLIDE 8

Getting the Data

Semi-automated exploration of open source projects. Survey for developers that actually added/updated the performance tests.

slide-9
SLIDE 9

Getting the Projects

GitHub as one of the prominent hosting platforms. Almost 2.5 million of Java projects.

slide-10
SLIDE 10

Getting the Projects

GitHub as one of the prominent hosting platforms. Almost 2.5 million of Java projects. We limit ourselves to projects with a fork to filter out abandoned repositories, toy examples and school assignments.

slide-11
SLIDE 11

Getting the Projects

GitHub as one of the prominent hosting platforms. Almost 2.5 million of Java projects. We limit ourselves to projects with a fork to filter out abandoned repositories, toy examples and school assignments. That totals to almost 100 thousand projects and over 3TB of data.

slide-12
SLIDE 12

Developer’s Survey

Targeted committers that modified the performance tests.

slide-13
SLIDE 13

Developer’s Survey

Targeted committers that modified the performance tests. 483 invitations, 111 responses. Questions steered by results of source code analysis.

slide-14
SLIDE 14

Results

slide-15
SLIDE 15

Which Frameworks Do the Projects Use?

Methodology: parse the Java code, look for typical annotations

  • r base classes . . .
slide-16
SLIDE 16

Which Frameworks Do the Projects Use?

Methodology: parse the Java code, look for typical annotations

  • r base classes . . .

Example: this marks JMH (or Caliper) test (we use full-fledged parser to resolve ambiguities through imports): @Benchmark public void x() { ... }

slide-17
SLIDE 17

Which Frameworks Do the Projects Use?

Methodology: parse the Java code, look for typical annotations

  • r base classes . . .

Example: this marks JMH (or Caliper) test (we use full-fledged parser to resolve ambiguities through imports): @Benchmark public void x() { ... } We look for two functional testing frameworks and five performance testing frameworks.

slide-18
SLIDE 18

Testing Frameworks

Framework Repositories Relative usage JUnit 4 30871 31.177 % TestNG 2053 2.073 % Caliper 12 0.012 % ContiPerf 17 0.017 % Japex 52 0.053 % JMH 278 0.281 % JUnitPerf 11 0.011 %

slide-19
SLIDE 19

Testing Frameworks

Framework Repositories Relative usage JUnit 4 30871 31.177 % TestNG 2053 2.073 % Caliper 12 0.012 % ContiPerf 17 0.017 % Japex 52 0.053 % JMH 278 0.281 % JUnitPerf 11 0.011 % From the survey: JMH is the most popular because developers trusts the results, like the documentation and active maintenance.

slide-20
SLIDE 20

Project Count Over Time

10000 25000

JUnit

Year Project count 2010 2013 2016 50 150 250

JMH

Year Project count 2010 2013 2016

slide-21
SLIDE 21

When Tests Are Introduced

JUnit 25 50 75 100 JMH 25 50 75 100 Relative test count [%] Project lifetime ( ← Initial commit; HEAD → )

slide-22
SLIDE 22

When Tests Are Introduced

JUnit 25 50 75 100 JMH 25 50 75 100 Relative test count [%] Project lifetime ( ← Initial commit; HEAD → )

From the survey: less than one third of developers maintains performance code regularly. Two thirds updates the tests only when addressing performance issues.

slide-23
SLIDE 23

Home-baked Performance Testing

Not everybody is using testing frameworks . . .

slide-24
SLIDE 24

Home-baked Performance Testing

Not everybody is using testing frameworks . . .

  • 1. Took 1 000 projects randomly.
slide-25
SLIDE 25

Home-baked Performance Testing

Not everybody is using testing frameworks . . .

  • 1. Took 1 000 projects randomly.
  • 2. Look for code that queries system clock.
slide-26
SLIDE 26

Home-baked Performance Testing

Not everybody is using testing frameworks . . .

  • 1. Took 1 000 projects randomly.
  • 2. Look for code that queries system clock.
  • 3. Determine whether it is used for benchmarking.
slide-27
SLIDE 27

Home-baked Performance Testing

Not everybody is using testing frameworks . . .

  • 1. Took 1 000 projects randomly.
  • 2. Look for code that queries system clock.
  • 3. Determine whether it is used for benchmarking.

About 3% of the projects use it for benchmarking (we err on the optimistic side).

slide-28
SLIDE 28

Exploring Projects with JMH Tests

JMH-based projects only. – Only framework with at least permille representation. – Standardized execution. – Apart from Caliper the only framework maintained till today.

slide-29
SLIDE 29

JMH Projects Classification

Category Count Database (ORM, SQL . . . ) 33 Tutorials and examples 30 Networking and distributed systems 29 Algorithms 27 Data structures 22 Object serialization, parsers (XML, JSON, . . . ) 22 Web frameworks or plugins 18

slide-30
SLIDE 30

Size of Projects Using JMH

50 100 150 200 250 300 50 100 150 Project size [thousands of lines of code] (9 largest projects omitted) Project count

slide-31
SLIDE 31

Can We Run the Tests After Each Commit?

50 100 150 200 5 10 15 20 Benchmark duration [min] Project count

From the survey: less than one half of developers run the tests regularly.

slide-32
SLIDE 32

One Hour Budget: Accuracy of Performance Data

2000 4000 6000 8000 10000 10 20 30 40 50 60 Relative confidence interval size [%] Project count

slide-33
SLIDE 33

One Hour Budget: Accuracy of Performance Data

20 40 60 80 100 2 4 6 8 10 12 Relative confidence interval size [%] Project count

slide-34
SLIDE 34

Greater Adoption of Performance Tests

In the survey, we have learned that – not enough time/money to maintain the tests (27%)

slide-35
SLIDE 35

Greater Adoption of Performance Tests

In the survey, we have learned that – not enough time/money to maintain the tests (27%) – regular performance testing is rare (42%)

slide-36
SLIDE 36

Greater Adoption of Performance Tests

In the survey, we have learned that – not enough time/money to maintain the tests (27%) – regular performance testing is rare (42%) – tool integration with build infrastructure is too complex (50%)

slide-37
SLIDE 37

Greater Adoption of Performance Tests

In the survey, we have learned that – not enough time/money to maintain the tests (27%) – regular performance testing is rare (42%) – tool integration with build infrastructure is too complex (50%) – developers miss (simple) automated evaluation (60%)

slide-38
SLIDE 38

Greater Adoption of Performance Tests

In the survey, we have learned that – not enough time/money to maintain the tests (27%) – regular performance testing is rare (42%) – tool integration with build infrastructure is too complex (50%) – developers miss (simple) automated evaluation (60%)

slide-39
SLIDE 39

Towards Simple Evaluation

From the survey: for one third of developers JMH is complex.

slide-40
SLIDE 40

Towards Simple Evaluation

From the survey: for one third of developers JMH is complex.

Our extension to JMH

– To JMH itself: dump raw data (more of them). – New Maven plugin: compares two runs of a JMH benchmark to simplify regression benchmarking.

slide-41
SLIDE 41

Unit Testing Performance in Java Projects: Are We There Yet?

slide-42
SLIDE 42

Unit Testing Performance in Java Projects: Are We There Yet?

Not really ;-)

slide-43
SLIDE 43

Unit Testing Performance in Java Projects: Are We There Yet?

Performance tests are in less than 3 % of Java projects (functional tests are present in about one third).

slide-44
SLIDE 44

Unit Testing Performance in Java Projects: Are We There Yet?

Performance tests are in less than 3 % of Java projects (functional tests are present in about one third). The existing tests are not suitable for build-time testing (complex setup, limited budget, little pressure to do so).

slide-45
SLIDE 45

Unit Testing Performance in Java Projects: Are We There Yet?

Performance tests are in less than 3 % of Java projects (functional tests are present in about one third). The existing tests are not suitable for build-time testing (complex setup, limited budget, little pressure to do so). Complexity of the benchmarking tools and a limited budget seems to be the biggest obstacles.

slide-46
SLIDE 46

Unit Testing Performance in Java Projects: Are We There Yet?

Performance tests are in less than 3 % of Java projects (functional tests are present in about one third). The existing tests are not suitable for build-time testing (complex setup, limited budget, little pressure to do so). Complexity of the benchmarking tools and a limited budget seems to be the biggest obstacles. http://d3s.mff.cuni.cz/resources/icpe2017/

Thank you!