clean performance test results By Vivek Venkatachalam - - PowerPoint PPT Presentation

▶
clean performance test results
SMART_READER_LITE
LIVE PREVIEW

clean performance test results By Vivek Venkatachalam - - PowerPoint PPT Presentation

Sep 11, 2023 284 likes •596 views

PerfCells: A case study in achieving clean performance test results By Vivek Venkatachalam ________________________________________________________________________________________________________________ Copies may not be made or distributed

slide-1
SLIDE 1

PerfCells: A case study in achieving clean performance test results

By Vivek Venkatachalam

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 1

slide-2
SLIDE 2

Performance is key for quality.

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 2

slide-3
SLIDE 3

Inconsistent results makes it hard to do performance testing well.

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 3

slide-4
SLIDE 4

The real problems are hard to find

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 4

slide-5
SLIDE 5

What if you could eliminate the haystack ?

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 5

slide-6
SLIDE 6

First, an overview of

  • ur performance

engineering system

  • Planning
  • Instrumentation
  • Testing
  • Analysis

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 6

slide-7
SLIDE 7

Planning - Key customer scenarios

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 7

slide-8
SLIDE 8

Planning - Metrics

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 8

slide-9
SLIDE 9

Instrumentation – Inserting code markers

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 9

slide-10
SLIDE 10

Testing - Manual

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 10

slide-11
SLIDE 11

Testing - Automation

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 11

slide-12
SLIDE 12

Analysis – Processing the results

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 12

slide-13
SLIDE 13

The problem: Lots of noise in the system.

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 13

slide-14
SLIDE 14

Unexplainable variations in results led to wasted time and effort

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 14

slide-15
SLIDE 15

Needed a quick fix

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 15

slide-16
SLIDE 16

Decided to focus only

  • n large anomalies

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 16

slide-17
SLIDE 17

The long term solution

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 17

slide-18
SLIDE 18

Brainstorming

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 18

slide-19
SLIDE 19

Realization : We had conflicting goals

Verify that real world performance goals were being met. Prevent regressions however small.

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 19

slide-20
SLIDE 20

The solution: Perf cells

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 20

slide-21
SLIDE 21

Key point: one machine

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 21

slide-22
SLIDE 22

Advantages - Isolated

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 22

slide-23
SLIDE 23

Advantages - Reliable behavior

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 23

slide-24
SLIDE 24

Advantages - Easy management

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 24

slide-25
SLIDE 25

Advantages - Scale Out

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 25

slide-26
SLIDE 26

Our results

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 26

slide-27
SLIDE 27

Histogram of elapsed times from tests on pre-production topology

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 27

slide-28
SLIDE 28

Histogram of elapsed times from tests on a PerfCell

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 28

slide-29
SLIDE 29

Key Takeaways

  • Steady losses add up.
  • PerfCells increased

test sensitivity.

  • Allowed us to accept

variability in real world testing.

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 29

slide-30
SLIDE 30

Some possible improvements

  • Test builds

simultaneously

  • Partition tests
  • PerfCells in the cloud

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 30

slide-31
SLIDE 31

Credits

My co-authors Marcelo and Shirley. Our reviewers Bob and Mano Microsoft Clip Art for the images.

Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG ________________________________________________________________________________________________________________ 31