Dealing with Data in Title Performance Testing Bogdan Veres March - - PowerPoint PPT Presentation

Title Dealing with Data in Performance Testing

Bogdan Veres March 2018

Hi, I’m Bogdan

Software Tester @ Softvision

since 2010

Performance & Automation Testing Professional Traveler QA Community Lead Mountain Biker Computer Hardware Enthusiast

Performance tests - why are they necessary?

• Performance testing is necessary to make sure that the application runs

quickly, is stable and can scale

• To prevent users being affected by low performance
• Find out how many resources are required to handle the expected load

What happens when the app is not performing?

Famous performance issues:

• Bieber bug - too popular to handle
• Google fail - caused by Michael Jackson’s death
• Diablo 3 - authentication server fail on release

Dan Downing’s 5 Steps of Load Testing

1. Discover

• a. Define use case workflows
• b. Model production workload

2. Develop

• a. Develop test scripts
• b. Configure environment monitors

3. Analyze

• a. Run tests
• b. Monitor system resources
• c. Analyze results

4. Fix

• a. Fix
• b. Diagnose
• c. Re-test

5. Report

• a. Interpret results
• b. Make recommendations
• c. Present to stakeholders

Generating the Load

Performance KPIs

Define required metrics for reporting

Execution Time vs Cycle

Metrics

• System (CPU, Memory, Processes)
• Network throughput
• Disk IO
• # of requests
• # of messages in queues
• # of connections
• # of errors
• # of DB connections
• Detailed memory stats
• Response size
• Apache/IIS/NGINX/Tomcat

Getting from this…

...to this

Tools

Metrics Tools OS CPU, Memory, Net, Netstat, Disk, Disk IO ,Swap, Processes perfmon Windows ps Linux prstat Linux dstat Linux Apache (connections, requests, errors) Apache Monitor Windows/Linux NGINX (connections, requests, errors) Telegraf (NGINX plugin) Windows/Linux IIS (connections, requests, errors) Perfmon Windows MSSQL(queries, heap memory, deadlocks) Perfmon Windows Telegraf Windows/Linux Oracle (queries, heap memory, deadlocks) Telegraf Windows/Linux Oracle DB Monitor Windows/Linux Network (tcp, http traffic) Wireshark Windows/Linux Telegraf Windows/Linux

Setup Environment

Use Docker for:

• Automation – Dockerfile for setup
• DevOps – infrastructure as code
• Scale – Docker compose
• Maintenance – entire isolated runtime environment

Docker environment

TICK Stack

• Telegraf - agent for collecting and reporting metrics
• InfluxDB - time series database for real time analytics (SQL like query

language)

• Chronograf - administrative user interface and visualization
• Kapacitor - data processing engine

TICK scripting language Influx Query Language

• SQL like query language
• A timestamp identifies a single point in any

given data series

• InfluxDB isn’t CRUD

Interpreting, Objectives & Recommendations

• Observations
• Correlations
• Hypotheses
• Conclusions
• Compare graphs
• Results from current build vs. previous build
• Create conclusions – tie them back to test objectives
• Review solution
• Quantify the benefit, cost and effort
• Final outcome is management’s judgement

Reporting

• Summary - 3 pages max
• Add test details (# of users, hardware configuration, build version, scenario)
• Key graphs in order of importance
• Annotate graphs
• Draw conclusions
• Recommendations
• Create a sections for errors (add details for errors)
• Present your report - no one is going to read the it

Resources

• https://www.soasta.com/blog/
• https://docs.influxdata.com/
• http://www.perftestplus.com/resources.htm
• http://focus.forsythe.com/articles/335/The-4-Hats-of-Application-Performance-Testing
• https://testingpodcast.com/tag/dan-downing/
• https://www.blazemeter.com/jmeter-blog-posts