Demystifying Benchmarks How to Use Them to Better Evaluate Databases - - PowerPoint PPT Presentation
Flexible transactional scale for the connected world. Demystifying Benchmarks How to Use Them to Better Evaluate Databases Peter Friedenbach, Performance Architect | Clustrix Demystifying Benchmarks Outline A Brief History of Database
Flexible transactional scale for the connected world.
How to Use Them to Better Evaluate Databases
Peter Friedenbach, Performance Architect | Clustrix
Outline
– Best Practices and Lessons Learned
2
In the beginning, there was the “TPS Wars”
3
database performance benchmark.
– ACID property transactions – Price/Performance – Response Time Constraints – Database Scaling Rules
4
Performance Council
– Founded in 1988, the TPC was chartered to established industry standards out of the madness.
TPC Timeline:
1988 : TPC Established 1989 : TPC-A Approved (Formalizes Debit/Credit) 1990 : TPC-B Approved (Database only version of TPC-A) 1992 : TPC-C Approved (Replaces TPC-A OLTP workload) 1995 : TPC-D Approved (1st Decision Support Workload) TPC-A & TPC-B retired. 1999 : TPC-H Approved (Replaces TPC-D Workload) 2000 : TPC-C v5 Approved (Major revision to TPC-C) 2006 : TPC-E Approved (Next Generation OLTP workload) 2009 : First TPC Technology Conference on Performance Evaluation & Benchmark (Held in conjunction with VLDB) 2012 : TPC-VMS Approved (1st Virtualized Database benchmark) 2014 : TPCx-HS Approved (1st Hadoop based benchmark) 2015 : TPC-DS Approved (Next Generation Decision Support Benchmark) 2016 : TPCx-V Approved (Virtual DB Server benchmark) 2016 : TPCx-BB Published (Hadoop Big Data benchmark)
5
– Established the rules to the game – For the first time, competitive performance claims could be compared – Audited results – Standard workloads focused the industry and drove performance improvements
6
– Expensive to play – “Benchmarketing” and gamesmanship – Dominated by vendors
– Slow to evolve to a changing marketplace.
7
8
– Open source toolkit
Percona – Implements multiple workloads designed to test the CPU, disk, memory, and database capabilities of a system – Database workload allows for a mixture of reads (singleton selects and range queries) and writes (inserts, updates, and deletes) – Sysbench is popular in the Mysql Marketplace
9
“sysbench is a modular, cross-platform and multi- threaded benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load.”
(Yahoo! Cloud Serving Benchmark) – Open source toolkit
– Multi-threaded driver exercising get and put operations against an
– YCSB is popular with the NoSQL Marketplace
10
“YCSB is a framework and common set of workloads for evaluating the performance of different “key-value” and “cloud” serving stores.”
– “TPC-like” workloads live on
– OLTPBench
– Others?
11
– What are you trying to measure/test?
12
Option 1: Rely on “published Results”
Option 2: Leverage open source benchmarks
Option 3: Model your own workload (“Proof of Concept”)
profiles
13
Performance Curves
14
Throughput Latency (Response Time)
Performance is a tradeoff
latency. Design your tests with a variable in mind.
mistake.)
– Where systems can bottleneck:
transaction managers, …)
15
In any test, there are:
– Three fundamental system variables
– Driver mode
– Database design variables
– Multiple test variables
16
Real performance work is an exercise of control.
17
A workload will scale
appropriately.
Benchmarks are not “representable workloads”
– The complexity of the benchmark does not determine the goodness of the benchmark.
18
Good benchmark performance is “necessary but not sufficient” for good application performance.
– CPU Processor: sysbench cpu – Disk Subsystem: sysbench fileio – Network Subsystem: iperf
– Sysbench for ACID transactions: point selects, point updates, and simple mixes (90:10 or 80:20) – YCSB for nonACID transactions: workloadc (readonly), workloada and workloadb (read/write mixes)
– DebitCredit and OrderEntry (“TPC like database only workloads.”)
any or all of the above.
19