building better benchmarks
play

Building Better Benchmarks PGCon 2020 . . . . . . . . . . - PowerPoint PPT Presentation

Nov 24, 2023 •220 likes •472 views

Building Better Benchmarks PGCon 2020 . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . . About me Employed by 2ndQuadrant PostgreSQL

  1. Building Better Benchmarks PGCon 2020 . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  2. About me … • Employed by 2ndQuadrant • PostgreSQL Contributor since 2005 • Director at United States PostgreSQL Association since 2011 • Portland PostgreSQL Users Group . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  3. Overview • What is benchmarking about. • What benchmarks are out there? • Issues with benchmarking. • Where do we want to go now? . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  4. Reasons for benchmarking • Competition • Performance • Costs • Self­assessment • How well am I utilizing the hardware? • How does this patch affect the system? . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  5. Competitive Benchmarking Taking a look at just a of couple of industry standards bodies… . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  6. https://www.spec.org/jAppServer2004/results/res2007q3/ Benchmark Publications Only a couple of published Postgres benchmarks: • Two SPECjAppServer2004 results in 2007 • No TPC publications for PostgreSQL . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  7. http://www.spec.org/ SPEC SPEC is a non­profit organization that establishes, maintains and endorses standardized benchmarks and tools to evaluate performance and energy efficiency for the newest generation of computing systems. • develops suites of benchmarks intended to measure computer performance • benchmarks suites are available to the public for a fee covering development and administrative costs • publishes news and benchmark results at . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  8. http://www.tpc.org/ TPC Transaction Processing Performance Council performs two major activity: 1. creating benchmarks 2. creating a process for reviewing and monitoring those benchmarks . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  9. TPC Benchmarks Many TPC benchmarks are not trivial: • Complete benchmarking kits generally not provided • Database design • Data generation • Transaction profiles • Auditing • Publication . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  10. Summarizing Competitive Benchmarking • Well­defined workloads • Expensive • Non­trivial effort to implement • Non­trivial effort to execute • Not intended for individuals . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  11. Needs for Self-assessment Important data in addition to benchmark metrics: • System statistics: • processor utilization • i/o throughput • etc. • Software profiling: • annotated source code • call graphs • flame charts • etc. . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  12. http://www.vldb.org/pvldb/vol7/p277-difallah.pdf OLTPBench In many cases, researchers and developers are limited to a small number of workloads to evaluate the performance characteristics of their work. This is due to the lack of a universal benchmarking infrastructure, and to the difficulty of gaining access to real data and workloads. This results in lots of unnecessary engineering efforts and makes the performance evaluation results difficult to compare. . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  13. Adapting benchmarks At least 12 open source projects already exist: • http://oltpbenchmark.com/ • https://github.com/tvondra/pg_tpch • https://github.com/Percona-Lab/sysbench-tpcc • https://sourceforge.net/projects/benchmarksql/ • http://jtpcc.sourceforge.net/ • https://www2.infor.uva.es/~diego/tpcc-uva.php • https://github.com/pivotalguru/TPC-DS • https://oss.oracle.com/projects/olt/ • https://www.hammerdb.com/ • https://github.com/2ndQuadrant/pg-tpch • http://osdldbt.sourceforge.net/ • https://github.com/jopereira/java-tpcw . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  14. http://www.tpc.org/tpc_documents_current_versions/current_specifications5.asp Potential Licensing Issues While subject to change, EULA may not be agreeable. Excerpt from a previous TPC EULA: 4. Restrictions. The following restrictions apply to all use of the Materials by You. a. General: You may not: … iii. submit or contribute the Materials, or any part thereof, to any open source software project or as part of any third­party software or project without the express written consent of the TPC chair. . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  15. https://ycsb.site Quick Mention of Other Open Source Database Workloads Want to discuss more in unconference session? • Yahoo Cloud Serving Benchmark ­ key­value store • LDBC ­ graph databases http://ldbcouncil.org/ . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  16. Summarizing Needs for Self-Assessment Benchmarking kits are more helpful if they: • Collect system statistics • Profile the system • Unencumbered licensing . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  17. Where should we go from here? . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  18. Develop Our Own Database Workloads Focusing on just two generate types of workload: • Online transaction processing (OLTP) • Multi­tier client­server • Electronic data processing, wholesaler supplier managing orders, brokerage firm executing customer transactions • Business Intelligence (BI) • Single system • Business oriented ah­doc queries, big data systems star­schema . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  19. Summary of Workload Architectures • OLTP ­ Complex: • multi­tier: develop driver, develop or use existing connection pooler • Scalability of driver • Server vs client side transaction logic • BI ­ Simple: • Single­tier • Scripts to execute queries and streams of queries. • Maybe we can outline a test in an unconference session? . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  20. Is there a current need for competitive benchmarking? • Lots to consider: • Fair playing field • Enforcing fair play • etc. • Further discussion in an unconference session? . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  21. A little fun: Compare a Java vs. C implementation of a TPC­C derived workload. . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  22. My hardware HP ZBook Studio G5 Mobile Workstation: • Intel(R) Core(TM) i7­8750H CPU @ 2.20GHz (6 cores / 12 threads) • 64 GB RAM • HP EX950 M.2 NVMe 2TB • Any hardware donations available? . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  23. Test Parameters OLTPBench vs DBT­2: • Scale factor: 1 warehouse • 10 terminals each with a database connection • No keying or thinking time • 6 Cores (SMT disabled) . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  24. Results OLTPBench: DBT­2: • Metric 21,580.49 • Metric 28,838.20 new­order TPM new­order TPM • Processor cores 80% ­ • Processor cores 60% 90% utilized utilized • driver ­ 1 java process • driver ­ 76% processor 100% processor utilization utilization • i/o 99% utilized • i/o 97% utilized • 10 postgres backends 30% • 10 postgres backends 30% processor utilization each processor utilization each . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

BENCHMARKS TOPIC SUMMARY Scott Adams, Dilbert BENCHMARKS The Investment Process and how BM fits

BENCHMARKS TOPIC SUMMARY Scott Adams, Dilbert BENCHMARKS The Investment Process and how BM fits

BENCHMARKS TOPIC SUMMARY Scott Adams, Dilbert BENCHMARKS The Investment Process and how BM fits BM definition and why is it important? Uses for BMs Different types Who are the relevant parties? Properties of Ideal BM Examples Problems in

584 views • 22 slides
AGENDA 1. Review of Metro Hartford Future 2. Vision and Benchmarks 3. Goals 4. Key

AGENDA 1. Review of Metro Hartford Future 2. Vision and Benchmarks 3. Goals 4. Key

AGENDA 1. Review of Metro Hartford Future 2. Vision and Benchmarks 3. Goals 4. Key Indicators 5. Proposed Strategies 6. Strategy Refinement and Prioritization 2 What are We Building More Than a Plan Vision Situational

950 views • 55 slides
Benchmarks Online Testing Data District Benchmarks English/Language Arts and Math

Benchmarks Online Testing Data District Benchmarks English/Language Arts and Math

Benchmarks Online Testing Data District Benchmarks English/Language Arts and Math Benchmark A: Sept. 16-25 Benchmark B: Jan. 25-Feb. 5 Benchmark C: June 6-17 All ELA/Math benchmarks will be delivered online and in LinkIt!

266 views • 14 slides
Demystifying Benchmarks How to Use Them to Better Evaluate Databases Peter Friedenbach,

Demystifying Benchmarks How to Use Them to Better Evaluate Databases Peter Friedenbach,

Flexible transactional scale for the connected world. Demystifying Benchmarks How to Use Them to Better Evaluate Databases Peter Friedenbach, Performance Architect | Clustrix Demystifying Benchmarks Outline A Brief History of Database

591 views • 20 slides
Establishing Realistic Investment Earnings Benchmarks What is a Benchmark? A benchmark is a

Establishing Realistic Investment Earnings Benchmarks What is a Benchmark? A benchmark is a

Establishing Realistic Investment Earnings Benchmarks What is a Benchmark? A benchmark is a standard reference point that is used to compare the results of similar investments over time. Institutional investors use benchmarks to analyze

543 views • 39 slides
Multicore OS Benchmarks: We Can Do Better Ihor Kuz* , Zachary Anderson, Pravin Shinde, Timothy

Multicore OS Benchmarks: We Can Do Better Ihor Kuz* , Zachary Anderson, Pravin Shinde, Timothy

Multicore OS Benchmarks: We Can Do Better Ihor Kuz* , Zachary Anderson, Pravin Shinde, Timothy Roscoe Systems Group, ETH Zurich * NICTA Australia 1 Multicore OS benchmarks do not evaluate performance isolation between independent apps. 2

214 views • 10 slides
Building Blocks for Value-Based Payment N A H D O 2 0 1 9 H E A L T H C A R E D A T A S U M M

Building Blocks for Value-Based Payment N A H D O 2 0 1 9 H E A L T H C A R E D A T A S U M M

Medicare Price Benchmarks: Building Blocks for Value-Based Payment N A H D O 2 0 1 9 H E A L T H C A R E D A T A S U M M I T N O V E M B E R 6 , 2 0 1 9 L I T T L E R O C K , A R C H A P I N W H I T E S E N I O R P O L I C Y R E S

464 views • 29 slides
and Benchmarks May 24, 2018 Panelists Katy Miller Regional Coordinator Jasmine Hayes Deputy

and Benchmarks May 24, 2018 Panelists Katy Miller Regional Coordinator Jasmine Hayes Deputy

Federal Criteria and Benchmarks for Achieving the Goal of Ending Youth Homelessness Part 1: : Understanding the Crit iteria and Benchmarks May 24, 2018 Panelists Katy Miller Regional Coordinator Jasmine Hayes Deputy Director Katie

597 views • 32 slides
FOUNDA TION FINANCE COMMITTEE September 23, 2016 ULF Performance vs. Peers and Benchmarks 2

FOUNDA TION FINANCE COMMITTEE September 23, 2016 ULF Performance vs. Peers and Benchmarks 2

FOUNDA TION FINANCE COMMITTEE September 23, 2016 ULF Performance vs. Peers and Benchmarks 2 ULF Performance vs. Selected Benchmarks 3 Erroneous Comparisons UK 46% increase between 2006 and 2015 UK moved $150M of hospital funds

336 views • 30 slides
The HPC Challenge Benchmarks and the PMaC project Certificates of relevance for benchmarks

The HPC Challenge Benchmarks and the PMaC project Certificates of relevance for benchmarks

The HPC Challenge Benchmarks and the PMaC project Certificates of relevance for benchmarks Certificates of relevance for benchmarks Do they cover a useful performance space? Do they cover a useful performance space? Do they

525 views • 4 slides
Ev Evaluation Benchmarks and Learning Criteria fo for Di Discou ourse-Aw Aware Sente ntence

Ev Evaluation Benchmarks and Learning Criteria fo for Di Discou ourse-Aw Aware Sente ntence

Ev Evaluation Benchmarks and Learning Criteria fo for Di Discou ourse-Aw Aware Sente ntence Represent ntations Mingda Chen Joint work with Zewei Chu and Kevin Gimpel Prior work on evaluation benchmarks Focus on capabilities of

650 views • 39 slides
Development of AI has been driven by benchmarks and datasets. Computer Vision: (Russakovsky et

Development of AI has been driven by benchmarks and datasets. Computer Vision: (Russakovsky et

Adversarial NLI: A New Benchmark for Natural Language Understanding Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, Douwe Kiela UNC Chapel Hill & Facebook AI Research 1 Development of AI has been driven by benchmarks and

851 views • 45 slides
Lessons from the Tevatron and QCD/SM benchmarks for the LHC Re-discovering the SM at the

Lessons from the Tevatron and QCD/SM benchmarks for the LHC Re-discovering the SM at the

Lessons from the Tevatron and QCD/SM benchmarks for the LHC Re-discovering the SM at the LHC Joey Huston Michigan State University LISHEP 2006 Albert and I also make up the experimental CDF group at Durham. Tevatron by this

931 views • 82 slides
BUILDING ON FIRM FOUNDATIONS DELIVERING A SUSTAINABLE FUTURE ENHANCING OUTCOMES KEY FEATURES

BUILDING ON FIRM FOUNDATIONS DELIVERING A SUSTAINABLE FUTURE ENHANCING OUTCOMES KEY FEATURES

KUMBA IRON ORE LIMITED AUDITED ANNUAL RESULTS FOR THE YEAR ENDED 31 DECEMBER 2017 BUILDING ON FIRM FOUNDATIONS DELIVERING A SUSTAINABLE FUTURE ENHANCING OUTCOMES KEY FEATURES Material improvement IN ALL KEY SAFETY BENCHMARKS AND NO FATAL

610 views • 40 slides
Construction of Realistic Gate Construction of Realistic Gate Sizing Benchmarks Sizing

Construction of Realistic Gate Construction of Realistic Gate Sizing Benchmarks Sizing

Construction of Realistic Gate Construction of Realistic Gate Sizing Benchmarks Sizing Benchmarks With Known Optimal Solutions With Known Optimal Solutions Andrew B. Kahng, Seokhyeong Kang VLSI CAD LABORATORY, UC San Diego International

501 views • 26 slides
WCET Tool Challenge 2014 Outline 1. Objectives of the challenge 2. Benchmarks and problems 3.

WCET Tool Challenge 2014 Outline 1. Objectives of the challenge 2. Benchmarks and problems 3.

WCET Tool Challenge 2014 Outline 1. Objectives of the challenge 2. Benchmarks and problems 3. Participants 4. State of progress / next? Objectives a set of benchmarks and problems to: o compare the properties of different WCET tools

355 views • 20 slides
Second Quarter Results 2010 Zurich July 22, 2010 Cautionary statement Cautionary statement

Second Quarter Results 2010 Zurich July 22, 2010 Cautionary statement Cautionary statement

Second Quarter Results 2010 Zurich July 22, 2010 Cautionary statement Cautionary statement regarding forward-looking and non-GAAP information This presentation contains forward-looking statements within the meaning of the Private Securities

653 views • 44 slides
I nformation Technology Advisory Committee (I TAC) Public Business Meeting December 2, 2016 In

I nformation Technology Advisory Committee (I TAC) Public Business Meeting December 2, 2016 In

I nformation Technology Advisory Committee (I TAC) Public Business Meeting December 2, 2016 In Person Hon. Sheila F . Hanson Chair, Information Technology Advisory Committee 1 Administrative Matters Open Meeting I. Call to Order, Roll

796 views • 42 slides
P R E S E N T A T I O N T O T H E B R O O K I N G S I N S T I T U T I O N The Future of

P R E S E N T A T I O N T O T H E B R O O K I N G S I N S T I T U T I O N The Future of

P R E S E N T A T I O N T O T H E B R O O K I N G S I N S T I T U T I O N The Future of the US Financial Sector June 2015 S T R I C T L Y P R I V A T E A N D C O N F I D E N T I A L Bank lending and capital markets: US vs. EU

373 views • 13 slides
The Carriers Perspective Transforming Insurance Operations Simon Squires - CPO Regional Manager

The Carriers Perspective Transforming Insurance Operations Simon Squires - CPO Regional Manager

The Carriers Perspective Transforming Insurance Operations Simon Squires - CPO Regional Manager International P&C, XL Insurance March 2015 24pt 20pt The London Insurance Company Market al t 18pt 18pt 16pt o IUA 40 members plus

491 views • 16 slides
e-Business e Business Michael P Papazoglou Michael P. Papazoglou and Pieter M.A. Ribbers

e-Business e Business Michael P Papazoglou Michael P. Papazoglou and Pieter M.A. Ribbers

e-Business e Business Michael P Papazoglou Michael P. Papazoglou and Pieter M.A. Ribbers 0-470-84376-4 www.wiley.com/go/ebusiness 1 Chapter 1 The World of E-Business Definitions Definitions Electronic Commerce Electronic-Commerce

465 views • 10 slides
Electronic Commerce Ali Kamandi Sep. 2007 kamandi@sharif.edu Sharif University of Technology

Electronic Commerce Ali Kamandi Sep. 2007 kamandi@sharif.edu Sharif University of Technology

Electronic Commerce Ali Kamandi Sep. 2007 kamandi@sharif.edu Sharif University of Technology Outline (1) Part 1- Introduction to EC Chapters 1,2 An overview of todays business environment, fundamental of EC, EC Terminology

336 views • 8 slides
Ch1. Introduction to Globalization 1 F D M 4 7 1 I N T E R N A T I O N A L A P P A R E L T R A

Ch1. Introduction to Globalization 1 F D M 4 7 1 I N T E R N A T I O N A L A P P A R E L T R A

Ch1. Introduction to Globalization 1 F D M 4 7 1 I N T E R N A T I O N A L A P P A R E L T R A D E I S S U E S Y O U N G J I N B A H N G , P H . D . U N I V E R S I T Y O F H A W A I I A T M A N O A Overview of chapter 1

553 views • 17 slides
Open eBusiness Ontology Usage: Investigating Community Implementation of GoodRelations >>

Open eBusiness Ontology Usage: Investigating Community Implementation of GoodRelations >>

Open eBusiness Ontology Usage: Investigating Community Implementation of GoodRelations >> LDOW2011 Jamshaid Ashraf, Richard Cyganiak, Sean ORiain, Maja Hadzic Building the Web of Linked Data Think Design Prototype Deploy

614 views • 27 slides

More recommend