How to Make Benchmarking Useful Scott Beamer sbeamer@ucsc.edu - - PowerPoint PPT Presentation

▶

Dec 10, 2023 262 likes •318 views

How to Make Benchmarking Useful Scott Beamer sbeamer@ucsc.edu University of California, Santa Cruz How Representative Should It Be? 2 Ideally - benchmark / proxy app should be very representative thus innovations/results from benchmark

SLIDE 1

How to Make Benchmarking Useful

Scott Beamer sbeamer@ucsc.edu University of California, Santa Cruz

SLIDE 2

How Representative Should It Be?

Ideally - benchmark / proxy app should be very representative … thus innovations/results from benchmark can then directly benefit target application Unfortunately not guaranteed even if close Representativity matters only insofar as results are useful for target domain

2

SLIDE 3

What is the Goal of Benchmarking?

Standardize evaluations Pick winners Track progress over time Guide research by setting target Not automatic - need to be pragmatic and critically assess each case

3

SLIDE 4

Things to Balance for a Benchmark

Size - single problem vs. suite Complexity - simplified vs. full detail Standards - strict rules vs. loose guidelines Longevity - single challenge vs. fixed forever Common tradeoff - simple / flexible may increase adoption but may decrease relevance Room for many benchmarks! (multiple goals)