SLIDE 1
Agenda
- Why are metrics used?
- How metrics are used in two Open Source
communities
- Common pitfalls and ways to avoid them
What we learned from Community Metrics Agenda Why are metrics - - PowerPoint PPT Presentation
What we learned from Community Metrics Agenda Why are metrics - - PowerPoint PPT Presentation
What we learned from Community Metrics Agenda Why are metrics used? How metrics are used in two Open Source communities Common pitfalls and ways to avoid them Why use Metrics? Transparency Who is contributing to the
SLIDE 2
SLIDE 3
SLIDE 4
Why use Metrics?
- Transparency
- Who is contributing to the community?
- Where (both organizationally & geographically) are the contributions coming
from?
- Help identify potential bottlenecks
- Are code reviews being done in a reasonable timeframe?
- Are bugs being closed?
- Are new community members able to quickly participate?
- Encourage community participation
SLIDE 5
How Metrics are used in OpenStack
- Yearly and per-cycle reviews
- Basic metrics
- Trend analysis
- Published activity metrics
- Extracting data from tools
- Target for gaming
SLIDE 6
How Metrics are used in OPNFV
Quarterly review of:
- Review Efficiency
- Time to Merge
- Backlog Management
SLIDE 7
SLIDE 8
Pitfalls and unintended consequences...
- “We’re one of the top X contributing organization
in Project ABC!”
- “Project XYZ is one of the fastest growing projects
based on …”
- People trying to game their contribution statistics
- e.g. submitting multiple patches for simple changes (in order to maximize
patch counts)
- Too much focus on code contributions
SLIDE 9
Pitfalls and unintended consequences...
- Making comparisons between
different communities based
- n a few metrics
- Or even making historical comparisons for
the same community
- Ignoring non-metrics
“Those who believe that what you cannot quantify does not exist also believe that what you can quantify does”
- -Aaron Haspel
SLIDE 10
- People often measure the most
easily measurable
- Focusing on input vs. outcomes
- Less insight through
standardization
- Ignores intrinsic motivation
Some of the flaws of metrics
SLIDE 11
Recommendations for metrics in open source communities
- Metrics should NOT be used as a basis to reward
people
- Lot of research questioning pay-for-performance as this has an effect of
reducing intrinsic motivation
- Consideration for using metrics for internal monitoring vs. external purposes
(e.g. reward/punishment)
- Using metrics as a basis for reward will likely increase the likelihood of
“gaming”
- Metrics are better used for identifying outliers (e.g.
poor performers)
SLIDE 12
Recommendations for metrics in open source communities
- Should be developed from the bottom up with
community input
- There shouldn’t be metrics people vs. the rest of the community
- Metrics should lead to informed interpretations and
judgement
- Should be conducted by people who are familiar with the
environment/community and can compare to previous conditions
- Hard part is knowing what metrics are meaningful and what they mean
- “Measurement demands judgement”
SLIDE 13
Parting questions/thoughts
- What metrics make sense for
- ur community?
- Are we looking at everything
that matters?
- What are the numbers telling us
and how should they be presented?
- What about things that can not
be counted?
- Some factors are beyond your
control (incl. metrics)
SLIDE 14