What we learned from Community Metrics Agenda Why are metrics - PowerPoint PPT Presentation

What we learned from Community Metrics

Agenda • Why are metrics used? • How metrics are used in two Open Source communities • Common pitfalls and ways to avoid them

Why use Metrics? • Transparency • Who is contributing to the community? • Where (both organizationally & geographically) are the contributions coming from? • Help identify potential bottlenecks • Are code reviews being done in a reasonable timeframe? • Are bugs being closed? • Are new community members able to quickly participate? • Encourage community participation

How Metrics are used in OpenStack • Yearly and per-cycle reviews • Basic metrics • Trend analysis • Published activity metrics • Extracting data from tools • Target for gaming

How Metrics are used in OPNFV Quarterly review of: ● Review Efficiency ● Time to Merge ● Backlog Management

Pitfalls and unintended consequences... • “We’re one of the top X contributing organization in Project ABC!” • “Project XYZ is one of the fastest growing projects based on …” • People trying to game their contribution statistics • e.g. submitting multiple patches for simple changes (in order to maximize patch counts) • Too much focus on code contributions

Pitfalls and unintended consequences... • Making comparisons between different communities based on a few metrics • Or even making historical comparisons for the same community • Ignoring non-metrics “Those who believe that what you cannot quantify does not exist also believe that what you can quantify does” --Aaron Haspel

Some of the flaws of metrics • People often measure the most easily measurable • Focusing on input vs. outcomes • Less insight through standardization • Ignores intrinsic motivation

Recommendations for metrics in open source communities • Metrics should NOT be used as a basis to reward people • Lot of research questioning pay-for-performance as this has an effect of reducing intrinsic motivation • Consideration for using metrics for internal monitoring vs. external purposes (e.g. reward/punishment) • Using metrics as a basis for reward will likely increase the likelihood of “gaming” • Metrics are better used for identifying outliers (e.g. poor performers)

Recommendations for metrics in open source communities • Should be developed from the bottom up with community input • There shouldn’t be metrics people vs. the rest of the community • Metrics should lead to informed interpretations and judgement • Should be conducted by people who are familiar with the environment/community and can compare to previous conditions • Hard part is knowing what metrics are meaningful and what they mean • “Measurement demands judgement”

Parting questions/thoughts • What metrics make sense for our community? • Are we looking at everything that matters? • What are the numbers telling us and how should they be presented? • What about things that can not be counted? • Some factors are beyond your control (incl. metrics)

“Extrinsic rewards become an important determinant of job satisfaction only among workers for whom intrinsic rewards are relatively unavailable” ➢ Barry Gruenberg: “The Happy Worker: An Analysis of Educational and Occupational Differences in Determinants of Job Satisfaction,“ American Journal of Sociology 86 (1980), pp247-71