Two Years of Work of Paid Contributors in the Debian Long Term - PowerPoint PPT Presentation

Two Years of Work of Paid Contributors in the Debian Long Term Support Project By Raphaël Hertzog <hertzog@debian.org> DebConf 16 / Cape Town / 2016-07-08

Plan of the talk ● Presentation of the LTS project/team ● Workflow of the team: how to contribute ● Statistics about the team ● Changes since last year ● How do we (try to) avoid money-related problems ● Questions ● Feel free to ask questions at any time

Presentation of the LTS project What is LTS about? What were the challenges? Choices made: at the technical level, at the organizational level

What is LTS about ? ● Thus allowing users ● Providing 5 years of to skip a release security support

Initial challenges ● Keeping a distribution secure for 5 years is hard work that is not very rewarding ● The security team ● has limited resources ● aims to support all Debian packages on all release architectures

Technical choices: restrict the perimeter ● Restrict architecture support to amd64, i386, armel and armhf (two more arches than squeeze). ● Exclude some “problematic” packages from security support (much less than squeeze so far): ● chromium-browser, openstack, iceape, libv8, mantis, mediawiki, movabletype-opensource, openjdk-6 (7.x is supported), openswan, redmine, rails 2.x (3.x is supported), sogo, swift, tomcat6 (end of 2016), typo3- src, virtualbox, vlc http://anonscm.debian.org/cgit/collab-maint/debian-security-support.git/tree/security-support-ended.deb7 ●

Organizational choice #1: creation of a new team ● Security team ≠ Debian LTS team ● But members of the security team helped to bootstrap the LTS team ● Different policies ● Different infrastructure ● Mailing list : debian-lts@lists.debian.org https://lists.debian.org/debian-lts/ ● IRC channel: #debian-lts on irc.debian.org (OFTC)

Organizational choice #2: seeking help of companies ● Try to pool the work of companies which were doing in-house long term security support already ➔ Press release to invite companies to join ● Let other organizations fund the project so that Debian contributors can be paid to do the work ➔ https://wiki.debian.org/LTS/Funding lists all ways to help with money ● In practice, most of the (wanting to be) paid contributors joined forces behind a single offer managed by Freexian SARL : https://www.freexian.com/services/debian-lts.html

Freexian's intermediary role

Workflow of the team Triage of security issues Preparation of security update Test of security update Upload and announce of update

Triage of security issues ● Done in the security tracker (common to Debian Security and Debian LTS) https://security-tracker.debian.org/ http://security-team.debian.org/security_track er.html 1.New issues added to data/CVE/list 2.Issues dispatched on source packages 3.Issues reviewed for each release 4.Classification according to analysis

Ways to classify security issues ● Depending on analysis: ➔ Package added to data/dla-needed.txt so that someone will take care of preparing the update (currently <unfixed>) ➔ Issue does not apply (<not-affected>) ➔ Issue ignored because package is not supported (<end-of-life>) ➔ Issue not important enough (<no-dsa>) ➔ Issue already fixed in a former version ● Keep the maintainers in the loop, they can always fix issues (even the non-important ones)

Extract of data/CVE/list CVE-2015-2317 (The utils.http.is_safe_url function in Django…) {DSA-3204-1} - python-django 1.7.7-1 (bug #780873) [squeeze] - python-django <no-dsa> (Minor issue, can wait next security upload) NOTE: https://github.com/django/django/commit/… (1.4.x) CVE-2015-2189 (Off-by-one error in the pcapng_read…) {DSA-3210-1} - wireshark 1.12.1+g01b65bf-4 (bug #780372) [squeeze] - wireshark <not-affected> (Vulnerable code not present) NOTE: https://bugs.wireshark.org/bugzilla/… CVE-2014-9701 [XSS issue in MantisBT permalink_page.php] - mantis <removed> (bug #780875) [wheezy] - mantis <no-dsa> (Minor issue) [squeeze] - mantis <end-of-life> (Unsupported in squeeze- lts) NOTE: Fixed by https://github.com/mantisbt/… (1.2.x)

Preparation of the security update ● Find a patch ● Backport it if required ● Prepare an upload with a “+ deb7uX ” suffix, applying the patch as appropriate ● Document fixed CVE in the changelog and in patch headers

Test the update and upload ● Build and test the result to ensure that ● the package still works ● the fix works as expected ● there's no obvious regression ● If unsure of your update, get in touch: ● Ask others to test ● Seek reviews of your debdiff ● If everything is ok, upload to wheezy-security

Announce the security update ● Prepare a “DLA” (Debian LTS Advisory) $ ./bin/gen-DLA --save expat CVE-2012-6702 CVE-2016-5300 Enter wheezy's version [unset]: 2.1.0-1+deb7u4 DLA text written to ./DLA-508-1 $ svn commit ● Send it to debian-lts-announce@lists.debian.org $ mutt -H DLA-508-1 ● This process updates data/DLA/list which is used by the security tracker to know the CVE fixed by the update

Statistics about the team Who uploaded packages? How did it evolve since the beginning? How is the funding evolving? Data between 2014-06-01 and 2015-07-31

Stats: 549 LTS uploads ● By affiliation: ● By contributor: ● Thorsten Alteholz: 125 (in 24 months) ● Freexian: 380 ● Santiago Ruano Rincon: 51 (13 months) ● Raphaël Hertzog: 40 (16 months) ● None (maintainers) : 88 ● Chris Lamb: 29 (8 months) ● Security team: 32 ● Ben Hutchings: 28 (15 months) ● Holger Levsen: 28 (8 months) ● EDF: 14 ● Markus Koschany: 25 (8 months) ● Mike Gabriel: 23 (11 months) ● credativ: 12 ● Thijs Kinkhorst: 17 (9 months) ● Individuals: 11 ● Guido Günther: 14 (11 months) ● Kurt Roeck: 13 (12 months) ● Toshiba: 7 ● Raphaël Geissert: 13 (6 months) ● Univention: 4 ● Scott Kitterman: 12 (8 months) ● Christoph Berg: 9 (8 months) ● Catalyst: 1 ● …

LTS uploads over time Debian L TS uploads 60 50 Univention 40 T oshiba None Freexian Number of uploads EDF 30 Debian Security Debian LTS credativ Catalyst 20 10 0 2014-07 2014-09 2014-11 2015-01 2015-03 2015-05 2015-07 2015-09 2015-11 2016-01 2016-03 2016-05 2014-06 2014-08 2014-10 2014-12 2015-02 2015-04 2015-06 2015-08 2015-10 2015-12 2016-02 2016-04

Statistics about sponsored hours managed by Freexian ● Sponsors: 46 ● Hours sponsored ● Platinum (>= 24h/month): 1 ● 135 h/month currently ● Gold (>= 8 h/month): 5 dispatched to 11 contributors ● Silver (>= 4 h/month): 10 ● 1854h since the start (740h ● Bronze (>= 1h/month): 22 already paid to be dispatched ● Iron (< 1 h/month): 8 over the next year) ● Average: 2.94 h/month/sponsor Hours sponsored and number of paid contributors (by month) 160 12 140 10 Number of paid contributors 120 8 100 80 6 Hours 60 4 40 2 20 0 0 2014-08 2014-10 2014-12 2015-02 2015-04 2015-06 2015-08 2015-10 2015-12 2016-02 2016-04 2016-06 2014-07 2014-09 2014-11 2015-01 2015-03 2015-05 2015-07 2015-09 2015-11 2016-01 2016-03 2016-05 Hours Nb Paid Contributors

Changes since last year Switch from Squeeze to Wheezy LTS New architectures Working with external partners to support some packages without upstream support

Switch from Squeeze LTS to Wheezy LTS ● No wheezy-lts repository ● We keep using wheezy-security ● No changes for the user ● Changes for the contributors ● More packages supported ● Xen, qemu/qemu-kvm, firefox, icedove, libav, libvirt, zabbix, … ● Made possible by larger amount of sponsorship

New architectures in Wheezy LTS ● armel and armhf ● Requested by new Japanese sponsor: ● Accepted by ftpmasters and buildd maintainers in the last days before the start of Wheezy LTS

Working with external partners ● To support important packages that do not benefit from upstream support (for the version we use in Wheezy) ● External partners: upstream developers that can be contracted, or consultants/companies with expertise on that specific software ● Two such cases currently: ● Xen with credativ (Bastian Blank so far) ● Libav with Diego Biurrun

How do we (try to) avoid money related problems ● Transparency ● External ● Internal ● Open rules to join the set of paid contributors ● Hours allocation rules ● Communication rules ● Point of contact for complaints

External transparency with public monthly reports ● From Freexian: ● How many hours were assigned to contributors ● Links to their respective reports ● Some high level analysis on what happened ● List of sponsors ● Syndicated on Planet Debian ● From paid contributors: ● How many hours they worked and what they did ● On their blog or on the debian-lts mailing list

Internal transparency with legder for hours allocation (1/3) ● Payments from sponsors transformed in work hours assigned to future months (split over all months from the payment period): 2016-06-13 Invoice 201606-063 (Offensive Security) Funded Available:2016:07 2h Available:2016:08 2h Available:2016:09 2h [...]