Two Years of Work of Paid Contributors in the Debian Long Term - - PowerPoint PPT Presentation

▶
two years of work of paid contributors in the debian long
SMART_READER_LITE
LIVE PREVIEW

Two Years of Work of Paid Contributors in the Debian Long Term - - PowerPoint PPT Presentation

Jun 12, 2023 674 likes •1.07k views

Two Years of Work of Paid Contributors in the Debian Long Term Support Project By Raphal Hertzog <hertzog@debian.org> DebConf 16 / Cape Town / 2016-07-08 Plan of the talk Presentation of the LTS project/team Workflow of the

slide-1
SLIDE 1

Two Years of Work of Paid Contributors in the Debian Long Term Support Project

By Raphaël Hertzog <hertzog@debian.org> DebConf 16 / Cape Town / 2016-07-08

slide-2
SLIDE 2

Plan of the talk

  • Presentation of the LTS project/team
  • Workflow of the team: how to contribute
  • Statistics about the team
  • Changes since last year
  • How do we (try to) avoid money-related

problems

  • Questions
  • Feel free to ask questions at any time
slide-3
SLIDE 3

Presentation of the LTS project

What is LTS about? What were the challenges? Choices made: at the technical level, at the

  • rganizational level
slide-4
SLIDE 4

What is LTS about ?

  • Providing 5 years of

security support

  • Thus allowing users

to skip a release

slide-5
SLIDE 5

Initial challenges

  • Keeping a distribution secure for 5 years is

hard work that is not very rewarding

  • The security team
  • has limited resources
  • aims to support all Debian packages on all

release architectures

slide-6
SLIDE 6

Technical choices: restrict the perimeter

  • Restrict architecture support to amd64, i386,

armel and armhf (two more arches than squeeze).

  • Exclude some “problematic” packages from

security support (much less than squeeze so far):

  • chromium-browser, openstack, iceape, libv8, mantis,

mediawiki, movabletype-opensource, openjdk-6 (7.x is supported), openswan, redmine, rails 2.x (3.x is supported), sogo, swift, tomcat6 (end of 2016), typo3- src, virtualbox, vlc

  • http://anonscm.debian.org/cgit/collab-maint/debian-security-support.git/tree/security-support-ended.deb7
slide-7
SLIDE 7

Organizational choice #1: creation of a new team

  • Security team ≠ Debian LTS team
  • But members of the security team helped to

bootstrap the LTS team

  • Different policies
  • Different infrastructure
  • Mailing list : debian-lts@lists.debian.org

https://lists.debian.org/debian-lts/

  • IRC channel: #debian-lts on irc.debian.org

(OFTC)

slide-8
SLIDE 8

Organizational choice #2: seeking help of companies

  • Try to pool the work of companies which were

doing in-house long term security support already

➔ Press release to invite companies to join

  • Let other organizations fund the project so that

Debian contributors can be paid to do the work

➔ https://wiki.debian.org/LTS/Funding lists all ways to

help with money

  • In practice, most of the (wanting to be) paid

contributors joined forces behind a single offer managed by Freexian SARL : https://www.freexian.com/services/debian-lts.html

slide-9
SLIDE 9

Freexian's intermediary role

slide-10
SLIDE 10

Workflow of the team

Triage of security issues Preparation of security update Test of security update Upload and announce of update

slide-11
SLIDE 11

Triage of security issues

  • Done in the security tracker (common to

Debian Security and Debian LTS) https://security-tracker.debian.org/ http://security-team.debian.org/security_track er.html

1.New issues added to data/CVE/list 2.Issues dispatched on source packages 3.Issues reviewed for each release 4.Classification according to analysis

slide-12
SLIDE 12

Ways to classify security issues

  • Depending on analysis:

➔ Package added to data/dla-needed.txt so that

someone will take care of preparing the update (currently <unfixed>)

➔ Issue does not apply (<not-affected>) ➔ Issue ignored because package is not supported

(<end-of-life>)

➔ Issue not important enough (<no-dsa>) ➔ Issue already fixed in a former version

  • Keep the maintainers in the loop, they can always

fix issues (even the non-important ones)

slide-13
SLIDE 13

Extract of data/CVE/list

CVE-2015-2317 (The utils.http.is_safe_url function in Django…) {DSA-3204-1}

  • python-django 1.7.7-1 (bug #780873)

[squeeze] - python-django <no-dsa> (Minor issue, can wait next security upload) NOTE: https://github.com/django/django/commit/… (1.4.x) CVE-2015-2189 (Off-by-one error in the pcapng_read…) {DSA-3210-1}

  • wireshark 1.12.1+g01b65bf-4 (bug #780372)

[squeeze] - wireshark <not-affected> (Vulnerable code not present) NOTE: https://bugs.wireshark.org/bugzilla/… CVE-2014-9701 [XSS issue in MantisBT permalink_page.php]

  • mantis <removed> (bug #780875)

[wheezy] - mantis <no-dsa> (Minor issue) [squeeze] - mantis <end-of-life> (Unsupported in squeeze- lts) NOTE: Fixed by https://github.com/mantisbt/… (1.2.x)

slide-14
SLIDE 14

Preparation of the security update

  • Find a patch
  • Backport it if required
  • Prepare an upload with a “+deb7uX” suffix,

applying the patch as appropriate

  • Document fixed CVE

in the changelog and in patch headers

slide-15
SLIDE 15

Test the update and upload

  • Build and test the result to ensure that
  • the package still works
  • the fix works as expected
  • there's no obvious regression
  • If unsure of your update, get in touch:
  • Ask others to test
  • Seek reviews of your debdiff
  • If everything is ok, upload to wheezy-security
slide-16
SLIDE 16

Announce the security update

  • Prepare a “DLA” (Debian LTS Advisory)

$ ./bin/gen-DLA --save expat CVE-2012-6702 CVE-2016-5300 Enter wheezy's version [unset]: 2.1.0-1+deb7u4 DLA text written to ./DLA-508-1 $ svn commit

  • Send it to

debian-lts-announce@lists.debian.org

$ mutt -H DLA-508-1

  • This process updates data/DLA/list which

is used by the security tracker to know the CVE fixed by the update

slide-17
SLIDE 17

Statistics about the team

Who uploaded packages? How did it evolve since the beginning? How is the funding evolving? Data between 2014-06-01 and 2015-07-31

slide-18
SLIDE 18

Stats: 549 LTS uploads

  • By affiliation:
  • Freexian: 380
  • None (maintainers): 88
  • Security team: 32
  • EDF: 14
  • credativ: 12
  • Individuals: 11
  • Toshiba: 7
  • Univention: 4
  • Catalyst: 1
  • By contributor:
  • Thorsten Alteholz: 125 (in 24 months)
  • Santiago Ruano Rincon: 51 (13 months)
  • Raphaël Hertzog: 40 (16 months)
  • Chris Lamb: 29 (8 months)
  • Ben Hutchings: 28 (15 months)
  • Holger Levsen: 28 (8 months)
  • Markus Koschany: 25 (8 months)
  • Mike Gabriel: 23 (11 months)
  • Thijs Kinkhorst: 17 (9 months)
  • Guido Günther: 14 (11 months)
  • Kurt Roeck: 13 (12 months)
  • Raphaël Geissert: 13 (6 months)
  • Scott Kitterman: 12 (8 months)
  • Christoph Berg: 9 (8 months)
  • …
slide-19
SLIDE 19

LTS uploads over time

2014-06 2014-07 2014-08 2014-09 2014-10 2014-11 2014-12 2015-01 2015-02 2015-03 2015-04 2015-05 2015-06 2015-07 2015-08 2015-09 2015-10 2015-11 2015-12 2016-01 2016-02 2016-03 2016-04 2016-05 10 20 30 40 50 60

Debian L TS uploads

Univention T

  • shiba

None Freexian EDF Debian Security Debian LTS credativ Catalyst

Number of uploads

slide-20
SLIDE 20

Statistics about sponsored hours managed by Freexian

  • Hours sponsored
  • 135 h/month currently

dispatched to 11 contributors

  • 1854h since the start (740h

already paid to be dispatched

  • ver the next year)
  • Sponsors: 46
  • Platinum (>= 24h/month): 1
  • Gold (>= 8 h/month): 5
  • Silver (>= 4 h/month): 10
  • Bronze (>= 1h/month): 22
  • Iron (< 1 h/month): 8
  • Average: 2.94 h/month/sponsor

2014-07 2014-08 2014-09 2014-10 2014-11 2014-12 2015-01 2015-02 2015-03 2015-04 2015-05 2015-06 2015-07 2015-08 2015-09 2015-10 2015-11 2015-12 2016-01 2016-02 2016-03 2016-04 2016-05 2016-06 20 40 60 80 100 120 140 160 2 4 6 8 10 12

Hours sponsored and number of paid contributors (by month)

Hours Nb Paid Contributors

Hours Number of paid contributors

slide-21
SLIDE 21

Changes since last year

Switch from Squeeze to Wheezy LTS New architectures Working with external partners to support some packages without upstream support

slide-22
SLIDE 22

Switch from Squeeze LTS to Wheezy LTS

  • No wheezy-lts repository
  • We keep using wheezy-security
  • No changes for the user
  • Changes for the contributors
  • More packages supported
  • Xen, qemu/qemu-kvm, firefox, icedove, libav,

libvirt, zabbix, …

  • Made possible by larger amount of sponsorship
slide-23
SLIDE 23

New architectures in Wheezy LTS

  • armel and armhf
  • Requested by new Japanese sponsor:
  • Accepted by ftpmasters and buildd

maintainers in the last days before the start

  • f Wheezy LTS
slide-24
SLIDE 24

Working with external partners

  • To support important packages that do not

benefit from upstream support (for the version we use in Wheezy)

  • External partners: upstream developers that

can be contracted, or consultants/companies with expertise on that specific software

  • Two such cases currently:
  • Xen with credativ (Bastian Blank so far)
  • Libav with Diego Biurrun
slide-25
SLIDE 25

How do we (try to) avoid money related problems

  • Transparency
  • External
  • Internal
  • Open rules to join the set of paid contributors
  • Hours allocation rules
  • Communication rules
  • Point of contact for complaints
slide-26
SLIDE 26

External transparency with public monthly reports

  • From Freexian:
  • How many hours were assigned to contributors
  • Links to their respective reports
  • Some high level analysis on what happened
  • List of sponsors
  • Syndicated on Planet Debian
  • From paid contributors:
  • How many hours they worked and what they did
  • On their blog or on the debian-lts mailing list
slide-27
SLIDE 27

Internal transparency with legder for hours allocation (1/3)

  • Payments from sponsors transformed in work

hours assigned to future months (split over all months from the payment period):

2016-06-13 Invoice 201606-063 (Offensive Security) Funded Available:2016:07 2h Available:2016:08 2h Available:2016:09 2h [...]

slide-28
SLIDE 28

Internal transparency with legder for hours allocation (2/3)

  • Work hours dispatched to contributors, once

per month (respecting the limit they set for themselves):

2016-06-01 Distribute work hours for June 2016 Available:2016:06 Contributors:Available:AntoineBeaupre 4.00h Contributors:Available:BalintReczey 16.00h Contributors:Available:BenHutchings 15.00h Contributors:Available:BrianMay 15.00h Contributors:Available:ChrisLamb 18.00h Contributors:Available:EmilioPozueloMonfort 16.00h Contributors:Available:GuidoGuenther 8.00h Contributors:Available:MarkusKoschany 18.75h Contributors:Available:OlaLundqvist 10.00h Contributors:Available:SantiagoRuanoRincon 18.75h Contributors:Available:ThorstenAlteholz 18.75h

slide-29
SLIDE 29

Internal transparency with legder for hours allocation (3/3)

  • Contributors report back work hours

completed over last month

2016-06-07 May hours of Thorsten Alteholz ; Report: http://blog.alteholz.eu/2016/06/my-debian- activities-in-may-2016/ Contributors:Available:ThorstenAlteholz Contributors:Invoiced:ThorstenAlteholz 31.00h 2016-06-10 May hours of Guido Guenther ; Report: http://honk.sigxcpu.org/con/Debian_Fun_in_May_2016.html Contributors:Available:GuidoGuenther Contributors:Invoiced:GuidoGuenther 17.25h

slide-30
SLIDE 30

Open rules to join the set of paid contributors

  • Documented on Freexian's website (from the

start), no arbitrary selection:

  • Debian Developer or Debian Maintainer
  • Prior experience with security updates
  • Good programming skills (multiple languages)
  • Can emit invoices to Freexian
  • Accept the rules

– Privacy of customer data – Public monthly report – Debian code of conduct, obligation to respond to queries – Best effort to meet high quality standards of security team

slide-31
SLIDE 31

Hours allocation rules (1/2)

  • Available hours split evenly across all

contributors (but each contributor can define a maximal number of hours that he wants to be assigned, hence the differences)

  • If we have too many active contributors

(never happened so far), then we would

  • rganize a rotation so that we don't assign

less than 8 hours per contributor.

slide-32
SLIDE 32

Hours allocation rules (2/2)

  • Freexian recruited contributors regularly:
  • No contributor dependant solely on paid LTS

work for their life

  • More redundancy (if contributor goes missing)

2 4 6 8 10 12 14 16 18

Average number of paid hours per contributor

slide-33
SLIDE 33

Communication rules

  • Contributors have to respect the Debian Code of

Conduct

  • Always offer the current package maintainer to

handle LTS updates

  • Temptation to do everything immediately (in particular

when it's easy) because the contributor wants to spend his work hours

  • Obligation to respond to queries from other

Debian contributors

  • Lack of communication is hardly acceptable for

volunteers, it's clearly intolerable for paid contributors

slide-34
SLIDE 34

Point of contact for complaints

  • Clear objective of high-quality work:
  • The website invites everybody to raise their concerns when it's

not the case.

  • I am the point of contact currently:
  • Raphaël Hertzog raphael@freexian.com
  • No official complaints so far.
  • But I complain privately (and politely) to some of the paid

contributors from time to time.

  • About the quality of the work, of the report, about the fact that

they did not use their assigned work hours, etc.

  • Some have stepped down when they realized that they did not

deliver what was expected.

slide-35
SLIDE 35

Lessons learned

  • It's possible to pay Debian contributors without

disrupting the entire community

  • Care must be taken at many levels:
  • To work transparently and in an inclusive way
  • To avoid someone getting locked in a paid position
  • To have fair criteria to use the money or at least a

fair chance of being paid

  • You must be aware that it will have

consequences

  • Change of priorities for some volunteers
slide-36
SLIDE 36

Questions ?

Come to the BoF at 14:00 Using Debian Money to Fund Debian Projects

slide-37
SLIDE 37

Credits & License

  • Content by Raphaël Hertzog

http://raphaelhertzog.com License: GPL-2+

  • Cliparts from https://openclipart.org

License: Public domain

  • OpenOffice.org template by Raphaël Hertzog

http://raphaelhertzog.com/go/ooo-template License: GPL-2+

  • Background image by Alexis Younes “ayo”

http://www.73lab.com License: GPL-2+