Introduction FreeBSD: Research results: incremental development - - PowerPoint PPT Presentation

FreeBSD: incremental development and democratic organization

Niels Jørgensen Roskilde University Center nielsj@ruc.dk

Report: www.ruc.dk/~nielsj/research/freebsd/freebsd.pdf Data: www.ruc.dk/~nielsj/research/freebsd/freebsd.html Slides: www.ruc.dk/~nielsj/research/freebsd/slides−dm−10−01−02.pdf

Introduction

Research results:

democracy can be very efficient (delegation of authority) "see bug, fix bug, see bug fixed" is highly motivating

Why FreeBSD ?

there are other projects than Linux + Apache ! Danish members of "core team" (2000) not because of FreeBSD democracy (I didn’t know)

Open source / democracy

source available, license allows use,copy, modify limited financial resources, "alternative" organization but not necessarily "flat" organization

Outline

• 1. Intro
• 2. Incremental development
• 3. FreeBSD background
• 4. FreeBSD’s "Life cycle for changes"
• 5. Case study of development of major new feature
• 6. Conclusion

Note:

generic sw development perspective nothing about DM’s IT−network initiative

What is FreeBSD ?

Unix−type operating system

very similar to Linux runs 15% of Internet−servers worldwide (Yahoo, Hotmail, ..)

Berkely Software Distribution 1977− approx. 90:

key role in early Unix, Internet, and ”Open Source” history good starting point in 90 (when given free by Berkeley U.)

FreeBSD / Linux non−technical differences

elitist image vs. anarchy, tolerance, young nerds, .. lawsuit vs. early 90s jump start copy centrism vs. copy leftism FreeBSD license allows companies to modify

and re−distribute (not "infecting" as GNU license)

FreeBSD / Linux technical differences

Full distribution vs. pure kernel

4000+ pieces of third party software 50+ Java−related ports (compilers, JVMs, JIT−compilers)

Internet server vs. desktop/laptop

project focus on security (eg. security auditing of ports) fewer drivers

I don’t know whether FreeBSD’s −

virtual memory system is more suited for servers ? kernel is more stable ?

Otherwise:

similar basic principles ("Unix−type") shares software for graphical interface, shells, threads, ..

FreeBSD / Linux merge ?

(Not that it is a real issue, but it is tempting to speculate) Pros:

stronger alternative to MS Windows (46%+ of Internet servers) manpower to smoothen installation/configuration

Cons:

alternative ideas can be tried out in practice waste of resources to fight over project decisions software is shared today (eg. LinuxThreads on FreeBSD) not a constant pool of programmes double manpower not same as double output

• 2. Incremental development

Definition Related life cycle models Why study incremental development ?

Definition

Piece−by−piece 1 piece = 1 week of programming/testing, for example Preserve software in sound/working state Main problem: −quality preservation difficult to achieve because the software is constantly changing

Related life cycle models

Incremental development ~ ~ iterative development

but: in Boehm’s Spiral model, analysis and design

is required in each iteration ~ evolutionary prototyping

but: purpose of e.p. is to obtain user feedback on design

~ software development as software maintenance (aka evolutionary system development)

bugfixes and new featueres

are merely different kind of changes

Why study incremental development?

Incremental development appears suitable for open source projects Incremental development is key property of Bazaar Model:

Release early, release often => ”parallel debugging” community of user/developers (or just advanced users)

Incremental development is related to ”flat” organization

long−term planning difficult without sales revenues write and read permissions (check−in, check out) when to integrate: hundreds of decision makers document−based progress assessment impossible

Incremental development challenges classical software engineering

design then code vs. code−n−fix systematic testing vs. ad hoc debugging

• 3. More FreeBSD background

The FreeBSD product & service. The FreeBSD release tree The FreeBSD organization The required design effort

The FreeBSD product & service

Software is Operating system kernel + ports

kernel: state of the art (eg. networking, virtual memory) 4000+ pieces of third party software 50+ Java−related ports (compilers, JVMs, JIT−compilers)

Release 4.2 contains 75 diverse items listed in release notes

adaptive changes: new drivers, ports updates, .. corrective changes: 10 security fixes, .. perfective changes: improved TCP/IP performance, ..

Several security advisories each month

victim program, problem, impact, workaround, fix five versions of Bash shell fix distributed (3 Intel x86, 2 Compaq

Alpha)

The FreeBSD release tree

Major releases along main branch every 18 months (approx.) 4−5 minor releases each year as improvements on latest major release Distribution

major/minor releases on CD snapshots and major/minor releases always downloadable

Major release Minor Snapshot

3.0 4.0 4.1 4.2

The FreeBSD organization

Core team (approx. 10), committers (200+), contributors (1200+) Absolutely necessary to have well−defined and well−understood processes because development branch is ”melting pot” Core team elected by ”committers” for 2 years

”being able to work together is our greatest asset” assigns managers of documentation, PR, web, mailing lists, .. manages access to repository development decisions

Basic organizational ”unit” is maintainer

sources are maintained by formal or ”de facto” maintainer maintainers are responsible for all changes not just corrective committers do corrective, perfective, etc. work interchangeably

The required design effort

"Code, then fix" works for Linux (or FreeBSD) because:

"By the time Linux came around, requirements and architecture

defects had already been flushed out during the development of many previousgenerations of Unix" (Steve McConnell, IEEE Software Magazine) Argument may apply even more to FreeBSD

FreeBSD goes back 20+ years Software that has been widely studied / taught

New requirements: Internet standards and processor technology

some requirements are very demanding symmetric multi−processing but most work on FreeBSD is maintenaince−type of work

• 4. FreeBSD’s "Life cycle for changes"

code review pre−commit test development release parallel debugging production release

I postulate a six phase model

applies to each individual change (bugfix, small feature, part of large feat.) consistent with but not immediately apparent in Handbook, guidelines, etc. model consists of phases + key points for each phase

I believe model captures "essense" of FreeBSD’s development process

why it works model is explaining, captures mechanism not just descriptive

Method

Method

webbased survey involving 72 project participants quantitative + qualitative

Questions designed to verify/falsify (partially)

Eric S. Raymond’s Bazaar model Classical life cycle model Hypotheses about learning

(i) Code

code review pre−commit test development release parallel debugging production release

Code guidelines exist Visibility of code is important:

85% encouraged to improve skills

because of visibility (Q 26)

"Embarrassment is a powerful thing"

(ii) Review

code review pre−commit test development release parallel debugging production release

Reviews strongly suggested but not absolutely required Committers frequently seek and receive code review

85% distributed code for review last 3 months

Main obstacle to more frequent reviews on design is lack of feedback

"Getting design feedback is like pulling teeth"

(iii) Pre−commit test

code review pre−commit test development release parallel debugging production release

The implied responsibility of the committer to integrate changes is a learning incentive

82% had improved their technical skills

by debugging build failures

Pre−commit test is really integration test !

(iv) Development release

code review pre−commit test development release parallel debugging production release

Commit authority is highly motivating

81% encouraged a lot by ability to integrate directly "I don’t feel I’m at the whim of a single person; I can

develop/commit under my own authority, and possibly be overridden by a general consensus"

"There is a tremendous satisfaction to the ’see bug, fix bug,

see bug fix get incorporated so that the fix helps others’ cycle"

(v) Parallel debugging

code review pre−commit test development release parallel debugging production release

Top priority

"If you know of any bugfixes which have been successfully

applied to −CURRENT but have not been merged into − STABLE after a decent interval, send the committer a polite reminder" (Handbook) Produces significant stream of PRs and other feedback But mostly on simple problems

"The bug reports we’ve gotten have been of limited use ..

the subtle problems are too ’unusual’ for most of the other developers to diagnose" (SMP project manager)

(vi) Production release

code review pre−commit test development release parallel debugging production release

One of few deadlines.

95% had no task deadline release 5.0 delayed 1+ year, oops

Prior to major/minor release, authority is delegated to a single individual

committers retain commit privileges "It was a really boring month, but the

4.x release is probably the best we’ve had"

• 5. Case study of a major new feature:

SMP (Symmetric Multi−Processing)

Utilize cheap Intel multiprocessor architecture: 1 cabinet, 1 RAM, X CPUs Operating system kernel (large oval) must be re−entrant

raises many new design problems requires fine−grained locking of kernel ressources (small ovals)

Evolutionary approach to keep kernel working (on all architectures) using temporary ”giant lock” (rectangle) Decision to use development branch (avoid unmanageable, late integration)

unstable development branch

• ngoing integration with other work is extremely difficult

"having people touch the same code is somewhat disruptive"

• 6. Conclusion

Incremental approach successful, model claims reason is:

motivates developers (ability to integrate changes directly) bugfixing is prioritized (maintainer’s mixed responsibility) encourages learning (via visibility and mandatory integration work) responsibility, authority => motivation & learning

SMP subproject also used classical approach:

tasks were broken down and organized into a planned sequence changes were described in separate design documents

SMP subproject faced severe problems:

decreased performance, partly because of extra locking de−stabilized development branch annoying and required extra work feedback from outside subproject was not valuable therefore, incremental life cycle has its limitations