SLIDE 1
Netflix and FreeBSD: Using Open Source to Deliver Streaming Video
Jonathan Looney FOSDEM 2019
NETFLIX AND FREEBSD FOSDEM 2019
Netflix and FreeBSD: Using Open Source to Deliver Streaming Video - - PowerPoint PPT Presentation
Netflix and FreeBSD: Using Open Source to Deliver Streaming Video - - PowerPoint PPT Presentation
Netflix and FreeBSD: Using Open Source to Deliver Streaming Video Jonathan Looney FOSDEM 2019 Open Connect NETFLIX AND FREEBSD FOSDEM 2019 Open Connect is Netflixs CDN. It is global, efficient, and purpose-built for distributing
SLIDE 2
SLIDE 3
Open Connect is Netflix’s CDN. It is global, efficient, and purpose-built for distributing Netflix’s content.
NETFLIX AND FREEBSD FOSDEM 2019
SLIDE 4
The Open Connect Appliance
The OCA is the “backbone”
- f the Open Connect
network. The OCA almost exclusively runs
- pen-source software.
NETFLIX AND FREEBSD FOSDEM 2019
40Gb/s Storage Appliance with 248TB storage (2RU form factor)
SLIDE 5
Open Connect Traffic
NETFLIX AND FREEBSD FOSDEM 2019
Video Apps Video Apps Video Apps Video Apps Open Connect
SLIDE 6
Netflix OCA Workload
NETFLIX AND FREEBSD FOSDEM 2019
CPU RAM Disks
Plain-text Data Encrypted Data
NICs
SLIDE 7
Using FreeBSD and commodity parts, we achieve 90 Gb/s serving TLS-encrypted connections with ~55% CPU on a 16-core 2.6-GHz CPU.
NETFLIX AND FREEBSD FOSDEM 2019
SLIDE 8
NETFLIX AND FREEBSD FOSDEM 2019
SLIDE 9
NETFLIX AND FREEBSD FOSDEM 2019
OCA Operating System
SLIDE 10
(Abridged) BSD “Family Tree”
NETFLIX AND FREEBSD FOSDEM 2019
BSD AT&T Unix NetBSD FreeBSD
SLIDE 11
FreeBSD Release Cycle
NETFLIX AND FREEBSD FOSDEM 2019
head stable/11 stable/12 12.0 12.1 11.0 11.1 11.2
?
SLIDE 12
We Track FreeBSD “Head”
NETFLIX AND FREEBSD FOSDEM 2019
FreeBSD head Netflix master Netflix release branches
SLIDE 13
Typical Release Cycle
NETFLIX AND FREEBSD FOSDEM 2019
Five Weeks of Development Five Weeks of Testing/Deployment
FreeBSD Merge Feature Development/Integration Testing Dev Testing Canary Testing Phased Rollout
SLIDE 14
- NUMA enhancements
- Asynchronous sendfile
- Kernel TLS
- Pbuf allocation enhancements
- “Unmapped” mbufs
- I/O scheduling
- TCP algorithms
- TCP logging infrastructure
Examples of Features
NETFLIX AND FREEBSD FOSDEM 2019
SLIDE 15
Tracking “head” lets us stay forward looking and focused on innovation.
NETFLIX AND FREEBSD FOSDEM 2019
SLIDE 16
Downstream users of open-source projects can be stuck in “vicious” or “virtuous” cycles.
NETFLIX AND FREEBSD FOSDEM 2019
SLIDE 17
Vicious Cycle
NETFLIX AND FREEBSD FOSDEM 2019
Infrequent Merges Many Conflicts/ Regressions Slower Feature Velocity
SLIDE 18
Virtuous Cycle
NETFLIX AND FREEBSD FOSDEM 2019
Frequent Merges Few Conflicts/ Regressions Faster Feature Velocity/ Collaboration
SLIDE 19
- Information covered under NDA
- Feature which is still in development/testing
- Feature which needs to be generalized
Reasons We Keep Local Diffs
NETFLIX AND FREEBSD FOSDEM 2019
SLIDE 20
It is our intention to upstream any code which we can.
NETFLIX AND FREEBSD FOSDEM 2019
SLIDE 21
- Quicker feature iteration
- Quicker access to new FreeBSD features
- Quicker bug fixes
- Enables collaboration
- Minimizes merge conflicts
- Amortizes merge “cost”
Benefits to Netflix of Tracking FreeBSD “Head”
NETFLIX AND FREEBSD FOSDEM 2019
SLIDE 22
- Wide deployment of “head” branch code (albeit in a narrow use case)
- Early intensive testing
- Incentive for Netflix to upstream code
Benefits to FreeBSD
NETFLIX AND FREEBSD FOSDEM 2019
SLIDE 23
- It isn’t stable
- Why should you pay to find the bugs others will find while testing head?
- Aren’t there more security bugs?
- No one runs development branches
- Pay monthly “cost” to do merges
- You get new bugs each month
Objections to Running “Development” Code
NETFLIX AND FREEBSD FOSDEM 2019
SLIDE 24
Running FreeBSD “head” lets us deliver large amounts of data to our users very efficiently, while maintaining a high velocity of feature development.
NETFLIX AND FREEBSD FOSDEM 2019
SLIDE 25