SLIDE 1
Scaling Continuous Deployment @ Etsy
Avleen Vig Staff Operations Engineer @avleen
With much credit: Daniel Schauenberg: (@mrtazz)
@avleen
Scaling Continuous Deployment @ Etsy Avleen Vig Staff Operations - - PowerPoint PPT Presentation
Scaling Continuous Deployment @ Etsy Avleen Vig Staff Operations - - PowerPoint PPT Presentation
Scaling Continuous Deployment @ Etsy Avleen Vig Staff Operations Engineer @avleen With much credit: Daniel Schauenberg: (@mrtazz) Statistics @avleen Statistics @avleen Our application Mostly monolithic @avleen Our application A
SLIDE 2
SLIDE 3
@avleen
Statistics
SLIDE 4
@avleen
Our application
Mostly monolithic
SLIDE 5
@avleen
Our application
A few services too
SLIDE 6
@avleen
Our application
Deploy frequency
SLIDE 7
@avleen
Our team
Before..
SLIDE 8
@avleen
Our team
Today.. ..and that’s just a fraction!
SLIDE 9
@avleen
Deploying code
The push train
Item by decomodwalls
SLIDE 10
@avleen
Deploying code
#push
- IRC channel to organize push trains
- Join a train if you want to deploy
changes
- Schedule is planned via the channel
topic
- First in the train is the driver
SLIDE 11
@avleen
Deploying code
<prod> kseever* + jameslee | jpaul | avleen (c)
#push
SLIDE 12
@avleen
Deploying code
<prod> bateman* + krunal* + enorris* | tristan (c) + jameslee (c) + jlaster (c) | dawa + corey + sandosh + jklein + magera + seth_home + mpascual + nathan | bateman | russp (c)
#push
SLIDE 13
@avleen
Deploying code
Deployinator
SLIDE 14
@avleen
https://github.com/ etsy/deployinator
SLIDE 15
@avleen
Deploying code
So what’s the problem?
SLIDE 16
@avleen
Deploying code
So what’s the problem?
- Deploy-time requests are not
atomic
- Weird limbo while syncing in-
place
- Limits on pushes-per-day
- Long wait times
SLIDE 17
@avleen
Deploying code
Um, limits per day?
- (push_queue_hours * 60)
minutes to deploy
- At 15 mins/deploy, we get ~32
deploys per day - not enough!
SLIDE 18
@avleen
How can we scale it?
Our options:
- More code in each deploy
- Allow concurrent deploys
- Reduce deploy times
- Make deploys atomic
- Fork more concurrent rsyncs
SLIDE 19
@avleen
How can we scale it?
More code in each deploy:
- Also has limits
- How many people can be in
each push?
- We found ~8 to be our limit
for reducing wait times
SLIDE 20
@avleen
How can we scale it?
Allow concurrent deploys:
- For config changes
- Code on independent systems
- The few services we have
SLIDE 21
@avleen
How can we scale it?
Concurrent deploys:
HELLO SPLIT QUEUES HELLO SPLIT QUEUES
SLIDE 22
@avleen
How can we scale it?
Reduce deploy times:
- Tweaks around rsync
- Keep codebase in RAM (tmpfs)
- Increase rsync concurrency
- Reduce timeouts and retry
intervals
SLIDE 23
@avleen
How can we scale it?
Make deploys atomic:
Yin Yang
Active Docroot
SLIDE 24
@avleen
How can we scale it?
Make deploys atomic:
Yin Yang
Active Docroot rsync
SLIDE 25
@avleen
How can we scale it?
Make deploys atomic:
Yin Yang
Active Docroot
SLIDE 26
@avleen
How can we scale it?
Make deploys atomic:
Yin Yang
Active Docroot
SLIDE 27
@avleen
How can we scale it?
Make deploys atomic:
- Not so trivial
- PHP opcache problems
- include_path troubles
- Swapping symlinks mid-request
SLIDE 28
@avleen
http://github.com/ etsy/mod_realdoc
SLIDE 29
@avleen
How can we scale it?
Make deploys atomic, mod_realdoc:
- Apache post_read_request hook
- Whole request works on
realpath of docroot
- Caches realpath for 2s
SLIDE 30
@avleen
http://github.com/ etsy/incpath
SLIDE 31
@avleen
How can we scale it?
Make deploys atomic, incpath:
- PHP extension
- Updates a portion of
include_path
- $_SERVER[“DOCUMENT_ROOT”]
SLIDE 32
@avleen
Infrastructure
SLIDE 33
@avleen
Scaling infrastructure
Before:
Deploy Host
Deployinator
Production Servers
SLIDE 34
@avleen
Scaling infrastructure
After:
Deploy Host
Deployinator
Production Servers Deploy Host
SLIDE 35
@avleen
Results!
SLIDE 36
@avleen
Results!
What did we gain?
- No need to restart apache
- Entire deploy in one push
- Opcode cache stays warm!
SLIDE 37
@avleen
Results!
Push frequency
- (push_queue_hours * 60)
minutes to deploy
- Still ~15mins/deploy:
Much more code going out Tests still run fast Less time waiting to deploy
SLIDE 38
@avleen