SLIDE 1
Running Unit Test on Serverless Infrastructure
Adinata Thayib @dieend
source: xkcd.com
Running Unit Test on Serverless Infrastructure Adinata Thayib - - PowerPoint PPT Presentation
Running Unit Test on Serverless Infrastructure Adinata Thayib - - PowerPoint PPT Presentation
Running Unit Test on Serverless Infrastructure Adinata Thayib @dieend source: xkcd.com 85% less Running Unit Test on Serverless Infrastructure Adinata Thayib @dieend About Pocket Gems Mobile Game Company War Dragons & Episode
SLIDE 2
SLIDE 3
SLIDE 4
SLIDE 5
SLIDE 6
85% less
SLIDE 7
Running Unit Test on Serverless Infrastructure
Adinata Thayib @dieend
SLIDE 8
About Pocket Gems
- Mobile Game Company
○ War Dragons & Episode
- 10 years old (and hiring!)
- > 4K tests
SLIDE 9
SLIDE 10
Frustration
SLIDE 11
What is the problem?
Running full unit tests suite is slow Is it worth solving?
- Productivity
- Deployment Speed
SLIDE 12
How to make test suite complete faster?
- 1. Have fewer unit tests
SLIDE 13
How to make test suite complete faster?
- 1. Have fewer unit tests
- 2. Divide & Conquer a.k.a Parallelize the unit
tests execution
SLIDE 14
How do we parallelize test execution?
- 1. Run it locally with multiple CPUs
- 2. Run it in multiple machine
- 3. Run it in serverless infrastructure
SLIDE 15
Running locally with multiple CPUs
Nosetests Multiprocess plugin:
- Collect tests into a Queue object
- Configure the child process
- Start process which poll from the test Queue
SLIDE 16
Running locally with multiple CPUs
Collect:
SLIDE 17
Running locally with multiple CPUs
Collect:
The plugin can't know (unless you tell it) if a context fixture can be called many times concurrently (is re-entrant),
- r if it can be shared among tests running in different
- processes. Therefore, if a context has fixtures, the
default behavior is to dispatch the entire suite to a worker as a unit.
SLIDE 18
Running with Multiple Machine
No out of the box python solution. Ruby Alternative (Knapsack Pro - for Ruby and JS). Python WIP
SLIDE 19
Running with Multiple Machine
SLIDE 20
Running with Multiple Machine
SLIDE 21
Designing the solution
Constraint:
- We won't have engineer resources to manage the worker machine full time
Opportunity:
- We use GAE: all the tests don't have external fixtures
- We use GAE: 100% python code and aren't using many binary dependencies
Proposed Solution
- Use AWS Lambda to remove the necessity on managing worker machine
SLIDE 22
Running with serverless infrastructure
SLIDE 23
Running with serverless infrastructure
SLIDE 24
Running with serverless infrastructure
nosetest plugin with custom selector
SLIDE 25
Running with serverless infrastructure
Running Test... thread.join() Parse output
SLIDE 26
What we learned
SLIDE 27
What we learned
1. AWS Lambda Source File Limits 2. Concurrent Executions 3. Network Bandwidth 4. Longest running unit tests is the weakest link 5. Binary Dependencies 6. No re-entrant fixture support
SLIDE 28
After 2 years...
SLIDE 29
~21min
faster
SLIDE 30
> 1000
executions/month
SLIDE 31
> 350hr
engineering hours are saved each month
SLIDE 32
Assuming engineering hour cost $50/month (Bay Area):
Minute saved each execution 21 mins Monthly Execution 1000 execution Hours 350 hr Monthly Saved Engineering Cost ~$17,500
Monthly Impact
SLIDE 33
~$120
monthly cost
SLIDE 34
> $17K
Saving per months
SLIDE 35
Assuming engineering hour cost $50/month (Bay Area): Initial Investment: 4 engineering weeks
Monthly Saved Engineering Cost ~$17,500 Monthly Cost ~$(120) Total ~$17,380 Engineering Cost to Build ~$8,000 Break Even Point < 1 month
SLIDE 36
>70
engineers adopted it
SLIDE 37
Frustration
Others might have the same frustration with you. You might solve everyone's problem.
SLIDE 38
Channel Discussion: #talk-unit-test-serverless Send me a tweet: @dieend
Q&A
SLIDE 39
Thank you
Channel Discussion: #talk-unit-test-serverless Send me a tweet: @dieend