SLIDE 1
Production: Designing for Testability
Michael Bryzek
mike@flow.io / @mbryzek Cofounder / CTO Flow Cofounder / ex-CTO Gilt
Production: Designing for Testability Michael Bryzek mike@flow.io - - PowerPoint PPT Presentation
Production: Designing for Testability Michael Bryzek mike@flow.io - - PowerPoint PPT Presentation
Production: Designing for Testability Michael Bryzek mike@flow.io / @mbryzek Cofounder / CTO Flow Cofounder / ex-CTO Gilt Think about something you deployed recently to production Is It Working? Right now in production? How do you know?
SLIDE 2
SLIDE 3
Is It Working?
SLIDE 4
Right now in production?
SLIDE 5
How do you know?
SLIDE 6
Feeling anxious?
SLIDE 7
7
Let’s remove that anxiety
SLIDE 8
About Me
www.flow.io www.gilt.com
SLIDE 9
Software Quality is Hard
Think end to end for entire lifecycle of code Verification in Production is a powerful technique to help us build quality software
SLIDE 10
True continuous delivery No staging environments Don’t run code locally
Life at Flow: Delivering Quality Software
SLIDE 11
True Continuous Delivery
Automated tests / No safeguards 1 way to do something Assume Continuous Delivery in Design Process
github.com/flowcommerce/delta
SLIDE 12
“I love my staging environment”
Said Nobody Ever
SLIDE 13
No Staging Environments
Bottlenecks Fragile Difficult to understand failure Expensive (30-40% of budget common) Create the wrong incentives
SLIDE 14
Don’t run code locally
If unsure, write the test! Learn to trust your tests
SLIDE 15
Quality Through Architecture
Extreme Isolation
- Rich event streams
- Own DNS, load balancer
- Private database
- No consul/zk/shared state
- Stop cascading failures
- “Delay” not “Outage”
SLIDE 16
Let’s look at real examples Successfully “testing in production”
SLIDE 17
Example: Know That Checkout Works
Bot places an order every few minutes Identify test orders and immediately cancel
SLIDE 18
Example: Support “Sandbox” Accounts
”SaaS” – even for internal accounts Mark individual accounts as sandbox One API Key for all sandbox accounts
"every service is a third party"
SLIDE 19
Example: End to End Integration Tests
Create Sandbox Org Run tests Delete Sandbox Org
“Safe and Repeatable”
SLIDE 20
Example: Using Sandbox Account for Test Orders
SLIDE 21
Example: Verifying Proxy Server Works as Expected
https://github.com/flowvault/proxy
SLIDE 22
Operating As Expected
SLIDE 23
But sometimes things go wrong Even to the best of teams
SLIDE 24
Considerations
Make production access explicit (not the default) Use defined paths (e.g. API calls) Restrict sensitive data Design for side effects
SLIDE 25
Unexpected Benefits
SLIDE 26
Perfect Documentation
SLIDE 27
Capture request/response of API Calls
SLIDE 28
Tooling: API | Builder
(formerly known as apidoc) 28
www.apibuilder.io Version control for APIs Backwards Compatibility High Quality Mocks
SLIDE 29
High Quality Mocks – From Specs
Full Mock Generated Implement Only What You Need To Test www.apibuilder.io
SLIDE 30
Tooling: Real Time DB Monitoring
https://www.vividcortex.com/blog/2014/04/22/visualizing-impact-index-change/
SLIDE 31
Tooling: Super Simple Alerts from Log
Log a prefix Schedule a real time alert
SLIDE 32
Key Takeaways – Design Production to be Testable
Trust your tests, run subset in production Invest in continuous delivery Sandbox accounts are powerful High quality, trustworthy mocks Real-time feedback from production
SLIDE 33
Thank You!
We’re hiring: https://www.flow.io/careers Michael Bryzek
mike@flow.io / @mbryzek Cofounder / CTO Flow Cofounder / ex-CTO Gilt