Deploy Early, Deploy Often, Deploy Safely Andy Lowe From User - - PowerPoint PPT Presentation

Deploy Early, Deploy Often, Deploy Safely

Andy Lowe

From User Story to Production Feature

@LoweKeyOne #AgileAZ Basic Process…

Write Some Code Commit UAT Deploy Acceptance Prod Deploy

Then There’s All The Other Stuff…

Staging Deploy Code Review Security Scans Change Management OSS License Review Performance Testing Integration Deploy Integration Testing Automated Tests

Why Automate Deployments?

• Multiple steps that happen exactly the same way each time
• May need to deploy to multiple systems
• Automated steps far easier to test

Manual deploys will go badly, your team will work nights and weekends, your users will be upset, and your business will be negatively impacted.

@LoweKeyOne #AgileAZ

Would you rather fix an incident…

• By manually deploying
• Changing configs by hand on

multiple servers

• Documenting later what

changed

• Without security controls

@LoweKeyOne #AgileAZ

• Deploying using the same

process you always do

• With automated rollback
• Using a fully tested and

understood process

• With your usual controls

Or…

Deploying fixes can look just like deploying any other change. Always have a widely understood exception process for any controls that may be in place.

How Often Do You Deploy?

Do you want to not hit your SLA because of maintenance windows?

@LoweKeyOne #AgileAZ

SLA Allowed Yearly Downtime (minutes)

Max Deploy Duration (minutes) Once per Month Once per Week Once per Day 99% 5256 438 105.12 30.918 99.9% 525.6 43.8 10.512 3.092 99.99% 52.56 4.38 1.051 0.309 99.999% 5.256 0.438 0.105 0.031 99.9999% 0.526 0.044 0.011 0.003

Blue/Green Deploys

• Zero downtime deployments of a new version of an app.
• Rollback strategy is simple.
• Requires the hardware resources to have two versions running

simultaneously.

@LoweKeyOne #AgileAZ

Blue/Green Deploys

@LoweKeyOne #AgileAZ

Your app is running on on your servers.

MyBlueApp Router

MyBlueApp.Example.com MyApp.Example.com

Blue/Green Deploys

@LoweKeyOne #AgileAZ

Deploy a new version.

MyBlueApp Router

MyBlueApp.Example.com MyApp.Example.com

MyGreenApp

MyGreenApp.Example.com

Blue/Green Deploys

@LoweKeyOne #AgileAZ

Update main route to point both apps.

MyBlueApp MyGreenApp Router

MyBlueApp.Example.com MyGreenApp.Example.com MyApp.Example.com

Blue/Green Deploys

@LoweKeyOne #AgileAZ

Remove main route from blue app.

MyGreenApp Router

MyGreenApp.Example.com MyApp.Example.com

Blue/Green Deploys

@LoweKeyOne #AgileAZ

Delete old version and route.

MyGreenApp Router

MyGreenApp.Example.com MyApp.Example.com

Blue/Green Deploys

$ cf push MyBlueApp –n MyApp $ cf push MyGreenApp –n MyGreenApp $ cf map-route MyGreenApp example.com –n MyApp $ cf unmap-route MyBlueApp example.com –n MyApp $ cf delete MyBlueApp

@LoweKeyOne #AgileAZ

Multi-Availability Zone Deploys – Baseline

@LoweKeyOne #AgileAZ MyBlueApp Load Balancer MyBlueApp Router Router Router MyBlueApp Router MyBlueApp

@LoweKeyOne #AgileAZ

Multi-Availability Zone Deploys – Push Green

MyBlueApp Load Balancer MyBlueApp Router Router Router MyBlueApp Router MyBlueApp MyGreenApp MyGreenApp MyGreenApp MyGreenApp

@LoweKeyOne #AgileAZ

Multi-Availability Zone Deploys – Map New Routes

MyBlueApp Load Balancer MyBlueApp Router Router Router MyBlueApp Router MyBlueApp MyGreenApp MyGreenApp MyGreenApp MyGreenApp

@LoweKeyOne #AgileAZ

Multi-Availability Zone Deploys – Unmap Routes

MyBlueApp Load Balancer MyBlueApp Router Router Router MyBlueApp Router MyBlueApp MyGreenApp MyGreenApp MyGreenApp MyGreenApp

@LoweKeyOne #AgileAZ

Multi-Availability Zone Deploys – Delete Blue

Load Balancer Router Router Router Router MyGreenApp MyGreenApp MyGreenApp MyGreenApp

Health Check That

@LoweKeyOne #AgileAZ

Aliveness checks

• Check if app is responding
• Should be very fast
• Run on startup and every 30

seconds

• TCP, Process, and http checks

Functional Checks

• Check if app is functional
• Should be fast
• Check for connections to

external systems, environment validity, basic functionality

• Run on deploy and as part of

monitoring solution

Health Checks During Deploy

@LoweKeyOne #AgileAZ MyBlueApp Load Balancer MyBlueApp Router Router Router MyBlueApp Router MyBlueApp MyGreenApp MyGreenApp MyGreenApp MyGreenApp HealthChecker

Health Check Each Green App

@LoweKeyOne #AgileAZ MyBlueApp Load Balancer MyBlueApp Router Router Router MyBlueApp Router MyBlueApp MyGreenApp MyGreenApp MyGreenApp MyGreenApp HealthChecker

Health Check Failed - Rollback

@LoweKeyOne #AgileAZ MyBlueApp Load Balancer MyBlueApp Router Router Router MyBlueApp Router MyBlueApp HealthChecker

@LoweKeyOne #AgileAZ

Health Check Succeeded - Continue

Load Balancer Router Router Router Router MyGreenApp MyGreenApp MyGreenApp MyGreenApp HealthChecker

Units of Deployment

SomeOtherSystem has to support MyBlueApp and MyGreenApp simultaneously. Options: 1. Have immutable API. 2. Have versioned API with support for two versions. 3. Combine deployment of new version of SomeOtherSystem with MyGreenApp.

@LoweKeyOne #AgileAZ Router MyGreenApp MyBlueApp SomeOtherSystem

Devs Are More Focused Now…

@LoweKeyOne #AgileAZ Basic Process…

Write Some Code Commit Acceptance

All The Other Stuff…

Code Review Security Scans Change Management Third Party License Review Performance Testing Integration Testing Automated Tests UAT Deploy Staging Deploy Integration Deploy Prod Deploy

Security Scans

• Static scans should happen for each artifact deployed to

prod

• May happen as part of deploying to lower environments
• Make asynchronous, due to time required
• Active scans, taking care with environment in which they

run

• Set criteria for deploy to succeed
• Start with scan results must exist for prod deploy
• Add criteria around severity of issues over time
• Always have an exception process in place

@LoweKeyOne #AgileAZ

Security Scans Security Scans

OSS Scans

Scan for OSS license compliance

• Are you using GPL or LGPL?
• What about BSD, MIT, WTFPL, APL, others?

Know which libraries are in use

• Are you using outdated versions with security issues?
• When a zero-day comes out, know which apps to patch

@LoweKeyOne #AgileAZ

Third Party License Review Third Party License Review

Change Management

Automated deployments mean automatically you know:

• Who initiated the deployment
• What artifacts and what configuration
• When the deploy happens
• Health checks on deploy give you deployment

validation.

@LoweKeyOne #AgileAZ

Change Management Change Management

Even More Focus

@LoweKeyOne #AgileAZ Basic Process…

Write Some Code Commit Automated Tests Dev Deploy Acceptance Prod Deploy

All The Other Stuff…

UAT Deploy Code Review Security Scans Change Management Third Party License Review Performance Testing Integration Deploy Integration Testing

Why Does All This Matter?

@LoweKeyOne #AgileAZ

• This is real pain felt across the industry.
• Freezes are a legitimate business

decision.

• All those freezes leave about 170 days
• n which you can deploy. Make the

most of them.

Case Study: Allstate

Need: Centralized, opinionated deployment tool for deployments

• Deployadactyl – Open source tool for multi-available zone blue/green

deploys

• Conveyor – Allstate-specific deployment logic
• Metrics
• Change management
• Security scans
• Compliance checks

@LoweKeyOne #AgileAZ

Case Study: Allstate

Dev UAT Int Staging Prod Dev UAT Int Staging Prod

• Pipelines include a full automated test suite run prior to publishing to

artifact repository and deployment.

• Dev pipelines run on commit to master in source control.
• Prod pipelines are manually initiated.

Source Control CI/CD Artifact Repository

Case Study: Allstate

@LoweKeyOne #AgileAZ

One month: 87 product teams 4000+ deployments 1000+ production deployments

Easy verification that 32% of prod deploys had security scans. Security scans on deploy success. Easy verification that >90% of prod deploys had security scans.

Teams don’t wait for the weekend to deploy.

Deploy Early, Deploy Often, Deploy Safely

Andy Lowe