Rise of the Machines Continuous Delivery at SEEK CO PRESENTED BY: - - PowerPoint PPT Presentation

Rise of the Machines

Continuous Delivery at SEEK

CO PRESENTED BY:

CD @ SEEK

“To the cloud”

Scorched Earth Deployments

• Create a pristine environment for every release.
• No need to migrate state of environment.

○ ∴ Configuration Management tools not required.

• State should not be changed at runtime.

○ ∴ Immutable Server

• Blue/Green deployment strategy:

○ Monitor health during deployment. ○ Auto-rollback if new environment is unhealthy.

Automate all the things!

• Delivery team is responsible for the automation of:

○ Server installation ○ Infrastructure creation ○ Deployment

• Automation is impossible without team managed

infrastructure.

Infrastructure as Code

• SEEK use AWS CloudFormation to automate their infrastructure.
• Operations are responsible for the network-level infrastructure,

including: ○ VPC ○ Routing ○ Users and Roles ○ Shared services

• Delivery team is responsible for infrastructure within the VPC, such

as: ○ Elastic Load Balancer ○ Security Group ○ Auto Scaling Group ○ Launch Configuration ○ Instance Profile

Server automation

• Provisioning Linux:

○ SSH ⇒ Easy!

• Provisioning Windows:

○ WinRM ⇒ Hard! ○ AWS makes bootstrapping easier: ■ EC2 User Data ■ cfn-init ○ Packer Windows Plugins can be used to create Windows Amazon Machine Images

• Combination of Packer and CloudFormation can be used regardless
• f target platform

Packaging

• Applications should be self-installable.
• Application installation should be separate from server automation.
• Installation should be independent of Continuous Deployment tool.
• Native package manager is ideal solution:

○ Linux: apt-get or yum ○ Windows: ■ One-Get (available in next version of Windows) ■ Chocolatey (available now!)

• Package manager is platform independent pattern

○ <package-manager> install <package-name>

• Windows Coding Horror:

○ Interacting with Windows resources is not trivial, custom PowerShell modules or PowerShell DSC Resources make this easier.

Pre-Baked Images

• Install a package on a base image.

○ But… don’t include configuration.

• Then, create an image (AMI).
• AMI is our environment independent deployable

artifact.

• Future is containerised...

○ Docker is the next evolution of our approach.

Late-binding of context

• Keep your artifact environment agnostic:

○ Frameworks want you to apply configuration at build or package-time. ○ Deploy tools want to apply configuration at install-time. ○ When deploying to the cloud, you should apply configuration at launch-time.

• When AMI is launched it doesn’t know where it is:

○ Need to inject environment-specific context at launch-time. ○ How do you do this?

Environment Variables

A platform independent way of specifying environment-specific config that’s been around since 1979

• Use environment variables for:

○ DB connection parameters. ○ Location of service endpoints.

• Don’t use them for:

○ Altering runtime behaviour.

• Overriding .NET configuration with environment variables:

○ .NET 4.5.x: write your own Configuration Provider. ○ .NET 4.6: works out of the box. ○ Azure + .NET 4.5.x: works out of the box.

• Service discovery is on the horizon:

○ SEEK is investigating. ○ Externalised config is first step.

Confidence to Release

• Put enough steps and checkpoints in pipeline to get enough

confidence to release.

• Too many checks will result in a slow pipeline and kill the benefits of

Continuous Delivery.

• Take things away until you cry.

○ Then... compensate some other way ○ For example, consumer-driven contract testing makes sure we honour API contracts without the need to integrate.

Ship It!

• Distinction between Deployment and release:

○ Deploy: create Blue environment and deploy application. ○ Release: cut-over from Green to Blue environment.

• Smoke test in Blue environment before release using synthetic

transaction.

• Same smoke test can be used to monitor health after release.
• Choose to enable manual release if change set is risky and additional

manual testing is required.

Towards Polyglot Architecture

• Continuous Delivery + Microservices gives teams the ability to choose

the right stack for the job.

• For SEEK, CD is opening up the possibility migrating legacy apps from

Windows to Linux. ○ Who knows, in the future, running Windows containers will be a viable alternative.

• SEEKs approach to Continuous Delivery provides better logical

separation automation steps.

• Time to first commit still a problem and this will only get worse with

the proliferation of different stacks and tools.

Machine Factories

Manual Development Environments

■ Are a bad experience for your developers. ■ Are time consuming and therefore costly. ■ Make it difficult for devs to move between projects. ■ Are rarely accurately reconstructed. ■ Encourages manual configuration rather than the more reliable and repeatable practice of Infrastructure-as-code. Can lead to environments that have all of the parts, but not quite in the right order.

“It works on my box”

■ Sign of environmental divergence. ■ The Butterfly effect: Minor differences in environment configuration can produce obscure bugs in Prod. ■ This issue is compounded when deployments carried out by another person (e.g. Ops).

Dev vs Production Parity

■ Environment Parity = Confidence. ■ Overcompensate lack of parity with more, possibly manual tests. ■ Environment Parity… ○ enables continuous delivery . ○ requires diligence and automation for accurate replication. ○ encourages infrastructure-as-code.

Wouldn’t it be great if we...

■ Could use declarative formats for setup automation, to minimize time and cost for new developers joining the project. ■ Had a clean contract with the underlying operating system,

• ffering maximum portability between execution

environments.

Wouldn’t it be great if we...

■ Built apps suitable for deployment on modern cloud platforms, obviating the need for servers and systems admin. ■ Could minimize divergence between development and production, enabling continuous deployment. ■ Could scale up without significant changes to tooling, architecture, or development practices.

The “12 Factor App”

■ These 5 principles are what loosely describe the 12 factor App, a guideline for modern SaaS products. ■ 12 Factor Apps, Factor #10: Dev/prod parity ■ “Keep development, staging, and production as similar as possible”. ■ Vagrant can help us solve the environment parity issue.

Vagrant, Docker Compose & Friends

Onward to immutable development environments

Scorched Earth Dev Environments

■ Dev automation typically come in 2 flavours: ○ Use of CM Tools like Puppet, Chef (Boxen, Babushka) ○ Virtual Machine Runners (Vagrant, Docker Compose) ■ CM Tools fraught with challenges: ○ Not isolated ○ Hard to automate across OS/Platforms ○ Doesn’t force developers to automate

What is Vagrant?

“A tool to help create and configure lightweight, reproducible, and portable development environments.” ■ A replacement for manually configured boxes. ○ Enforces infrastructure-as-code. ■ A machine runner (typically a VM or a Container) ■ Separates authoring environment from development runtime ■ Isolated & repeatable

What is Vagrant?

The following is a logical view of how Vagrant interacts with your authoring environment: ■ Vagrant has spun up Virtualbox VM with the name ‘Web’, and is running it on a Mac OSX desktop ■ Code is synchronised from /c/proj/x -> /vagrant of the running environment

What is Vagrant?

■ The Web application spins up a server using the code in /vagrant, which binds to a private network on the VM on port 80 ■ Port 8080 is exposed to the outside ‘world’ (in this case, a Mac OSX desktop) which forwards requests to port 80 in the running VM ■ The Developer is free to author in whichever IDE of her choosing

The TAO of Vagrant

■ vagrant up to get a fully working dev runtime. ■ Development occurs in preferred IDE: Visual Studio, Sublime, ViM, with your terminal, your shortcuts - etc. ■ vagrant destroy && vagrant up will destroy the machine and create a new, clean, isolated environment. ■ Workflow repeatable and transferrable to other projects.

Developing on Ecosystems

From monoliths to ecosystems

From Monolith to Microservices

■ Applications are usually more interesting than a single UI, they are usually connected to things. ○ They form part of a functioning ecosystem. ■ Vagrant good for Monoliths and ecosystems. ■ Explosion of microservices highlights this need.

From Monolith to Microservices

Commentary on Architecture: ■ Multiple components, in disparate technologies ■ Distributed system ■ Layered architecture ■ = complexity Increased complexity increases the number of things that can go wrong. We can combat this with parity and local simulation.

From Monolith to Microservices

■ Vagrant can replicate ecosystems locally, while still having the benefits of isolation, repeatability, immutability etc. ■ Running ecosystems locally gives us flexibility. ■ Vagrant can simulate or test a number of scenarios, for example to: ○ Model distributed systems behaviour ○ Perform disaster testing; outages, network partitions ... ○ Replace components independently ○ Test OS/DB upgrades ○ Run entire production replica ecosystems, locally

Local Ecosystem Setup

The following is a Logical View of this fictional ecosystem with Web and API applications exposed to the ‘World’ and a database hidden ‘behind the firewall’:

Local Ecosystem Setup

■ Note that we are running multiple virtualisation technologies here as if they were all running locally ■ Vagrant will spin up 3 local VMs: ○ Web and API applications exposed to the ‘World’ and a database box hidden ‘behind the firewall’: ■ The 4 VMs are able to communicate through a private network on their native ports (80, 80 and 27017 respectively).

Native to Docker and is fast, but some caveats: ■ Unless on linux, networking is not automatically exposed to the host machine. ■ You need to ensure that your apps are compiled/built within the Docker containers - otherwise you’ll still need to bootstrap your dev box anyway! ■ Hybrid VM/Docker setups not supported.

Docker Compose

Machine Image Pipelines with Packer

Where environments are born

Cattle in Production, Pets elsewhere

■ Dev == Prod - but what about our build/other servers? ■ We don't want cattle in production and pets elsewhere. ■ Quality starts before coding begins ○ Must think about environments we develop, build and deploy to. ■ How do we ensure other environments are configured identically?

• > Immutable servers everywhere!

The Machine Factory

■ How do we go about creating and managing all of these machine images? ■ At SEEK, we needed a way to generate all of these machine images in a sensible, standardised and organised way. ■ Enter: The Machine Factory: ○ “A process for generating machine images for use in different contexts such as development, test, build, prod and so on.” ■ So what does this look like?

[Insert diagram here]

Windows Server 2012

AWS AMI

Machine Factory

■ Assembly-line style manufacturing: ○ Input images (AMI, ISO, Docker Image etc.) ○ Apply Build Steps with shared infrastructure code, using CM tools, shell scripts etc. ○ Output Images (AMI, ISO, Docker etc.) ■ Images put in stasis for their destined environment/context where they will be reanimated. ■ CM tools as a code smell: ○ Your application might be too big. For this step, lightweight options usually suffice.

Tuning the assembling line: Pipelines

■ Booting machines and provisioning from scratch is slow. ■ 2 main strategies: Intermediate images & enrichment. ■ Create intermediate images to shortcut future builds: ○ Install common infra such as log forwarders, statsd, AV, patches/updates etc. ○ Engineering wide (Platform), team wide (Base) or specific Application images.

Tuning the assembling line: Enrichment

■ Prod should have only the runtime dependencies ○ Better Security. ○ Better Performance. ■ Dev will have the most. This can be expressed as: Prod ⊊ Build Server ⊊ Dev ■ Enrich machines from left to right in your pipeline, feeding previous image into next step.

Machine Factory @ SEEK

Walking the walk

Applying CD to Talent Search

■ Talentsearch previously part of .NET monolith. ■ Talent Search is a web-based tool that Recruiters use to connect with candidates, and was previously part of the monolith Warner discussed earlier.

Carving out the relevant bits

■ Isolate scope, cut out vertical slices from the monolith. ■ Enables us to Focus our automation efforts. ■ Broadly this is achieved using 2 approaches: ○ Mocking out downstream APIs for local dev. ○ Consumer Driven Contracts, using Pact. ■ These give us isolation and speed.

Automating Windows

■ Windows tooling support in past year is much improved. ■ AWS & Vagrant all have first class support. ■ Seek have contributed a number of plugins for Vagrant to make Windows development easier, ■ ...a number of useful chocolatey packages, ■ ...and a large suite of DSC resources to assist configuring Windows machines. ■ And then there is Packer...

Automating Windows: Packer

“A tool for creating identical machine images for multiple platforms from a single source configuration” ■ Packer is the core of our Machine Factory. ○ It will boot a machine from an image such as an ISO ○ Configure it with scripts or CM tools. ○ Export it into multiple formats - such as a Vagrant box, Docker image or an AMI. ■ Uses a simple JSON DSL to easily comprehend and visualise the steps applied in the manufacturing process.

Automating Windows: Packer

■ Packer uses SSH, which is very fragile on Windows. ■ SEEK have helped create a port of Packer to communicate natively to Windows over WinRM. ■ The result is Packer Community, a set of plugins for Packer that make Windows machine automation a breeze. ■ Using Packer we have automated: ○

• ur local dev boxes in Vagrant,

○

• ur build/test/prod servers in AWS,

○ ...across Linux, Windows Server and Mac OSX.

#winning

■ Dev env < 10 mins for .NET API + UI ■ Dev -> Build -> Prod from Factory ○ Vagrant: Vbox, Parallels, Mac + Windows ○ AWS ■ Commit to Release in 7 mins. ■ Proven ‘vertical slices’ strategy.

Bonus: Challenges & Lessons on Windows

Tips on effective use of Vagrant, Packer and DSC.

Vagrant

■ Other authoring environment dependencies = anti-pattern. ○ Installing VM provider (e.g. Virtualbox) + Vagrant should be the goal ○ Consider scripting the install (DSC, BoxStarter, Powershell scripts etc.) if you can’t avoid. ■ Shared folders and IIS Applications don’t play nicely ○ Use rsync synced folder type instead ■ When provisioners get too big/slow, move into a Packer script - vagrant up should be fast and reliable

Vagrant

■ Always set $VAGRANT_HOME on multi-partitioned machines due to disk space restrictions (n+1 space requirements) ■ Distributing Images ○ Use Versions + Atlas, if possible ○ S3 also good option ■ GUI Testing (e.g. Protractor tests) ○ Consider having a GUI fallback (at the cost of slightly bigger images) ○ X-Forwarding on *nix hosts not ideal

Vagrant

■ Cross-OS concerns do occasionally come into play: ○ NPM for example has issues writing to a linux networked file system where the underlying host is Windows. ■ Domain access can be problematic and increases complexity - avoid if possible. ○ Use Vagrant Windows Domain plugin if you can’t. ■ Use multiple NICs with private_network if application requires multiple SSL certificates.

Vagrant

Box cleanup: ■ Clean up after yourself and remove any binaries/installers/cruft not required for day-to-day use ■ Defrag and compress virtual disk images if possible ■ Matt Wrock has a number of good tips on how to cut it right down.

Vagrant

Build .NET apps in Vagrant without Visual Studio: ■ Install MS Build Tools and VS Agents ■ Package reference assemblies from VS installation into guest machine. Install the remote debugger and port forward on 4018 to debug from Visual Studio

DSC

■ Expect pain/effort, it’s in its infancy and is immature. ■ Error messages can be very unhelpful/confusing, and you should be prepared to get comfortable with Windows debugging in the event viewer and the CLI. ■ IMHO, DSC is still a better choice than the *nix open source alternatives such as Chef/Ansible (although support is improving).

DSC

■ Use the Vagrant and the DSC Plugin to iterate on a box; Vagrant allows you to test your DSC configurations in a clean environment and test any potential issues with conditions/ordering, without polluting your host environment ■ Prefer the feature rich SEEK-Jobs/DSC community resources

• ver the Powershell team (experimental) xModules

■ Avoid the Script resource - It is rarely a good idea to use it. Write your own custom resource instead.

Packer

■ Don't try to do it all in one go. Build complexity ■ When building a new image, create scripts, run them manually and then automate once you’re comfortable they’ re working as expected. ■ Iteration times can be slow, particularly when booting and installing an OS ○ Create intermediate boxes to build off to avoid that

• verhead.

■ Using Vagrant to test those scripts is often a good starting point.

Packer

■ Sysprep Windows machines so licenses don’t expire. ■ If you’re creating a Windows image, Use Packer Community (only if you can’t use Linux! :)). ■ Don’t forget to disable Windows Updates (at least for dev machines). ■ Windows boxes often need restarting, use the restart- windows Packer Community provisioner to do so safely.

Packer

■ Build Packer images in pipelines - feed output of Packer runs into the start of another Packer run. ■ For example, use the Virtualbox ISO builder to boot and install a Windows machine with updates applied. ■ That box can now be fed into the Virtualbox OVF builder which runs the application specific provisioners (typically this is the more error prone step).

Thank you

@SEEK_Geek

CO PRESENTED BY:

@terrencebenade @dius_au @warnergodfrey @matthewfellows

Resources

■ Vagrant (http://vagrantup.com) ■ The TAO of Vagrant (http://mitchellh.com/the-tao-of-vagrant) ■ Vagrant DSC Plugin (https://github.com/mefellows/vagrant-dsc/) ■ Vagrant Windows Domain Plugin (https://github.com/SEEK- Jobs/vagrant-windows-domain) ■ DSC (http://blogs.technet. com/b/privatecloud/archive/2013/08/30/introducing-powershell- desired-state-configuration-dsc.aspx) ■ Read the free DSC eBook (https://onedrive.live.com/? cid=7f868aa697b937fe&id=7F868AA697B937FE!107) for a small but comprehensive guide to DSC

Resources

■ SEEK-Jobs/DSC resource (https://github.com/SEEK-Jobs/DSC/) ■ Setting up a server for use with DSC (see packer templates https: //github.com/mefellows/packer-community-templates) ■ Packer (http://packer.io) ■ Packer Windows Plugins (aka Packer for Windows https://github. com/packer-community/packer-windows-plugins) ■ Sample Windows Packer Templates (https://github. com/mefellows/packer-community-templates) ■ Legacy Windows Packer Templates and provisioning scripts (Uses OpenSSH, however there is plenty of inspiration to be found - https://github.com/joefitzgerald/packer-windows)