Co Conti tinuous De Delivery y with th Co Containers: Th The Good, the Bad, and the Ug Ugly ly Daniel Bryant @danielbryantuk
Containers: Expectations versus reality “DevOps” 21/05/2018 @danielbryantuk
@danielbryantuk • Independent Technical Consultant, Product Architect at Datawire • Architecture, DevOps, Java, microservices, cloud, containers • Continuous Delivery (CI/CD) advocate • Leading change through technology and teams 21/05/2018 @danielbryantuk
Setting the scene… bit.ly/2jWDSF7 • Continuous delivery is a large topic • No business focus today (value stream etc) • PaaS and Serverless are super interesting… • But I’m assuming you’re all-in on containers • Focusing today on the process and tooling • No live coding today • Mini-book contains more details (thanks nginx!) 21/05/2018 @danielbryantuk
TL;DR – Containers and CD • Container image becomes the build pipeline ‘single binary’ • Adding metadata to containers images is vital, but challenging • Must validate container constraints on system quality attributes (NFRs) 21/05/2018 @danielbryantuk
Continuous Delivery 101 21/05/2018 @danielbryantuk
Continuous Delivery • Produce valuable and robust software in short cycles • Optimising for feedback and learning • Not (necessarily) Continuous Deployment 21/05/2018 @danielbryantuk
Velocity (with stability) is key to business success “Continuous delivery is achieved when stability and speed can satisfy business demand. Discontinuous delivery occurs when stability and speed are insufficient.” - Steve Smith (@SteveSmithCD) 21/05/2018 @danielbryantuk
Creation of a build pipeline is mandatory for continuous delivery 21/05/2018 @danielbryantuk
Feedback: - Was our initial hypothesis proven? - How can we improve business, architecture and ops? 21/05/2018 @danielbryantuk
Stability 21/05/2018 @danielbryantuk
21/05/2018 @danielbryantuk
Speed 21/05/2018 @danielbryantuk
Feedback: - Was our initial hypothesis proven? - How can we improve business, architecture and ops? 21/05/2018 @danielbryantuk
The impact of containers on CD 21/05/2018 @danielbryantuk
Container technology (and CD) • OS-level virtualisation • cgroups, namespaces, rootfs • Share the OS kernel • Package and execute software • Container image == ‘single binary’ 21/05/2018 @danielbryantuk
21/05/2018 @danielbryantuk
21/05/2018 @danielbryantuk
Should I build my own container platform? Probably not (Unless you are Google, AWS or IBM) Whatever you decide… push it through a pipeline ASAP! 21/05/2018 @danielbryantuk
But what about microservices? 21/05/2018 @danielbryantuk
PATTERN Independent service deployment https://www.slideshare.net/dbryant_uk/deliveragile-2018-continuous- delivery-patterns-for-modern-architectures 21/05/2018 @danielbryantuk
Working Locally 21/05/2018 @danielbryantuk
21/05/2018 @danielbryantuk
Make your dev environment like production • Develop locally or copy/code in container • Must build/test containers locally • Perform (at least) happy path tests • Use identical base images from production • With same configuration 21/05/2018 @danielbryantuk
Quick digression: Working remotely, locally… https://opencredo.com/working-locally-with-microservices/ https://www.telepresence.io/ 21/05/2018 @danielbryantuk
Make your dev environment like production • Develop locally or copy/code in container • Must build/test containers locally • Perform (at least) happy path tests • Use identical base images from production • With same configuration 21/05/2018 @danielbryantuk
Lesson learned: Dockerfile content is su super important • OS choice (distroless?) • Java • JDK vs JRE and Oracle vs OpenJDK? • Configuration • Golang • Statically compiled binary in scratch? • Build artifacts • Python • Exposing ports, user • Virtualenv? 21/05/2018 @danielbryantuk
Please talk to the sysadmin people: Their operational knowledge is invaluable 21/05/2018 @danielbryantuk
Different test and prod containers? • Create “test” version of container • Full OS (e.g. Ubuntu) • Test tools and data • Easy to see app/configuration drift • Use test sidecar containers instead http://blog.terranillius.com/post/docker_testing/ • ONTEST proposal by Alexi Ledenev 21/05/2018 @danielbryantuk
Docker multi-stage builds https://docs.docker.com/develop/develop-images/multistage-build/ https://github.com/GoogleContainerTools/distroless 21/05/2018 @danielbryantuk
Building Artifacts 21/05/2018 @danielbryantuk
21/05/2018 @danielbryantuk
Building images with Jenkins • My report covers this • Build as usual… • Build Docker Image • Cloudbees Docker Build and Publish Plugin • Push image to registry 21/05/2018 @danielbryantuk
Building in the Cluster? https://github.com/GoogleContainerTools/kaniko https://www.infoq.com/news/2018/03/jenkins-x-kubernetes 21/05/2018 @danielbryantuk
Deployment https://blog.hasura.io/draft-vs-gitkube-vs-helm-vs-ksonnet-vs-metaparticle-vs-skaffold-f5aa9561f948 21/05/2018 @danielbryantuk
Building Artifacts: Metadata 21/05/2018 @danielbryantuk
Lesson learned: Metadata is valuable • Application metadata • Version / GIT SHA • Build metadata • Build date • Image name • Vendor • Quality metadata • QA control, signed binaries, ephemeral support • Security profiles (AppArmor), Security audited etc 21/05/2018 @danielbryantuk
Metadata – Beware of “latest” Docker Tag • Beware of the ‘latest’ Docker tag • “Latest” simply means • the last build/tag that ran without a specific tag/version specified • Ignore “latest” tag • Version your tags, every time • danielbryantuk/test:2.4.1 21/05/2018 @danielbryantuk
Metadata - Adding Labels at build time • Docker Labels • Add key/value data to image 21/05/2018 @danielbryantuk
Metadata - Adding Labels at build time • Microscaling Systems’ Makefile • Labelling automated builds on DockerHub (h/t Ross Fairbanks) • Create file ‘/hooks/build’ • label-schema.org • microbadger.com 21/05/2018 @danielbryantuk
Metadata - Adding Labels at runtime $ docker run -d -- label uk.co.danielbryant.lbname=frontdoor nginx • Can ’docker commit’, but creates new image • Not possible to update running container • Docker Proposal: Update labels #21721 21/05/2018 @danielbryantuk
External registry with metadata support 21/05/2018 @danielbryantuk
New Solution: Grafeas + Kritis https://github.com/grafeas/grafeas https://www.infoq.com/news/2018/05/grafeas-kritis-security 21/05/2018 @danielbryantuk
Grafeas Metadata “kinds” (Schema) 21/05/2018 @danielbryantuk
21/05/2018 @danielbryantuk
Grafeas Metadata “kinds” (Schema) 21/05/2018 @danielbryantuk
21/05/2018 @danielbryantuk
Quality Assurance 21/05/2018 @danielbryantuk
21/05/2018 @danielbryantuk
Validating Container Structure https://github.com/GoogleContainerTools/container-structure-test https://github.com/GoogleContainerTools/container-diff 21/05/2018 @danielbryantuk
Quality Assurance: Functional 21/05/2018 @danielbryantuk
Testing (in Production??) martinfowler.com/bliki/TestPyramid.html https://medium.com/@copyconstruct/testing-microservices-the-sane-way-9bb31d158c16 21/05/2018 @danielbryantuk
Component testing 21/05/2018 @danielbryantuk
Testing: Jenkins Pipeline (as code) 21/05/2018 @danielbryantuk
21/05/2018 @danielbryantuk
Testing individual containers 21/05/2018 @danielbryantuk
Integration testing 21/05/2018 @danielbryantuk
Introducing Docker Compose 21/05/2018 @danielbryantuk
Quality Assurance: Nonfunctional 21/05/2018 @danielbryantuk
Testing NFRs in the build pipeline • Architecture • Performance and Load testing • Gatling / jmeter / Flood.io • Security testing • Findsecbugs / OWASP Dependency check • Bdd-security (OWASP ZAP) / Arachni • Gauntlt / Serverspec • Docker Bench for Security / CoreOS Clair 21/05/2018 @danielbryantuk
Architectural Visibility 21/05/2018 @danielbryantuk
Quick digression: Testing Architecture https://www.archunit.org/ 21/05/2018 @danielbryantuk
Performance/soak testing 21/05/2018 @danielbryantuk
Mechanical sympathy: Docker and Java • Watch for JVM cgroup/taskset awareness (with JDK <= 8) • getAvailableProcessors() may incorrectly report the number of cpus in Docker (JDK-8140793) • Runtime.availableProcessors() ignores Linux taskset command (JDK-6515172) • Default fork/join thread pool sizes (and others) is based from host CPU count • Set container memory appropriately • JVM requirements = Heap size (Xmx) + Metaspace + JVM overhead • Account for native thread requirements e.g. thread stack size (Xss) • Entropy • Host entropy can soon be exhausted by crypto operations and /dev/random blocks • -Djava.security.egd=file:/dev/./urandom (notes on this) 21/05/2018 @danielbryantuk 64
Security Visibility: Basic Code Scanning 21/05/2018 @danielbryantuk
Recommend
More recommend
