Co Conti tinuous De Delivery y with th Co Containers: Th The - - PowerPoint PPT Presentation

▶
co conti tinuous de delivery y with th co containers th
SMART_READER_LITE
LIVE PREVIEW

Co Conti tinuous De Delivery y with th Co Containers: Th The - - PowerPoint PPT Presentation

Aug 29, 2022 208 likes •1.08k views

Co Conti tinuous De Delivery y with th Co Containers: Th The Good, the Bad, and the Ug Ugly ly Daniel Bryant @danielbryantuk Containers: Expectations versus reality DevOps 21/05/2018 @danielbryantuk @danielbryantuk

slide-1
SLIDE 1

Co Conti tinuous De Delivery y with th Co Containers: Th The Good, the Bad, and the Ug Ugly ly

Daniel Bryant @danielbryantuk

slide-2
SLIDE 2

Containers: Expectations versus reality

21/05/2018 @danielbryantuk

“DevOps”

slide-3
SLIDE 3

@danielbryantuk

  • Independent Technical Consultant, Product Architect at Datawire
  • Architecture, DevOps, Java, microservices, cloud, containers
  • Continuous Delivery (CI/CD) advocate
  • Leading change through technology and teams

21/05/2018 @danielbryantuk

slide-4
SLIDE 4

Setting the scene…

  • Continuous delivery is a large topic
  • No business focus today (value stream etc)
  • PaaS and Serverless are super interesting…
  • But I’m assuming you’re all-in on containers
  • Focusing today on the process and tooling
  • No live coding today
  • Mini-book contains more details (thanks nginx!)

21/05/2018 @danielbryantuk

bit.ly/2jWDSF7

slide-5
SLIDE 5

TL;DR – Containers and CD

  • Container image becomes the build pipeline ‘single binary’
  • Adding metadata to containers images is vital, but challenging
  • Must validate container constraints on system quality attributes (NFRs)

21/05/2018 @danielbryantuk

slide-6
SLIDE 6

Continuous Delivery 101

21/05/2018 @danielbryantuk

slide-7
SLIDE 7

Continuous Delivery

  • Produce valuable and robust software in short cycles
  • Optimising for feedback and learning
  • Not (necessarily) Continuous Deployment

21/05/2018 @danielbryantuk

slide-8
SLIDE 8

Velocity (with stability) is key to business success

“Continuous delivery is achieved when stability and speed can satisfy business demand. Discontinuous delivery occurs when stability and speed are insufficient.”

  • Steve Smith (@SteveSmithCD)

21/05/2018 @danielbryantuk

slide-9
SLIDE 9

Creation of a build pipeline is mandatory for continuous delivery

21/05/2018 @danielbryantuk

slide-10
SLIDE 10

21/05/2018 @danielbryantuk

Feedback:

  • Was our initial

hypothesis proven?

  • How can we improve

business, architecture and ops?

slide-11
SLIDE 11

Stability

21/05/2018 @danielbryantuk

slide-12
SLIDE 12

21/05/2018 @danielbryantuk

slide-13
SLIDE 13

Speed

21/05/2018 @danielbryantuk

slide-14
SLIDE 14

21/05/2018 @danielbryantuk

Feedback:

  • Was our initial

hypothesis proven?

  • How can we improve

business, architecture and ops?

slide-15
SLIDE 15

The impact of containers on CD

21/05/2018 @danielbryantuk

slide-16
SLIDE 16

Container technology (and CD)

  • OS-level virtualisation
  • cgroups, namespaces, rootfs
  • Share the OS kernel
  • Package and execute software
  • Container image == ‘single binary’

21/05/2018 @danielbryantuk

slide-17
SLIDE 17

21/05/2018 @danielbryantuk

slide-18
SLIDE 18

21/05/2018 @danielbryantuk

slide-19
SLIDE 19

Should I build my own container platform?

Probably not (Unless you are Google, AWS or IBM) Whatever you decide… push it through a pipeline ASAP!

21/05/2018 @danielbryantuk

slide-20
SLIDE 20

But what about microservices?

21/05/2018 @danielbryantuk

slide-21
SLIDE 21

21/05/2018 @danielbryantuk

Independent service deployment PATTERN

https://www.slideshare.net/dbryant_uk/deliveragile-2018-continuous- delivery-patterns-for-modern-architectures

slide-22
SLIDE 22

Working Locally

21/05/2018 @danielbryantuk

slide-23
SLIDE 23

21/05/2018 @danielbryantuk

slide-24
SLIDE 24

Make your dev environment like production

  • Develop locally or copy/code in container
  • Must build/test containers locally
  • Perform (at least) happy path tests
  • Use identical base images from production
  • With same configuration

21/05/2018 @danielbryantuk

slide-25
SLIDE 25

Quick digression: Working remotely, locally…

21/05/2018 @danielbryantuk https://opencredo.com/working-locally-with-microservices/ https://www.telepresence.io/

slide-26
SLIDE 26

Make your dev environment like production

  • Develop locally or copy/code in container
  • Must build/test containers locally
  • Perform (at least) happy path tests
  • Use identical base images from production
  • With same configuration

21/05/2018 @danielbryantuk

slide-27
SLIDE 27

Lesson learned: Dockerfile content is su super important

  • OS choice (distroless?)
  • Configuration
  • Build artifacts
  • Exposing ports, user
  • Java
  • JDK vs JRE and Oracle vs OpenJDK?
  • Golang
  • Statically compiled binary in scratch?
  • Python
  • Virtualenv?

21/05/2018 @danielbryantuk

slide-28
SLIDE 28

Please talk to the sysadmin people: Their operational knowledge is invaluable

21/05/2018 @danielbryantuk

slide-29
SLIDE 29

Different test and prod containers?

  • Create “test” version of container
  • Full OS (e.g. Ubuntu)
  • Test tools and data
  • Easy to see app/configuration drift
  • Use test sidecar containers instead
  • ONTEST proposal by Alexi Ledenev

21/05/2018 @danielbryantuk

http://blog.terranillius.com/post/docker_testing/

slide-30
SLIDE 30

Docker multi-stage builds

21/05/2018 @danielbryantuk

https://docs.docker.com/develop/develop-images/multistage-build/ https://github.com/GoogleContainerTools/distroless

slide-31
SLIDE 31

Building Artifacts

21/05/2018 @danielbryantuk

slide-32
SLIDE 32

21/05/2018 @danielbryantuk

slide-33
SLIDE 33

Building images with Jenkins

  • My report covers this
  • Build as usual…
  • Build Docker Image
  • Cloudbees Docker Build and Publish Plugin
  • Push image to registry

21/05/2018 @danielbryantuk

slide-34
SLIDE 34

Building in the Cluster?

21/05/2018 @danielbryantuk https://github.com/GoogleContainerTools/kaniko https://www.infoq.com/news/2018/03/jenkins-x-kubernetes

slide-35
SLIDE 35

Deployment

21/05/2018 @danielbryantuk https://blog.hasura.io/draft-vs-gitkube-vs-helm-vs-ksonnet-vs-metaparticle-vs-skaffold-f5aa9561f948

slide-36
SLIDE 36

Building Artifacts: Metadata

21/05/2018 @danielbryantuk

slide-37
SLIDE 37

Lesson learned: Metadata is valuable

  • Application metadata
  • Version / GIT SHA
  • Build metadata
  • Build date
  • Image name
  • Vendor
  • Quality metadata
  • QA control, signed binaries, ephemeral support
  • Security profiles (AppArmor), Security audited etc

21/05/2018 @danielbryantuk

slide-38
SLIDE 38

Metadata – Beware of “latest” Docker Tag

  • Beware of the ‘latest’ Docker tag
  • “Latest” simply means
  • the last build/tag that ran without

a specific tag/version specified

  • Ignore “latest” tag
  • Version your tags, every time
  • danielbryantuk/test:2.4.1

21/05/2018 @danielbryantuk

slide-39
SLIDE 39

Metadata - Adding Labels at build time

  • Docker Labels
  • Add key/value data to image

21/05/2018 @danielbryantuk

slide-40
SLIDE 40

Metadata - Adding Labels at build time

  • Microscaling Systems’ Makefile
  • Labelling automated builds on

DockerHub (h/t Ross Fairbanks)

  • Create file ‘/hooks/build’
  • label-schema.org
  • microbadger.com

21/05/2018 @danielbryantuk

slide-41
SLIDE 41

Metadata - Adding Labels at runtime

21/05/2018 @danielbryantuk

$ docker run -d --label uk.co.danielbryant.lbname=frontdoor nginx

  • Can ’docker commit’, but creates new image
  • Not possible to update running container
  • Docker Proposal: Update labels #21721
slide-42
SLIDE 42

External registry with metadata support

21/05/2018 @danielbryantuk

slide-43
SLIDE 43

New Solution: Grafeas + Kritis

21/05/2018 @danielbryantuk

https://github.com/grafeas/grafeas https://www.infoq.com/news/2018/05/grafeas-kritis-security

slide-44
SLIDE 44

Grafeas Metadata “kinds” (Schema)

21/05/2018 @danielbryantuk

slide-45
SLIDE 45

21/05/2018 @danielbryantuk

slide-46
SLIDE 46

Grafeas Metadata “kinds” (Schema)

21/05/2018 @danielbryantuk

slide-47
SLIDE 47

21/05/2018 @danielbryantuk

slide-48
SLIDE 48

Quality Assurance

21/05/2018 @danielbryantuk

slide-49
SLIDE 49

21/05/2018 @danielbryantuk

slide-50
SLIDE 50

Validating Container Structure

21/05/2018 @danielbryantuk https://github.com/GoogleContainerTools/container-structure-test https://github.com/GoogleContainerTools/container-diff

slide-51
SLIDE 51

Quality Assurance: Functional

21/05/2018 @danielbryantuk

slide-52
SLIDE 52

Testing (in Production??)

21/05/2018 @danielbryantuk martinfowler.com/bliki/TestPyramid.html https://medium.com/@copyconstruct/testing-microservices-the-sane-way-9bb31d158c16

slide-53
SLIDE 53

Component testing

21/05/2018 @danielbryantuk

slide-54
SLIDE 54

Testing: Jenkins Pipeline (as code)

21/05/2018 @danielbryantuk

slide-55
SLIDE 55

21/05/2018 @danielbryantuk

slide-56
SLIDE 56

Testing individual containers

21/05/2018 @danielbryantuk

slide-57
SLIDE 57

Integration testing

21/05/2018 @danielbryantuk

slide-58
SLIDE 58

Introducing Docker Compose

21/05/2018 @danielbryantuk

slide-59
SLIDE 59

Quality Assurance: Nonfunctional

21/05/2018 @danielbryantuk

slide-60
SLIDE 60

Testing NFRs in the build pipeline

  • Architecture
  • Performance and Load testing
  • Gatling / jmeter / Flood.io
  • Security testing
  • Findsecbugs / OWASP Dependency check
  • Bdd-security (OWASP ZAP) / Arachni
  • Gauntlt / Serverspec
  • Docker Bench for Security / CoreOS Clair

21/05/2018 @danielbryantuk

slide-61
SLIDE 61

Architectural Visibility

21/05/2018 @danielbryantuk

slide-62
SLIDE 62

Quick digression: Testing Architecture

21/05/2018 @danielbryantuk https://www.archunit.org/

slide-63
SLIDE 63

Performance/soak testing

21/05/2018 @danielbryantuk

slide-64
SLIDE 64

Mechanical sympathy: Docker and Java

  • Watch for JVM cgroup/taskset awareness (with JDK <= 8)
  • getAvailableProcessors() may incorrectly report the number of cpus in Docker (JDK-8140793)
  • Runtime.availableProcessors() ignores Linux taskset command (JDK-6515172)
  • Default fork/join thread pool sizes (and others) is based from host CPU count
  • Set container memory appropriately
  • JVM requirements = Heap size (Xmx) + Metaspace + JVM overhead
  • Account for native thread requirements e.g. thread stack size (Xss)
  • Entropy
  • Host entropy can soon be exhausted by crypto operations and /dev/random blocks
  • -Djava.security.egd=file:/dev/./urandom (notes on this)

21/05/2018 @danielbryantuk 64

slide-65
SLIDE 65

Security Visibility: Basic Code Scanning

21/05/2018 @danielbryantuk

slide-66
SLIDE 66

Dependency Scanning

21/05/2018 @danielbryantuk

www.owasp.org/index.php/OWASP_Dependency_Check

slide-67
SLIDE 67

Static Image Scanning

21/05/2018 @danielbryantuk

github.com/arminc/clair-scanner

slide-68
SLIDE 68

Verifying Container/Platform Security

21/05/2018 @danielbryantuk https://github.com/docker/docker-bench-security https://github.com/aquasecurity/kube-bench https://github.com/Shopify/kubeaudit

slide-69
SLIDE 69

Delaying NFRs to the ‘Last Responsible Moment’

Newsflash! Sometimes the last responsible moment is up-front Modern platforms/architectures don’t necessarily make this easier

21/05/2018 @danielbryantuk

slide-70
SLIDE 70

Quality Assurance: Post-Deploy

21/05/2018 @danielbryantuk

slide-71
SLIDE 71

Observability is core to continuous delivery

21/05/2018 @danielbryantuk

www.infoq.com/articles/monitoring-containers-at-scale

slide-72
SLIDE 72

Canarying and Synthetic Txns

21/05/2018 @danielbryantuk https://blog.getambassador.io/canary-deployments-a-b-testing-and-microservices-with-ambassador-f104d0458736 https://martinfowler.com/bliki/SyntheticMonitoring.html

slide-73
SLIDE 73

Choas Engineering/Testing

21/05/2018 @danielbryantuk https://github.com/asobti/kube-monkey https://www.infoq.com/news/2018/01/powerfulseal-chaos-kubernetes

slide-74
SLIDE 74

When bad things happen, people are always involved…

21/05/2018 @danielbryantuk | @oakinger

slide-75
SLIDE 75

HealthCare.gov

21/05/2018 @danielbryantuk

slide-76
SLIDE 76

Mikey Dickerson's Hierarchy of Reliability

21/05/2018 @danielbryantuk

www.infoq.com/news/2015/06/too-big-to-fail

slide-77
SLIDE 77

Containers are not a silver bullet

21/05/2018 @danielbryantuk

slide-78
SLIDE 78

Moving to containers: Going all-in?

21/05/2018 @danielbryantuk

OR

slide-79
SLIDE 79

Containerise an existing (monolithic) app?

  • For
  • We know the monolith well
  • Allows homogenization of the

pipeline and deployment platform

  • Can be a demonstrable win for

tech and the business

  • Against
  • Can be difficult (100+ line scripts)
  • Often not designed for operation

within containers, nor cloud native

  • Putting lipstick on a pig?

21/05/2018 @danielbryantuk

slide-80
SLIDE 80

Key lessons learned

  • Conduct an architectural review
  • Architecture for Developers, by Simon Brown
  • Architecture Interview, by Susan Fowler
  • Look for data ingress/egress
  • File system access
  • Support resource constraints/transience
  • Optimise for quick startup and shutdown
  • Evaluate approach to concurrency
  • Store configuration (secrets) remotely

21/05/2018 @danielbryantuk

slide-81
SLIDE 81

Using containers does not get rid of the need for good architectural practices

21/05/2018 @danielbryantuk

slide-82
SLIDE 82

21/05/2018 @danielbryantuk

https://speakerdeck.com/caseywest/containercon-north-america-cloud-anti-patterns

slide-83
SLIDE 83

Summary

21/05/2018 @danielbryantuk

slide-84
SLIDE 84

In summary

  • Continuous delivery is vitally important in modern architectures/ops
  • Container images must be the (single) source of truth within pipeline
  • And metadata added as appropriate…
  • Mechanical sympathy is important (assert properties in the pipeline)
  • Not all developers are operationally aware
  • The tooling is now becoming stable/mature
  • We need to re-apply existing CD practices with new technologies/tooling

21/05/2018 @danielbryantuk

slide-85
SLIDE 85

Thanks for listening…

Twitter: @danielbryantuk Email: daniel.bryant@tai-dev.co.uk

Writing: https://www.infoq.com/profile/Daniel-Bryant Talks: https://www.youtube.com/playlist?list=PLoVYf_0qOYNeBmrpjuBOOAqJnQb3QAEtM

21/05/2018 @danielbryantuk

bit.ly/2jWDSF7 Coming soon!

slide-86
SLIDE 86

Bedtime reading

21/05/2018 @danielbryantuk