From Zero to Secure Continuous Deployment in 60 Minutes Florian - - PowerPoint PPT Presentation

▶
from zero to secure continuous deployment
SMART_READER_LITE
LIVE PREVIEW

From Zero to Secure Continuous Deployment in 60 Minutes Florian - - PowerPoint PPT Presentation

Oct 01, 2023 129 likes •725 views

From Zero to Secure Continuous Deployment in 60 Minutes Florian Barth Matthias Luft 1 o THE Digital Wallet o 20.000.000 users 50 countries 5 offices o 25.000.000,000 rpm (onth ;) ) 100 micro services 10 servers 5 persons doing backend

slide-1
SLIDE 1

1

From Zero to Secure Continuous Deployment in 60 Minutes

Florian Barth Matthias Luft

slide-2
SLIDE 2

2

  • THE Digital Wallet
  • 20.000.000 users

50 countries 5 offices

  • 25.000.000,000 rpm (onth ;) )

100 micro services 10 servers 5 persons doing backend DevOps

slide-3
SLIDE 3

3

  • Vendor-independent
  • Established 2001
  • 65 employees, 42 FTE consultants
  • Continuous growth in revenue/profits
  • No venture/equity capital, no external financial
  • bligations of any kind
  • Customers predominantly large/very large

enterprises

  • Industry, telecommunications, finance
slide-4
SLIDE 4

4

Avengers, assemble!

Captain Security DareDeveloper

slide-5
SLIDE 5

5

I dare you to create an IaC-based automated CD pipeline! Give me 60 minutes! You have 30, as I will need to secure that thing! I’ll do it in 10!

slide-6
SLIDE 6

6

Agenda

  • Intro CD, DevOps, and key technologies
  • “Let‘s get our brains and hands dirty“
  • Sermons and Preaching by Cpt Security
slide-7
SLIDE 7

7

DevOps

“DevOps is the philosophy of unifying Development and Operations at the cu culture re, system, practic ice, and tool levels, to achieve accelerated ated and more freque uent deliv ivery ry of value to the customer, by improving quality in

  • rder to increase velocity.”

Rob England nd, 2014

slide-8
SLIDE 8

8

Continuous Delivery

“Continuous Delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliab iably ly releas ased at any time. It aims at building lding, testin ting, and releasin asing software faster and more freque uently tly.”

DOI: 10.1109/MS.2015.27

slide-9
SLIDE 9

9

Continuous Deployment

… is often confused with Continuous Delivery and “means that every change goes through the pipeline and automat matically ally gets put into production, resulting in many producti uction n deploym loyments nts every day.”

https://ma mart rtin infow

  • wler.com/

r.com/bliki/ iki/Cont

  • ntin

inuou uousDel eliver ivery.ht .html ml

slide-10
SLIDE 10

10

Immutable Infrastructure

  • “Servers are cattle not pets”
slide-11
SLIDE 11

11

Immutable Infrastructure

  • “Servers are cattle not pets”
  • “Servers are syringes

es”

  • Spawn your infra from a template or recipe
  • Update

te, Test it, Spawn wn new, Trash old

slide-12
SLIDE 12

12

Infrastructure-as-Code

  • Manage und provision data

a centers through mach chine ine-readable adable definition files

  • Version control your infrastructure
  • Documentation & Evolution
  • Static/Dynamic analysis
slide-13
SLIDE 13

13

Prepare to be steamrolled…

  • A lot of content and prerequisites
  • We want to explain the actually difficult

parts, not the parts that are routine work.

slide-14
SLIDE 14

14

Goal

  • See the buzzwords in action
  • Gain a general understanding
slide-15
SLIDE 15

15

My First Contact with DevOps

  • Role: Unifying Dev, Ops, Net, Sec, …
  • Scale: users growing exponentially
  • Infra: 5€ “vServer”
  • Stack/Tooling: nginx, php, ssh, vi
  • CD: “:w

:w”

  • Since then we learned a lot!
slide-16
SLIDE 16

16

Ingredients

App Stack Definition Platform Infrastructure Definition App Source Code Software Infrastructure

slide-17
SLIDE 17

17

This is DareDeveloper, Welcome to Infrastructure as Code

slide-18
SLIDE 18

18

slide-19
SLIDE 19

19

Demo

Terraform

slide-20
SLIDE 20

20

This is DareDeveloper, Welcome to App Stack as Code

slide-21
SLIDE 21

21

vote Python queue redis worker .NET db PostgreSQL result node.js sockpuppet node.js

slide-22
SLIDE 22

22

Demo

Docker Compose Service Infra

slide-23
SLIDE 23

23

This is DareDeveloper, Welcome to Full Auto CD

slide-24
SLIDE 24

24

“CI” Service Artifact Repository Execution Environment VCS build & test push

slide-25
SLIDE 25

25

Demo

Gitlab Service Config

slide-26
SLIDE 26

26

Big Picture :tm:

slide-27
SLIDE 27

27

Deployments

  • Deployment Strategies
  • Rolling Deployment
  • Green/Blue Deployment
  • Canary Deployment
  • Feedback from Logging & Metrics
slide-28
SLIDE 28

28

Docker Build

  • Gitlab Runner running in a Docker container
  • Dedicated Docker container for each job in

the pipeline

  • By distributing the runner „swarm-scale“

build cluster can be formed

slide-29
SLIDE 29

29

slide-30
SLIDE 30

30

This is Captain Security, Welcome to Swarm Pwnage!

slide-31
SLIDE 31

31

4444 4445 4446

slide-32
SLIDE 32

32

So Far: Build System Breakout

  • We looked at one particular aspect
  • Raising awareness for capabilities of the

Docker socket

  • Good summary: blog.heroku.com
slide-33
SLIDE 33

33

More Security Challenges

  • Build System Breakout
  • Traversing to Production Swarm
  • Breakout on Production Swarm
  • Deployment of Vulnerable Code to Production
  • Registry Overwrite
  • Leaking of Secrets
slide-34
SLIDE 34

34

Build System Breakout

  • Inherent problem of the requirement to build

docker containers in build containers

  • Only configuration-based review of drone.io,

Travis, and Jenkins, but most likely the same issues

  • Potential Solution:
  • Verify Build Scripts
  • Build System Zoning
  • Docker Image Build Without Docker
slide-35
SLIDE 35

35

Docker Image Build Without Docker

  • Several projects available
  • https://github.com/genuinetools/img
  • https://bazel.build/
  • https://github.com/projectatomic/buildah
  • Need to build a custom build

image/runner/plugin

slide-36
SLIDE 36

36

Build System Breakout – Lateral Movement

  • Common approach: Build system deploys to

production platform

  • If so:
  • Credentials can be accessed after breakout
  • Lateral Movement to target systems
slide-37
SLIDE 37

37

Breakout on Production Swarm

  • Container breakout vulnerabilities are

relevant

  • Cluster escalation vulnerabilities are

relevant

  • Not the scope here!
  • Refer to H2HC 2017 – Pentesting DevOps
slide-38
SLIDE 38

38

Breakout on Production Swarm

  • In scope: Deploying privileged container
  • Docker: privileged: true in compose file
  • Kubernetes: privileged flag in deployment

yml

  • No proper way to restrict container runtime
  • ptions on a cluster level
slide-39
SLIDE 39

39

DoS on Production Swarm

  • Binding container to relevant (host) port
  • docker service create -p 22:22

IMAGE …

  • Potential Solution for both DoS/Breakout:
  • Deployment Verification
slide-40
SLIDE 40

40

Deployment of Vulnerable Images

  • Probably the most popular “Secure Pipeline”

aspect

  • Possibility has existed longer than

containers/DevOps/CD

  • Still a very important aspect
slide-41
SLIDE 41

41

Deployment of Vulnerable Images

  • Challenge: Scan all of
  • OS packages
  • App code
  • App dependencies
  • But again:
  • Build Verification Required
slide-42
SLIDE 42

42

Registry Overwrite

  • Build System must be able to push to

registry

  • Broad registry push access allows to
  • verwrite images
  • => Review your registry

user/role/permission concept!

slide-43
SLIDE 43

43

Leaking of Secrets

  • Builds (and thus repositories) must contain

secrets on a regular basis

  • For example:
  • Registry login credentials
  • Credentials for application stack
  • Secrets in a repository are a bad idea ;-)
slide-44
SLIDE 44

44

Secret Injection

  • All major build systems support secret injection
  • f some kind
  • docker login -u gitlab-ci-token -p

$CI_BUILD_TOKEN $DOCKER_REGISTRY

  • Ensure usage!
  • Scan repos for secret storage
  • Various tools available
  • Support developers
  • Awareness
  • Provide .gitignore/commit hook configs
slide-45
SLIDE 45

45

Validate Builds & Deployments

  • We saw the need for validation to address
  • Build Breakout
  • App/Image Vulnerabilities
  • Deployment Mal-/Mis-Configuration
  • How to enforce validation?
  • Build Systems don’t provide mandatory

check steps without “manual work”…

slide-46
SLIDE 46

46

Architecture Overview

slide-47
SLIDE 47

47

What do we want to scan for?

  • Application Vulnerabilities
  • OS Package and Library Vulnerabilities
  • Build Script Breakout
  • As always, hard to detect.
  • Docker-less builds essential
  • Deployment Mal-/Mis-Configuration
slide-48
SLIDE 48

48

What do we want to scan for?

  • Deployment Mal-/Mis-Configuration
  • Capabilities
  • AppArmor/seccomp profile deactivation
  • Container running as root
  • Secrets
  • Network policies
  • Namespace sharing
  • Mount propagation/Volumes
  • See CIS Docker/K8s benchmarks as well.
slide-49
SLIDE 49

49

Tooling?

  • Some “Container Security” tools/scanners

start to provide the above.

  • … some really do not.
  • No tool recommendation here, no extensive

technical evaluation performed.

  • The slides above provide your evaluation

checklist

slide-50
SLIDE 50

50

Security Zoning

  • Captain Security’s favorite approach:
  • Use dedicated environments per application project/stack/team.
  • Basically remove multi-tenancy from the vulnerability list
  • We saw above how easy it is to deploy the complete

infrastructure.

  • Granularity of zoning depends on your environment.
  • E.g. according to business unites, protection need/classification, …
slide-51
SLIDE 51

51

Network Isolation

  • Old controls still relevant in the old world
  • Build system/production swarm only needs filtered outbound

network access

  • Proxy-based whitelisting
  • Very feasible!
  • Of course no prevention, but raising the bar.
slide-52
SLIDE 52

52

Empower & Collaborate

  • As a security team, provide
  • Training in the new CD approaches
  • Repository scaffolding
  • Commit hooks including checks for included

secrets

  • .gitignore
  • .gitlab-ci.yml including all scanning

checks

  • Provide scanning checks as well ;-)
  • Container security policies (e.g. K8s Pod Security

Policies) available in the cluster

slide-53
SLIDE 53

53

Empower & Collaborate

  • If implemented properly and securely:
  • Immutable infrastructures are immutable
  • And easy to baseline!
  • Executable and version-able

infrastructure/design documentation

  • Automated security checks
  • Timely patching
  • Centralized secret management
slide-54
SLIDE 54

54

Summary

  • Practical demo of a functional CD pipeline
  • Including war stories from production
  • Not just some marketing slides of distant blog

posts from some .io startup.

  • Illustration of multi-tenancy issues
  • Limitations of build systems when it comes

to mandatory checks

slide-55
SLIDE 55

55

Summary

  • Focus on technical aspects
  • Short touch on awareness/motivation via

belts/guilds

  • Extending CD security aspects beyond

“integrate a source code scanner”

  • CD provides awesome possibilities to

improve security!

slide-56
SLIDE 56

56

www.stocard.de www.ernw.de www.insinuator.net

Questions?

Thank you for your attention!

barth@stocard.de mluft@ernw.de @der_cthulhu @uchi_mata

slide-57
SLIDE 57

57

Shoutouts

  • Simon who built a lot of basic infrastructure

that was (re-) used in this talk!

slide-58
SLIDE 58

58 58

Disclaimer

  • All trademarks, product names, company name, publisher name

and their respective logos/images/media cited herein are the property of their respective owners.