The age of orchestration From Docker basics to cluster management - - PowerPoint PPT Presentation

▶
the age of orchestration
SMART_READER_LITE
LIVE PREVIEW

The age of orchestration From Docker basics to cluster management - - PowerPoint PPT Presentation

Jan 02, 2024 110 likes •509 views

The age of orchestration From Docker basics to cluster management NICOLA PAOLUCCI DEVELOPER INSTIGATOR ATLASSIAN @DURDN Three minute Docker intro? Time me and ring a bell if I am over it. Just kidding Ill be over by a bit

slide-1
SLIDE 1

The age of orchestration

NICOLA PAOLUCCI • DEVELOPER INSTIGATOR • ATLASSIAN • @DURDN

From Docker basics to cluster management

slide-2
SLIDE 2 3

Time me and ring a bell if I am over it. Just kidding I’ll be over by a bit but it’s ok. We’re friends.

Three minute Docker intro?

slide-3
SLIDE 3 4

BINS/LIBS GUEST OS HYPERVISOR HOST OS SERVER

ROBUST MONOLITHIC SLOW TO BOOT HEAVY OVERHEAD

APP A BINS/LIBS GUEST OS APP B

DEFINITIONS

Traditional Virtual Machines

slide-4
SLIDE 4 5

DOCKER ENGINE HOST OS SERVER

LEAN LIGHTWEIGHT ISOLATED PORTABLE EFFICIENT

BINS/LIBS APP A BINS/LIBS APP B

DEFINITIONS

Docker Containers

slide-5
SLIDE 5 6

DEFINITIONS

Seen from another angle, the core of Docker is four distinct things

Clearly defjned interfaces Central registry of ready images Caching mechanism to re-use steps A standard format to package applications

slide-6
SLIDE 6 7

DO WE USE IT?

We have embraced Docker on two fronts

For our internal PaaS In our products

N O T P A R T O F T H I S S E S S I O N

slide-7
SLIDE 7

@durdn

Our internal PaaS, called Micros

slide-8
SLIDE 8 9

Overall Micros Numbers

Microservices

600+

9

Java Node.js Python

Docker containers

40%

Rest is pre-made stacks

slide-9
SLIDE 9

@durdn

Media Services

slide-10
SLIDE 10

Our Conversion Stack

slide-11
SLIDE 11

6M

12

Media Services Numbers

Microservices

10

12

containers spun per month Processed every month

TBs

slide-12
SLIDE 12 13

CONCLUSIONS

Docker has been a great fit for our Media Services team

Worked around tools not easy to parallelise Manage resource control Isolating data from difgerent customers Easily scale horizontally

slide-13
SLIDE 13

@durdn

Orchestration is the next arena

slide-14
SLIDE 14

What’s Orchestration?

15

Services

YOUR APPLICATION

Orchestration

FRAMEWORKS

Data Center

PHYSICAL INFRA

slide-15
SLIDE 15

@durdn

slide-16
SLIDE 16

@durdn

slide-17
SLIDE 17

@durdn

slide-18
SLIDE 18 19
slide-19
SLIDE 19 20

Docker’s Own Orchestration Tools

Docker machine Docker compose Docker swarm Docker network

slide-20
SLIDE 20

Where is the DEMO Lebowski?

slide-21
SLIDE 21 22

TOOL NR.1

Docker machine

Simple command line tool to provision local and remote hosts with Docker installed. Fantastic to get up and running fast. It has drivers for many Internet service providers and PaaS.

Docker machine

$ docker-machine create -d virtualbox dev INFO[0000] Downloading boot2docker.iso from... INFO[0001] Creating SSH key... INFO[0001] Creating VirtualBox VM... INFO[0006] Starting VirtualBox VM... INFO[0007] Waiting for VM to start... INFO[0041] "dev" has been created and is now active

slide-22
SLIDE 22
  • Provision a machine with

Docker installed and ready

  • Pull a minimal image
  • Run a few docker commands
  • Tear down the machine

Docker machine DEMO

$ docker-machine create -d virtualbox dev INFO[0000] Downloading boot2docker.iso from... INFO[0001] Creating SSH key... INFO[0001] Creating VirtualBox VM... INFO[0006] Starting VirtualBox VM... INFO[0007] Waiting for VM to start... INFO[0041] "dev" has been created and is now active

slide-23
SLIDE 23
  • “docker-machine create” to provision the host, locally or remotely
  • “docker-machine ls” to list the machines
  • “docker-machine stop/rm” to stop and remove

Recap of what you saw

$ docker-machine create -d virtualbox dev INFO[0000] Downloading boot2docker.iso from... INFO[0001] Creating SSH key... INFO[0001] Creating VirtualBox VM... INFO[0006] Starting VirtualBox VM... INFO[0007] Waiting for VM to start... INFO[0041] "dev" has been created and is now active

slide-24
SLIDE 24 26

TOOL NR.2

Docker compose

Docker compose

Describe the relation of your components in a simple YAML file called docker-compose.yml and docker-compose takes care of starting them and linking them in order.

1 bitbucket: 2 image: atlassian/bitbucket-server 3 ports: 4 - "7990:7990" 5 - "7999:7999" 6 links: 7 - db 8 volumes_from: 9 - license 10 user: root 11 privileged: true 12 db: 13 image: postgres 14 ports: 15 - "5432:5432" 16 environment: 17 - "POSTGRES_PASSWORD=somepassword" 18 license: 19 build: .

slide-25
SLIDE 25
  • Provision a machine on a PaaS
  • Pull PostgreSQL and a Java

app from the Registry

  • Use Compose to start the app
  • Tear down the machine

Docker compose DEMO

$ docker-compose up -d

slide-26
SLIDE 26
  • “docker-machine create” to provision the host
  • Edit “docker-compose.yml” to describe our app
  • “docker-compose up -d” to start our application
  • “docker-machine rm compose-demo“ to

remove it

Recap of what you saw

$ docker-compose up -d

slide-27
SLIDE 27

1 bitbucket: 2 image: atlassian/bitbucket-server 3 ports: 4 - "7990:7990" 5 - "7999:7999" 6 volumes_from: 7 - license 8 user: root 9 privileged: true 10 environment: 11 - "constraint:instance==java" 12 db: 13 image: postgres 14 ports: 15 - "5432:5432" 16 environment: 17 - "POSTGRES_PASSWORD=somepassword" 18 - "constraint:instance==db" 19 license: 20 build: .

30

TOOL NR.3

Docker swarm

Deploy images and run containers on a full clusters as if you’re handling a single machine

Docker swarm

slide-28
SLIDE 28

@durdn

Docker swarm

swarm master swarm node swarm node swarm node container container container container container container discovery service

High level architecture

scheduler

slide-29
SLIDE 29
  • Strategies
  • Spread
  • Binpack
  • Random
  • Filters
  • Constraint
  • Affinity
  • Port
  • Dependency
  • Health

Swarm comes with strategies and filters

$ docker run -e \ constraint:instance==database --name db

slide-30
SLIDE 30 33

HELPER TOOL

Discovery Service

For our Swarm to know which nodes are added to the infrastructure and store information about them we need to use a key- value discovery service, like Consul.

Consul from HashiCorp

slide-31
SLIDE 31 34

TOOL NR.4

Docker network

New Docker command to manage advanced and transparent networking, like creating VXLAN-based overlay networks that span across data centers.

Docker network

$ docker network create \

  • -driver overlay mynet
slide-32
SLIDE 32
  • Provision a Docker swarm
  • Made up of three hosts
  • Master node
  • Node with 2gb of RAM
  • simple Node
  • Use labels to deploy to nodes
  • Run Java app and PostgreSQL
  • n different nodes

The plan for the Swarm DEMO

$ docker pull swarm $ docker run --rm swarm create 6856663cdefdec325839a4b7e1de38e8

slide-33
SLIDE 33

@durdn

After Part 1

demo-master node1 node2 label: java Consul: discovery service

Current Architecture

scheduler RAM: 2GB label: database RAM: 512MB

slide-34
SLIDE 34 38

Multi-host Docker compose

1 bitbucket: 2 image: atlassian/bitbucket-server 3 ports: 4 - "7990:7990" 5 - "7999:7999" 6 volumes_from: 7 - license 8 user: root 9 privileged: true 10 environment: 11 - "constraint:instance==java" 12 db: 13 image: postgres 14 ports: 15 - "5432:5432" 16 environment: 17 - "POSTGRES_PASSWORD=somepassword" 18 - "constraint:instance==db" 19 license: 20 build: .

docker-compose.yml

slide-35
SLIDE 35

@durdn

What we did

demo-master node1 node2 label: java

Final Architecture

scheduler RAM: 2GB label: database RAM: 512MB

PostgreSQL Java App

Consul: discovery service

slide-36
SLIDE 36
  • We created a 3-node cluster with

“docker-machine”

  • We tagged the nodes with labels
  • We started our components using

label constraints and not IP addresses

Recap of Swarm DEMO

slide-37
SLIDE 37

I hope you are hyped as I am for all this coolness, come talk to me afterwards!

slide-38
SLIDE 38

Twitter: @durdn