CernVM-FS Graphdriver Plugin for Docker CernVM Workshop 2018 CERN - - PowerPoint PPT Presentation

▶
cernvm fs graphdriver plugin for docker
SMART_READER_LITE
LIVE PREVIEW

CernVM-FS Graphdriver Plugin for Docker CernVM Workshop 2018 CERN - - PowerPoint PPT Presentation

Jan 19, 2024 98 likes •398 views

CernVM-FS Graphdriver Plugin for Docker CernVM Workshop 2018 CERN Nikola Hardi hardi.nikola@gmail.com January 30th, 2018 Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 1 / 28 Containers get

slide-1
SLIDE 1

CernVM-FS Graphdriver Plugin for Docker

CernVM Workshop 2018 CERN Nikola Hardi hardi.nikola@gmail.com January 30th, 2018

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 1 / 28

slide-2
SLIDE 2

Containers get heavy!

Joint Blog Post Mesosphere & CERN (03/2016)

Network traffic gets congested as gigabytes worth of Docker downloads are moving across the pipe [. . . ]. Companies [. . . ] such as Twitter have already experienced this phenomenon.

Red Hat, “Containers for Grownups” (02/2016)

10 things to avoid in docker containers: . . . 3) Don’t create large images.

Medallia (10/2015, CERN KT Screening)

The problem today with Docker is that distribu- tion of software is a mess, it is a “bottleneck” in

  • ur system.

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 2 / 28

slide-3
SLIDE 3

CernVM and CernVM-FS

The experiment software is delivered to the appliance just in time by means of a network file system (CVMFS) specifically designed for effi- cient software distribution and installation.

— Predrag Buncic et al. “CernVM - a virtual appliance for LHC applications”, ACAT08

The virtual machine image is less than 20 megabyte in size. The actual

  • perating system is delivered on demand by the CernVM File System.

— Gerardo Ganis et al. “Status and Roadmap of CernVM”, J. Phys., 2015

To serve the needs of the HEP community, several iterations were made to create a scalable, user-level filesystem that delivers software worldwide

  • n a daily basis.

— Jakob Blomer et al. “Status and Roadmap of CernVM”, Comp. in Sci. and Eng., 2017 Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 3 / 28

slide-4
SLIDE 4

The solution proposed by CernVM team

Docker “App Store” Docker Software pull & push “apps” KT-Funded Project CernVM File System Improved Docker Software file-based transfer

Our analysis shows that pulling packages accounts for 76% of container start time, but only 6.4% of that data is read.

— T. Harter et al. “Slacker: Fast Distribution with Lazy Containers”, FAST16 Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 4 / 28

slide-5
SLIDE 5

Container Engines

Rocket Docker Singularity

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 5 / 28

slide-6
SLIDE 6

About the Docker engine

Main components: Docker client Docker daemon Registry: hub.docker.com Docker versions: 2017 - Jan: Docker 1.13.0 2017 - Feb: Docker 1.13.1 2017 - Mar: Docker 17.03 2018 - Mar: Docker 18.01

Host machine Docker client Docker daemon Internet Docker registry HTTP HTTP

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 6 / 28

slide-7
SLIDE 7

About the Docker images

Docker image is a stack of read-only layers. The list of read-only layers is specified in the image manifest. Layer is a filesystem diff between two snapshots. Layers are content addressable and reusable. Container is a Docker image in the state of execution. Each container has a dedicated read-write layer.

nginx OwnCloud ubuntu

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 7 / 28

slide-8
SLIDE 8

The CernVM-FS graphdriver plugin for Docker

Docker client Docker daemon Graphdriver plugin This is a container! Docker registry CernVM-FS server Minio S3 Host machine Remote server HTTP CernVM-FS S3

$ docker plugin install cvmfs/graphdriver

Restart Dockerd with −−experimental −s cvmfs/graphdriver

$ docker run cvmfs/thin ubuntu echo ’’Hello CVM workshop!’’

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 8 / 28

slide-9
SLIDE 9

DEMO

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 9 / 28

slide-10
SLIDE 10

Results

Ubuntu OwnCloud R 10 100 5 50 25 200 Just 3 MB!

Transferred Data [MB]

Ubuntu OwnCloud R 10 20 30 40 Constant startup time!

Startup Time [s]

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 10 / 28

slide-11
SLIDE 11

Results 2

The cluster startup time reduced from 5 min to less than 5 s.

Tomorrow @ 5:30pm, Ricardo Brito Da Rocha:

Magnum and HNSciCloud

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 11 / 28

slide-12
SLIDE 12

Results 3

Tomorrow @ 5:30pm, Ricardo Brito Da Rocha:

Magnum and HNSciCloud

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 12 / 28

slide-13
SLIDE 13

The CernVM-FS graphdriver — DETAILS —

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 13 / 28

slide-14
SLIDE 14

The CernVM-FS graphdriver plugin for Docker

Docker client Docker daemon Graphdriver plugin This is a container! Docker registry CernVM-FS server Minio S3 Host machine Remote server HTTP CernVM-FS S3

1 Plugins run in containers. 2 Plugins communicate with the Docker daemon over HTTP. 3 Layer upload to CernVM-FS through a portal (Minio S3). Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 14 / 28

slide-15
SLIDE 15

The thin image format

Regular Docker Image Scratch layer Local read-only layer Thin image descriptor CernVM-FS provided read-only layer Thin Image CernVM-FS graphdriver

Additional benefits comming from CernVM-FS

1 Files that are never accessed will never be transferred. 2 CernVM-FS controlled local cache size. 3 File based deduciplation / reuse for free. Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 15 / 28

slide-16
SLIDE 16

Example: the thin image descriptor

$ docker2cvmfs thin library/ubuntu:latest test.cern.ch { "version": "1.0", "origin": "library/ubuntu:latest@https://registry-1.docker.io/v2", "layers": [ { "digest": "1be7f2b886e89a58e59c4e685fcc5905a26ddef3201f290b96f1eff7d778e122", "url": "cvmfs://test.cern.ch/1be7f2b886e89a58e59c4e685fcc5905a26ddef3201f290b96f1eff7d778e122" }, { "digest": "6fbc4a21b806838b63b774b338c6ad19d696a9e655f50b4e358cc4006c3baa79", "url": "cvmfs://test.cern.ch/6fbc4a21b806838b63b774b338c6ad19d696a9e655f50b4e358cc4006c3baa79" }, { "digest": "c71a6f8e13782fed125f2247931c3eb20cc0e6428a5d79edb546f1f1405f0e49", "url": "cvmfs://test.cern.ch/c71a6f8e13782fed125f2247931c3eb20cc0e6428a5d79edb546f1f1405f0e49" }, { "digest": "4be3072e5a37392e32f632bb234c0b461ff5675ab7e362afad6359fbd36884af", "url": "cvmfs://test.cern.ch/4be3072e5a37392e32f632bb234c0b461ff5675ab7e362afad6359fbd36884af" }, { "digest": "06c6d2f5970057aef3aef6da60f0fde280db1c077f0cd88ca33ec1a70a9c7b58", "url": "cvmfs://test.cern.ch/06c6d2f5970057aef3aef6da60f0fde280db1c077f0cd88ca33ec1a70a9c7b58" } ] } Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 16 / 28

slide-17
SLIDE 17

The docker2cvmfs

1 Inspect an image manifest. 2 Pull all layers of an image from a Docker registry. 3 Produce a thin image descriptor. 1 Manipulating Docker images is not an easy problem. 2 Special files, permissions, ownership, hardlinks, whiteout files... 3 Everyone has their own solution. Let’s work together!

Checkout following presentations

1 Today @ 3pm, Tom Downes:

Automated Conversion of Docker images to CVMFS for LIGO and the Open Science Grid

2 Tomorrow @ 10am, Michael Bauer:

Building Reproducible Science with Singularity Containers

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 17 / 28

slide-18
SLIDE 18

The pull scenario

Pull cvmfs/thin ubuntu from the Docker registry This image contains just the layer with the thin image descriptor. Request the graphdriver to create the scratch layer. Mount the required CernVM-FS repositories. Creates the union mount of the scratch layer and the read-only layers mounted directly from CernVM-FS. The list of repositories and read-only layers is available in the thin image descriptor.

Regular image Thin image

read-write layer thin image layer

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 18 / 28

slide-19
SLIDE 19

The push scenario

Regular image:

1 The changeset is stored in the r/w branch (r/w layer). 2 Publish this changes as new r/o layer on the Docker registry. 3 Create new Docker image with updated list or r/o layers.

Thin image:

1 The changeset is stored in the r/w branch (r/w layer). 2 Publish this changes as new r/o layer on CVMFS repository. 3 Create new Docker image with updated thin image descriptor.

CVMFS stratum 0 Docker registry

+

read-write layer read-only layer thin layer

Note: this functionality is currently disabled.

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 19 / 28

slide-20
SLIDE 20

Project status and roadmap

Not production ready but can do useful work. Ready for evaluation. We already have few early adpoters. Improve stability and make workflows even smoother. CernVM-FS portals feature in CernVM-FS 2.6. CernVM team continues the development.

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 20 / 28

slide-21
SLIDE 21

Conclusion

The plugin works for both running and modifying container images. All requirements are available in current Docker releases. Plugin can be installed from Docker registry. Creating thin images in simple and automated way. Preserves all Docker features, operation is transparent to users. Supports both thin and regular images.

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 21 / 28

slide-22
SLIDE 22

Thank you!

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 22 / 28

slide-23
SLIDE 23

CernVM-FS Graphdriver Plugin for Docker

CernVM Workshop 2018 CERN Nikola Hardi hardi.nikola@gmail.com January 30th, 2018

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 23 / 28

slide-24
SLIDE 24

Linux containers - history

Support in the Linux kernel

◮ Namespaces mean isolation of resources. ◮ Cgroups (control groups) capabilities and resources accounting.

Available namespaces: 6

◮ mnt - mount ◮ net - network ◮ ipc - interprocess communication ◮ user - users and groups ◮ pid - process id ◮ uts - hostname

Mount namespace implemented in 2002! (Linux 2.4.19) Similar approaches in other OSs:

◮ 1980 - Unix/BSD chroot ◮ 1992 - Plan9 namespaces ◮ 2000 - BSD jail ◮ 2004 - Solaris zones Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 24 / 28

slide-25
SLIDE 25

Containers and Virtual Machines

Container pros and cons:

Smaller virtualization overhead

for system calls, I/O, memory translation

Better at overcommitting with

idle services

Boots faster (with caveats) Orchestration tools available Weaker isolation No “privileged operations”,

e.g. mount

Linux only More moving parts

Server Hardware Server Hardware Host OS Kernel Host OS Kernel Hypervisor

OS Binaries / Libs App

Guest OS Kernel

Guest OS Kernel OS Binaries / Libs App OS Binaries / Libs App OS Binaries / Libs App Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 25 / 28

slide-26
SLIDE 26

Docker plugin infrastracture

Daemon talks to the plugin process over unix socket or tcp port. Protocol is JSON RPC over HTTP. Two versions of plugins:

◮ V1: separate process running on host machine ◮ V2: plugin process running inside of a container (plugin container)

Supported plugins:

◮ network ◮ volumes ◮ graphdrivers! (since Docker 1.13, Jan. 2017)

config.json + rootfs V2 plugins are managed (plugin install and push, shared via registry) $ docker plugin install atlantic777/plugin (WARNING: still just for preview!)

Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 26 / 28

slide-27
SLIDE 27

The graphdriver plugin API

Very limited interface:

◮ Create() / CreateRW() ◮ ApplyDiff() - import (extract) layer archive ◮ Diff() - export (create) layer archive ◮ Get() - create union mount ◮ Put() - release union mount

We introduce the concept of thin images. Like regular image but store metadata instead of content. Can be published on standard Docker registry. Thin images contain single json file, the image descriptor. Thin image descriptor contains list of r/o layers to be mounted from CernVM-FS repository. Container rootfs now consists of:

◮ Dedicated r/w layer. ◮ And r/o layers stored in CernVM-FS. Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 27 / 28

slide-28
SLIDE 28

CernVM-FS portals

Changes in graphdriver plugin Diff() method: Create archive containing changed files. Compress it and calculate hash. Upload this new layer archive to stratum 0 server over S3 API with hash as key. Create updated image descriptor. Return archive with image descriptor. Stratum 0 publisher agent: S3 server sends notification about new object. Publisher agent receives notification. Publisher agent will:

◮ start CVMFS transaction, ◮ extract this new layer and ◮ publish transaction. Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 28 / 28