pot: FreeBSD containers on FreeBSD Luca Pizzamiglio - - PowerPoint PPT Presentation

▶
pot freebsd containers on freebsd
SMART_READER_LITE
LIVE PREVIEW

pot: FreeBSD containers on FreeBSD Luca Pizzamiglio - - PowerPoint PPT Presentation

Nov 21, 2023 97 likes •451 views

pot: FreeBSD containers on FreeBSD Luca Pizzamiglio pizzamig@FreeBSD.org FOSDEM 2018 whoami(1) Luca Pizzamiglio aka pizzamig@ FreeBSD enthusiast Port committer since August 2017 Building packages at trivago 2018-02-03 2 pot:

slide-1
SLIDE 1

pot: FreeBSD containers on FreeBSD

Luca Pizzamiglio pizzamig@FreeBSD.org FOSDEM 2018

slide-2
SLIDE 2

2018-02-03 pot: FreeBSD containers for FreeBSD 2

whoami(1)

  • Luca Pizzamiglio aka pizzamig@
  • FreeBSD enthusiast
  • Port committer since August 2017
  • Building packages at trivago
slide-3
SLIDE 3

2018-02-03 pot: FreeBSD containers for FreeBSD 3

Motivations 1/2

I needed a tool to easily create/run FreeBSD “instances” to

  • build/develop/test ports
  • develop/test Saltstack tests
  • run web services

Several really good solutions already available, even if not perfect for my use cases:

  • ezjail, iocage, …
slide-4
SLIDE 4

2018-02-03 pot: FreeBSD containers for FreeBSD 4

Motivations 2/2

It should run on a laptop

  • limited hardware resources
  • fmexible network confjguration (DHCP)

I wanted to

  • imitate docker, FreeBSD containers for FreeBSD
  • force automation → user oriented CLI
  • experiment difgerent solutions/layouts/concepts
  • use and learn more about FreeBSD features
slide-5
SLIDE 5

2018-02-03 pot: FreeBSD containers for FreeBSD 5

So, what is pot?

pot is a tool to automate the management of those container

Currently, pot is a bunch of shell scripts Basic features are covered by standard tools Advanced features will be implemented with a proper programming language

Why ‘pot’?

slide-6
SLIDE 6

2018-02-03 pot: FreeBSD containers for FreeBSD 6

Pasta analogy [1/2]

slide-7
SLIDE 7

2018-02-03 pot: FreeBSD containers for FreeBSD 7

File system components

Split the fjle system in several logic components:

  • FreeBSD base
  • It determines the FreeBSD version
  • Packages
  • Installed packages
  • Customization
  • Confjguration fjles, home directories, /var
slide-8
SLIDE 8

2018-02-03 pot: FreeBSD containers for FreeBSD 8

Pot: level 1

pot 11.1 Base 11.1 Package 11.1 Custom 11.1 Level 0 pot A Package A Custom A Base 11.1 Level 1 pot B Custom B Package B Base 11.1 Level 1

slide-9
SLIDE 9

2018-02-03 pot: FreeBSD containers for FreeBSD 9

CL workfmow

# pot init # pot create-base -r 11.1 # pot create -p A -b 11.1 # pot create -p B -b 11.1 # pot start A # pot stop A

Create base 11.1 datasets Create pot base-11_1 Mounts ZFS datasets via nullfs(5) Starts the jail Stop the jail Unmounts ZFS datasets Download of FreeBSD 11.1

slide-10
SLIDE 10

2018-02-03 pot: FreeBSD containers for FreeBSD 10

File system components

File system components as building blocks

  • Mandatory
  • Base
  • Package
  • Customization
  • Whatever you need
  • Code repository
  • Databases
  • Caches
  • ...
slide-11
SLIDE 11

2018-02-03 pot: FreeBSD containers for FreeBSD 11

Example: saltmaster

pot 11.1 Base 11.1 Package 11.1 Custom 11.1 pot saltmaster Package salt Custom salt Base 11.1 Repository

slide-12
SLIDE 12

2018-02-03 pot: FreeBSD containers for FreeBSD 12

CL workfmow

# pot init # pot create-base -r 11.1 # pot create-fscomp -f repository # pot create -p saltmaster -b 11.1 # pot add-fscomp -p saltmaster \

  • f repository \
  • m /mnt
slide-13
SLIDE 13

2018-02-03 pot: FreeBSD containers for FreeBSD 13

Pasta analogy [2/2]

slide-14
SLIDE 14

2018-02-03 pot: FreeBSD containers for FreeBSD 14

pot: level 2

pot 11.1 Base 11.1 Package 11.1 Custom 11.1 pot salt-base Package s-base Custom s-base Base 11.1 pot salt-work Package s-base Custom s-work Base 11.1 Repo s-work pot salt-home Package s-base Custom s-home Base 11.1 Repo s-home

slide-15
SLIDE 15

2018-02-03 pot: FreeBSD containers for FreeBSD 15

CL workfmow

# pot init # pot create-base -r 11.1 # pot create-fscomp -f repo-work # pot create-fscomp -f repo-home # pot create -p salt-base -b 11.1 # pot create -p salt-work -P salt-base -l 2 # pot create -p salt-home -P salt-base -l 2 # pot add-fscomp -p salt-work -f repo-work -m /mnt # pot add-fscomp -p salt-home -f repo-home -m /mnt

slide-16
SLIDE 16

2018-02-03 pot: FreeBSD containers for FreeBSD 16

Network

Two network confjgurations available:

  • Inherit
  • Inherit the network stack of the host
  • static IP in the internal virtual network
  • Exploits VNET(9) (kernel manually rebuilt)
  • NAT supported by pf(4)
  • the physical network interface as default gateway
  • all network interfaces are on the same bridge
slide-17
SLIDE 17

2018-02-03 pot: FreeBSD containers for FreeBSD 17 Host

Internal virtual network

bridge0

epair0a epair0b 10.192.0.2 epair1a epair1b 10.192.0.3 epair2a epair2b 10.192.10.1 epair3a epair3b 10.192.10.2 10.192.0.1

pf NAT

epair4a epair4b 10.192.100.2 Network: 10.192.0.0/10 lagg0

slide-18
SLIDE 18

2018-02-03 pot: FreeBSD containers for FreeBSD 18

Network: missing features

  • Add support to static IP without NAT
  • As currently provided by jails
  • SHCP: Static DHCP
  • Currently, IP addresses have to be manually specifjed
  • SHCP would be a tool to provide valid static IP addresses
  • Expose network services
  • A special dns pot running dnsmasq and consul
  • Network services registration to consul
  • haproxy running in the host can redirect request to the

right pot using the information provided by the dns pot

slide-19
SLIDE 19

2018-02-03 pot: FreeBSD containers for FreeBSD 19

pot is ZFS!

A pot is a bunch of ZFS datasets!

  • zfs snapshot => pot snapshot
  • zfs rollback => pot rollback
  • zfs clone => pot clone
  • zfs rename => pot rename

Work in progress

  • zfs promote => pot promote
slide-20
SLIDE 20

2018-02-03 pot: FreeBSD containers for FreeBSD 20

Pot fmavor

Two kinds of fmavors

  • A typical shell script, executed inside the

container

  • Ideal for provisioning
  • A default fmavor is also available
  • A set of pot commands, to enrich the pot

confjguration

  • Ideal to attach fjle system components
  • Possibility to enforce priority between pots
slide-21
SLIDE 21

2018-02-03 pot: FreeBSD containers for FreeBSD 21

Pot fmavor

Imitating poudriere(8) # pot create -p builder -b 11.1 -f buildport ## buildport

add-fscomp -f svnport -m /usr/ports add-fscomp -f distfiles -m /usr/ports/distfiles add-fscomp -f ccache -m /mnt

## buildport.sh

#!/bin/sh pkg install -y ccache pkg clean -ayq echo "setenv CCACHE_DIR /mnt" >> /root/.cshrc

slide-22
SLIDE 22

2018-02-03 pot: FreeBSD containers for FreeBSD 22

pot add-dep : Runtime dependency

Add dynamic dependencies between container Example: salt-test needs saltmaster

  • salt-test is the client
  • saltmaster is the server
  • pot add-dep -p salt-test -P saltmaster
  • pot start salt-test
  • saltmaster will start automatically
  • saltmaster will start fjrst
  • Then, salt-test will start
slide-23
SLIDE 23

2018-02-03 pot: FreeBSD containers for FreeBSD 23

Resource limitation: cpuset(1)

Limiting CPU usage

  • Statically assign a pot to one or more CPUs

# pot set-rss -p pot -C 0,2 Implemented via cpuset(1)

  • Applied immediately after the start of the jail

Possible improvement

  • Set the number of CPUs wanted
  • During the start, a static allocation is performed that

balance the load between CPUs

slide-24
SLIDE 24

2018-02-03 pot: FreeBSD containers for FreeBSD 24

Resource limitation: rctl(8)

  • rctl(8) is a relatively new resource limitation

framework implemented in FreeBSD 9, but not enabled by default

  • To be enabled at boot time via

kern.racct.enable=1 in /boot/loader.conf

  • Used to show used resources and set specifjc

limits

slide-25
SLIDE 25

2018-02-03 pot: FreeBSD containers for FreeBSD 25

Resource limitation: rctl(8) memoryuse

To limit the physical memory used by a pot

  • How much?
  • If the limit is reached, what happen?
  • Out of memory?
  • Soft limit?

Example: pot saltmaster

  • Physical memory used: 430MB
  • pot show is the command showing the resource used

by a pot

slide-26
SLIDE 26

2018-02-03 pot: FreeBSD containers for FreeBSD 26

Resource limitation: rctl(8) memoryuse

Physical memory used: 430MB

  • Limit 400MB → still working, memory 400MB
  • Limit 200MB → still working, memory ~200MB,

sometimes above

  • Limit 50MB → still working, memory ~52MB, often

above

  • Limit 10MB → still working, memory ~11MB, often a lot

above the limit

The memory limit reduce the RSS of a process to fjt the constraint

The processes “working set” are drastically reduced Possible big performance penalty

slide-27
SLIDE 27

2018-02-03 pot: FreeBSD containers for FreeBSD 27

Resource limitation: rctl(8) pcpu

To limit the cpu percentage used by a pot

  • I wasn’t able to fjnd a proper setup
  • pcpu counter in kernel space has an odd behavior
  • 20k % of CPU usage?
  • To enforce the CPU% limits, the processes are

simply blocked

  • Delay of seconds observed, causing timeouts to expire

Not adopted in pot and probably it won’t in the future

slide-28
SLIDE 28

2018-02-03 pot: FreeBSD containers for FreeBSD 28

Moonshot : the big picture

slide-29
SLIDE 29

2018-02-03 pot: FreeBSD containers for FreeBSD 29

pot migration : a look to the future

pot salt-work Package s-base Custom s-work Base 11.1 Repo s-work pot salt-home Package s-base Custom s-home Base 11.1 Repo s-home pot salt-base Package s-base Custom s-base Base 11.1 pot base-11_1 Package 11.1 Custom 11.1 Base 11.1 pot php-base Package php Custom php Base 11.1 pot web1 Package php Custom web1 Base 11.1 Repo web1 pot web2 Package php Custom web2 Base 11.1 Repo web2 pot web3 Package php Custom web3 Base 11.1 Repo web3

slide-30
SLIDE 30

2018-02-03 pot: FreeBSD containers for FreeBSD 30

pot migration : a look to the future

Snapshot exporter

pot salt-work

Package s-base Custom s-work

Base 11.1

Repo s-work

pot salt-home

Package s-base Custom s-home

Base 11.1

Repo s-home

pot salt-base

Package s-base Custom s-base

Base 11.1 pot base-11_1

Package 11.1 Custom 11.1

Base 11.1 pot php-base

Package php Custom php

Base 11.1 pot web1

Package php Custom web1

Base 11.1

Repo web1

pot web2

Package php Custom web2

Base 11.1

Repo web2

pot web3

Package php Custom web3

Base 11.1

Repo web3

Web123-1

pot web1

Package php Custom web1

Base 11.1

Repo web1

pot web3

Package php Custom web3

Base 11.1

Repo web3

pot web2

Package php Custom web2

Base 11.1

Repo web2

Base 11.1

Package php Custom web1 Custom web2 Custom web3

Repo web1 Repo web2 Repo web3

Web123-2

pot web1

Package php Custom web1

Base 11.1

Repo web1

pot web3

Package php Custom web3

Base 11.1

Repo web3

pot web2

Package php Custom web2

Base 11.1

Repo web2

Base 11.1

Package php Custom web1 Custom web2 Custom web3

Repo web1 Repo web2 Repo web3

slide-31
SLIDE 31

2018-02-03 pot: FreeBSD containers for FreeBSD 31

Orchestration?

server1 web1 web2 web3 server2 web1 web2 web3 server3 web1 web2 web4 server4 web1 web2 web4 Load Balancer

slide-32
SLIDE 32

2018-02-03 pot: FreeBSD containers for FreeBSD 32

Conclusion → TILs

pot is a possible implementation of a container model entirely based on FreeBSD The project is on github

https://github.com/pizzamig/pot Fork it and submit pull requests Submit issues (it’s still full of bugs, help!)

TIL: containers cannot be better than the host Operating System

slide-33
SLIDE 33

2018-02-03 pot: FreeBSD containers for FreeBSD 33

Thanks!

Thanks a lot! Questions?

slide-34
SLIDE 34

2018-02-03 pot: FreeBSD containers for FreeBSD 34

Contributions

[1] pot logo

Daniela Spoto https://danielaspoto.wixsite.com/illustrations

[2] Pasta

Junya Ogura https://www.fmickr.com/photos/sooey/5089711764

[3] spaghetti carbonara

Martin Krolikowski https://www.fmickr.com/photos/martinkrolikowski/6302915547

[4] Pici with ragù

Luca Nebuloni https://www.fmickr.com/photos/nebulux/8524965788

[5] The Moonshot

Diego T

  • rres Silvestre

https://www.fmickr.com/photos/3336/6039485059