Scaling container policy management with kernel features Joe - - PowerPoint PPT Presentation

▶
scaling container policy management with kernel features
SMART_READER_LITE
LIVE PREVIEW

Scaling container policy management with kernel features Joe - - PowerPoint PPT Presentation

Apr 14, 2023 290 likes •838 views

Scaling container policy management with kernel features Joe Stringer Cilium.io Linux Plumbers 2019, Lisbon, Portugal Joe Stringer Scaling container policy with eBPF Sep 11, 2019 1 / 29 Overview 1 Background 2 Deploying fast datapaths fast

slide-1
SLIDE 1

Scaling container policy management with kernel features

Joe Stringer

Cilium.io

Linux Plumbers 2019, Lisbon, Portugal

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 1 / 29

slide-2
SLIDE 2

Overview

1 Background 2 Deploying fast datapaths fast 3 Identity-based security 4 Layer 7 security

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 2 / 29

slide-3
SLIDE 3

Background

Kubernetes Architecture 101

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 3 / 29

slide-4
SLIDE 4

Background

Kubernetes networking plugins

Plumb local connectivity (CNI) Connect remote nodes Services / loadbalancing Network policy

https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/ Joe Stringer Scaling container policy with eBPF Sep 11, 2019 4 / 29

slide-5
SLIDE 5

Background

Cilium

Agent runs on each node Native eBPF dataplane Identity-based security Scalable

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 5 / 29

slide-6
SLIDE 6

Background

What does it mean to scale?

Manage cluster interactions

Minimize unnecessary events Reduce event sizes ...

Optimize work within the node

Apply datapath changes efficiently

https://cilium.io/blog/2019/04/24/cilium-15 Joe Stringer Scaling container policy with eBPF Sep 11, 2019 6 / 29

slide-7
SLIDE 7

Deploying fast datapaths fast

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 7 / 29

slide-8
SLIDE 8

Deploying fast datapaths fast

BPF plumbing

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 8 / 29

slide-9
SLIDE 9

Deploying fast datapaths fast

ELF Templating

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 9 / 29

slide-10
SLIDE 10

Deploying fast datapaths fast

1K nodes: Scaling to 60k pods

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 10 / 29

slide-11
SLIDE 11

Deploying fast datapaths fast

Future directions

Optimize verifier execution: O(n) → O(1) Support code path templatization

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 11 / 29

slide-12
SLIDE 12

Identity-based security

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 12 / 29

slide-13
SLIDE 13

Identity-based security Joe Stringer Scaling container policy with eBPF Sep 11, 2019 13 / 29

slide-14
SLIDE 14

Identity-based security

Policy example

a p i V e r s i o n : "cilium.io/v2" kind: CiliumNetworkPolicy d e s c r i p t i o n : "Restrict deathstar access to empire ships" metadata: name: "deathstar -ingress" spec: e n d p o i n t S e l e c t o r : matchLabels:

  • rg: empire

c l a s s : d e a t h s t a r i n g r e s s :

  • fromEndpoints:
  • matchLabels:
  • rg: empire

toPorts:

  • p o r t s :
  • port: "80"

p r o t o c o l : TCP

https://docs.cilium.io/en/stable/gettingstarted/http/ Joe Stringer Scaling container policy with eBPF Sep 11, 2019 14 / 29

slide-15
SLIDE 15

Identity-based security

Label selectors

12345 {org:empire, class:deathstar} 12468 {org:empire, class:tiefighter} 12465 {org:alliance, class:xwing} matchLabels: 12345 {org:empire, class:deathstar}

  • rg: empire

12468 {org:empire, class:tiefighter} matchLabels:

  • rg: empire

12345 {org:empire, class:deathstar} class: deathstar

https://cilium.io/blog/2019/08/20/cilium-16 Joe Stringer Scaling container policy with eBPF Sep 11, 2019 15 / 29

slide-16
SLIDE 16

Identity-based security

Datapath Configuration: Ingress

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 16 / 29

slide-17
SLIDE 17

Identity-based security

Datapath Configuration: Egress

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 17 / 29

slide-18
SLIDE 18

Identity-based security

Datapath Configuration: Egress

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 17 / 29

slide-19
SLIDE 19

Layer 7 security

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 18 / 29

slide-20
SLIDE 20

Layer 7 security

L7 is the new L4

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 19 / 29

slide-21
SLIDE 21

Layer 7 security

Rejecting traffic in a protocol-aware manner

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 20 / 29

slide-22
SLIDE 22

Layer 7 security

Rejecting traffic in a protocol-aware manner

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 20 / 29

slide-23
SLIDE 23

Layer 7 security

Rejecting traffic in a protocol-aware manner

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 20 / 29

slide-24
SLIDE 24

Layer 7 security

Rejecting traffic in a protocol-aware manner

cilium-agent –http-403-msg="..." Joe Stringer Scaling container policy with eBPF Sep 11, 2019 20 / 29

slide-25
SLIDE 25

Layer 7 security

Rejecting traffic in a protocol-aware manner

cilium-agent –http-403-msg="..." Joe Stringer Scaling container policy with eBPF Sep 11, 2019 20 / 29

slide-26
SLIDE 26

Layer 7 security

Datapath Configuration: L3 flow

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 21 / 29

slide-27
SLIDE 27

Layer 7 security

Datapath Configuration: L7 flow

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 22 / 29

slide-28
SLIDE 28

Layer 7 security

Datapath Configuration: L7 flow

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 22 / 29

slide-29
SLIDE 29

Layer 7 security

L7 Configuration: Past

Per-endpoint configuration A: 192.0.2.3:33333 ; B: 192.0.2.4:80 ; L: 192.0.2.1:12345

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 23 / 29

slide-30
SLIDE 30

Layer 7 security

L7 Configuration: Past

A: 192.0.2.3:33333 ; B: 192.0.2.4:80 ; L: 192.0.2.1:12345

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 23 / 29

slide-31
SLIDE 31

Layer 7 security

L7 Configuration: Past

A: 192.0.2.3:33333 ; B: 192.0.2.4:80 ; L: 192.0.2.1:12345

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 23 / 29

slide-32
SLIDE 32

Layer 7 security

L7 Configuration: Past

A: 192.0.2.3:33333 ; B: 192.0.2.4:80 ; L: 192.0.2.1:12345

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 23 / 29

slide-33
SLIDE 33

Layer 7 security

L7 Configuration: Past

A: 192.0.2.3:33333 ; B: 192.0.2.4:80 ; L: 192.0.2.1:12345

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 23 / 29

slide-34
SLIDE 34

Layer 7 security

L7 Configuration: Present

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 24 / 29

slide-35
SLIDE 35

Layer 7 security

L7 Configuration: Present

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 24 / 29

slide-36
SLIDE 36

Layer 7 security

L7 Configuration: Present

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 24 / 29

slide-37
SLIDE 37

Layer 7 security

L7 Configuration: Present

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 24 / 29

slide-38
SLIDE 38

Layer 7 security

L7 Configuration: Present

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 24 / 29

slide-39
SLIDE 39

Layer 7 security

L7 Configuration: Proposal

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 25 / 29

slide-40
SLIDE 40

Layer 7 security

L7 Configuration: Proposal

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 25 / 29

slide-41
SLIDE 41

Layer 7 security

L7 Configuration: Socket redirect

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 26 / 29

slide-42
SLIDE 42

Layer 7 security

Socket assign: Hiccup

BPF progs are attached at TC ingress skb_orphan() invoked directly before PREROUTING

1https://www.mail-archive.com/netdev@vger.kernel.org/msg303851.html 2https://www.mail-archive.com/netdev@vger.kernel.org/msg304057.html Joe Stringer Scaling container policy with eBPF Sep 11, 2019 27 / 29

slide-43
SLIDE 43

Layer 7 security

Socket assign: Hiccup

BPF progs are attached at TC ingress skb_orphan() invoked directly before PREROUTING TC folks are already carrying hacks for this1

1https://www.mail-archive.com/netdev@vger.kernel.org/msg303851.html 2https://www.mail-archive.com/netdev@vger.kernel.org/msg304057.html Joe Stringer Scaling container policy with eBPF Sep 11, 2019 27 / 29

slide-44
SLIDE 44

Layer 7 security

Socket assign: Hiccup

BPF progs are attached at TC ingress skb_orphan() invoked directly before PREROUTING TC folks are already carrying hacks for this1 Just move to ____dev_forward_skb()2?

1https://www.mail-archive.com/netdev@vger.kernel.org/msg303851.html 2https://www.mail-archive.com/netdev@vger.kernel.org/msg304057.html Joe Stringer Scaling container policy with eBPF Sep 11, 2019 27 / 29

slide-45
SLIDE 45

Layer 7 security

Summary

Minimize processing cost

Number of events Cost for each event

Separate concerns: Policy vs addressing Frontload expensive operations ... while keeping runtime costs low

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 28 / 29

slide-46
SLIDE 46

Thank you

More information

https://cilium.io https://cilium.io/slack https://github.com/cilium/cilium https://twitter.com/ciliumproject

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 29 / 29

slide-47
SLIDE 47

Thank you

More information

https://cilium.io https://cilium.io/slack https://github.com/cilium/cilium https://twitter.com/ciliumproject

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 29 / 29

slide-48
SLIDE 48

Socket redirect

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 30 / 29

slide-49
SLIDE 49

Early demux

i n t tcp_v4_early_demux ( s t r u c t sk_buff ∗ skb ) { // . . . snipped : v a l i d a t e packet type , tcphdr

  • f f s e t s

s t r u c t sock ∗ sk = __inet_lookup_established ( dev_net ( skb− >dev ) , &tcp_hashinfo , iph− >saddr , th− >source , iph− >daddr , ntohs ( th− >dest ) , skb− >s k b _ i i f , i n e t _ s d i f ( skb ) ) ; i f ( sk ) { skb− >sk = sk ; skb− >d e s t r u c t o r = sock_edemux ; i f ( s k _ f u l l s o c k ( sk ) ) { s t r u c t dst_entry ∗ dst = READ_ONCE( sk− >sk_rx_dst ) ; i f ( dst ) dst = dst_check ( dst , 0) ; i f ( dst && inet_sk ( sk )− >r x _ d s t _ i f i n d e x == skb− >s k b _ i i f ) skb_dst_set_noref ( skb , dst ) ; } } return 0 ; }

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 31 / 29

slide-50
SLIDE 50

Socket assign: RFC

/∗ i n t bpf_sk_assign ( s t r u c t sk_buff ∗skb , s t r u c t bpf_sock ∗sk , ∗ u64 f l a g s ) ∗ D e s c r i p t i o n ∗ Assign the ∗ sk ∗ to the ∗ skb ∗. ∗ ∗ This

  • p e r a t i o n

i s

  • nly

v a l i d from TC i n g r e s s path . ∗ ∗ The ∗ f l a g s ∗ argument must be zero . ∗ Return ∗ 0 on success ,

  • r

a n e g a t i v e e r r n o i n case

  • f

f a i l u r e . ∗ ∗ ∗ ∗∗−EINVAL∗∗ Unsupported f l a g s s p e c i f i e d . ∗ ∗ ∗∗−EOPNOTSUPP∗∗: Unsupported

  • peration ,

f o r example ∗ a c a l l from

  • u t s i d e
  • f TC i n g r e s s .

∗ ∗ ∗∗−ENOENT∗∗: The socket cannot be a s s i g n e d . ∗/

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 32 / 29

slide-51
SLIDE 51

Socket assign: API quirks

Nit: Need to use sock_common socket lookups Add bpf_skc_lookup_udp() Optimization: Add lookup flags to bpf_sk*_lookup_*()

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 33 / 29

slide-52
SLIDE 52

ELF Templating: Initial implementation

Treat ELF symbol table as a data table Declare symbols for all static data (constants) (hack) Cast desired static data symbol pointer to data Copy ELF, replacing values in symbol table with desired constants Loader relocation copies the value from the symbol table into the instructions

Joe Stringer Scaling container policy with eBPF Sep 11, 2019 34 / 29

slide-53
SLIDE 53

ELF Templating: Upstream libbpf

Create maps for .bss, .data, .rodata Relocate loads from static symbols to be map loads Kernel-side, fix up to be direct load

https://lwn.net/ml/bpf/d406e2487f0dc1bf0326ed12e5e0cd0d17eae89c.1554314902.git.daniel@iogearbox.net/ Joe Stringer Scaling container policy with eBPF Sep 11, 2019 35 / 29