A Kubernetes Operator for etcd jetstack.io Open Source Tools - - PowerPoint PPT Presentation

▶
a kubernetes operator for etcd
SMART_READER_LITE
LIVE PREVIEW

A Kubernetes Operator for etcd jetstack.io Open Source Tools - - PowerPoint PPT Presentation

Dec 12, 2022 346 likes •1.09k views

Presented by James Laverack, Solutions Engineer A Kubernetes Operator for etcd jetstack.io Open Source Tools Subscription Consulting Training Enhancing the Kubernetes Reference architecture, online Consulting and engineering to Learn and

slide-1
SLIDE 1

jetstack.io

A Kubernetes Operator for etcd

Presented by James Laverack, Solutions Engineer

slide-2
SLIDE 2

Open Source Tools

Enhancing the Kubernetes experience with open source

  • tooling. Includes provisioning,

stateful services and security related projects

Training

Learn and engage directly alongside our team, with courses for all stages of your Kubernetes journey

jetstack.io

Subscription

Reference architecture, online training and SLA support 24x7 for your production Kubernetes deployment

Consulting

Consulting and engineering to make the most of Kubernetes and move you to production quickly

slide-3
SLIDE 3

jetstack.io

“We need to run etcd in Kubernetes.”

slide-4
SLIDE 4

jetstack.io

slide-5
SLIDE 5

jetstack.io

etcd-0.example.com etcd-1.example.com etcd-2.example.com

LEADER

client-app.example.com

CLIENT APPLICATION

etcd.example.com

slide-6
SLIDE 6

“What about a stateful set?”

jetstack.io

slide-7
SLIDE 7

jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-8
SLIDE 8

jetstack.io

etcd-0.example.com etcd-1.example.com etcd-2.example.com

LEADER

etcd-0.example.com etcd-1.example.com etcd-2.example.com

slide-9
SLIDE 9

jetstack.io

etcd-0.example.com etcd-1.example.com etcd-2.example.com

LEADER

etcd-0.example.com etcd-1.example.com etcd-2.example.com etcd-3.example.com

slide-10
SLIDE 10

jetstack.io

etcd-0.example.com etcd-1.example.com etcd-2.example.com

LEADER

etcd-3.example.com

slide-11
SLIDE 11

jetstack.io

etcd-0.example.com etcd-1.example.com etcd-2.example.com

LEADER

etcd-0.example.com etcd-1.example.com etcd-2.example.com

slide-12
SLIDE 12

jetstack.io

etcd-0.example.com etcd-1.example.com

LEADER

etcd-0.example.com etcd-1.example.com

slide-13
SLIDE 13

jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-14
SLIDE 14

jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0 Init container: if !bootstrap and firstLaunch { err := addPeerToEtcd(peer) if err != nil { // Comms failure? fail() } markAsLaunched() } Pre-stop hook: err := removePeerFromEtcd(peer) if err != nil { // Uhh… // Shut down anyway? }

slide-15
SLIDE 15

jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0 Init container: if !bootstrap and firstLaunch { err := addPeerToEtcd(peer) if err != nil { // Comms failure? fail() } markAsLaunched() } Pre-stop hook: err := removePeerFromEtcd(peer) if err != nil { // Uhh… // Shut down anyway? } Only if we’re really removing a peer, not if we just move the Pod...

slide-16
SLIDE 16

jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0 Init container: if !bootstrap and firstLaunch { err := addPeerToEtcd(peer) if err != nil { // Comms failure? fail() } markAsLaunched() } Pre-stop hook: err := removePeerFromEtcd(peer) if err != nil { // Uhh… // Shut down anyway? } Only if we’re really removing a peer, not if we just move the Pod...

slide-17
SLIDE 17

jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0 Init container: if !bootstrap and firstLaunch { err := addPeerToEtcd(peer) if err != nil { // Comms failure? fail() } markAsLaunched() } Pre-stop hook: err := removePeerFromEtcd(peer) if err != nil { // Uhh… // Shut down anyway? } If we scale to zero do we delete all the PVCs and all the data? Detect firstLaunch by presence of file on the filesystem? What if we crash halfway through adding ourselves to etcd? Can we retry safely? Detect bootstrap by trying to dial the cluster? What if it’s just a comms failure? How do we recover from that? If we are bootstrapping, how do we determine the domain names of all the other peers? Query the STS to check the size? What if someone scales it while we’re bootstrapping? Means the etcd pod needs permissions to list stateful sets... Only if we’re really removing a peer, not if we just move the Pod...

slide-18
SLIDE 18

We need an Operator.

jetstack.io

slide-19
SLIDE 19

jetstack.io kubernetes.io/docs/concepts/extend-kubernetes/operator

“Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components.”

slide-20
SLIDE 20

jetstack.io kubernetes.io/docs/concepts/extend-kubernetes/operator

“The Operator pattern aims to capture the key aim of a human operator […] Human operators […] have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems.”

slide-21
SLIDE 21

An Operator encodes knowledge.

jetstack.io

slide-22
SLIDE 22

jetstack.io

cert-manager.io strimzi.io

And many more

slide-23
SLIDE 23

jetstack.io coreos.com/blog/introducing-operators

slide-24
SLIDE 24

“How do you actually build an Operator?”

jetstack.io

slide-25
SLIDE 25

jetstack.io

EtcdCluster

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-26
SLIDE 26

jetstack.io

$ kubectl api-resources NAME SHORTNAMES APIGROUP NAMESPACED KIND deployments deploy apps true Deployment replicasets rs apps true ReplicaSet statefulsets sts apps true StatefulSet pods po true Pod etcdclusters etcd.improbable.io true EtcdCluster < many lines omitted >

slide-27
SLIDE 27

jetstack.io

eco-system

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-28
SLIDE 28

jetstack.io

  • apiGroups:
  • apps

resources:

  • replicasets

verbs:

  • create
  • get
  • list
  • watch
  • apiGroups:
  • ""

resources:

  • services

verbs:

  • create
  • get
  • list
  • watch
  • apiGroups:
  • etcd.improbable.io

resources:

  • etcdclusters

verbs:

  • get
  • list
  • watch

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-29
SLIDE 29

jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-30
SLIDE 30

jetstack.io

slide-31
SLIDE 31

jetstack.io github.com/kubernetes-sigs/kubebuilder

slide-32
SLIDE 32

jetstack.io

Operator logic

slide-33
SLIDE 33

jetstack.io

etcd-cluster-operator EtcdCluster 1 Replica Set, Persistent Volume Claim, Service, etc. 2

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-34
SLIDE 34

jetstack.io

etcd-cluster-operator EtcdCluster 1 Replica Set, Persistent Volume Claim, Service, etc. 3 2

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-35
SLIDE 35

jetstack.io

EtcdCluster EtcdPeer

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-36
SLIDE 36

jetstack.io

EtcdCluster 1 EtcdPeer, Service 3 2 EtcdPeer 4 Replica Set, Persistent Volume Claim 5

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-37
SLIDE 37

jetstack.io

Design considerations

slide-38
SLIDE 38

jetstack.io

Be level-triggered.

slide-39
SLIDE 39

jetstack.io

Do one thing at a time.

slide-40
SLIDE 40

jetstack.io

The cache might lie to you.

slide-41
SLIDE 41

jetstack.io

Deploying an etcd Cluster

slide-42
SLIDE 42

jetstack.io

apiVersion: etcd.improbable.io/v1alpha1 kind: EtcdCluster metadata: name: my-etcd namespace: foo spec: replicas: 3

slide-43
SLIDE 43

jetstack.io

my-etcd foo my-etcd

CLUSTER PEER

my-etcd-0

PEER

my-etcd-1

PEER

my-etcd-2

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0 Custom Resource

slide-44
SLIDE 44

jetstack.io

EtcdCluster 1 EtcdPeer, Service 3 2 EtcdPeer 4 Replica Set, Persistent Volume Claim 5

¯\_(ツ)_/¯

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-45
SLIDE 45

jetstack.io

apiVersion: etcd.improbable.io/v1alpha1 kind: EtcdPeer metadata: name: my-etcd-0 Namespace: foo spec: clusterName: my-etcd bootstrap: initialClusterState: New static: initialCluster:

  • name: my-etcd-0

host: my-etcd-0.my-etcd.foo.svc

  • name: my-etcd-1

host: my-etcd-1.my-etcd.foo.svc

  • name: my-etcd-2

host: my-etcd-2.my-etcd.foo.svc

slide-46
SLIDE 46

jetstack.io

my-etcd-0 my-etcd foo my-etcd

CLUSTER PEER

my-etcd-0 my-etcd-0 my-etcd-0.my-etcd.foo.svc my-etcd.foo.svc

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0 Custom Resource

slide-47
SLIDE 47

jetstack.io

my-etcd-0.my-etcd.foo.svc my-etcd-1.my-etcd.foo.svc my-etcd-2.my-etcd.foo.svc my-etcd-0.my-etcd.foo.svc my-etcd-1.my-etcd.foo.svc my-etcd-2.my-etcd.foo.svc

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-48
SLIDE 48

jetstack.io

Scale up

slide-49
SLIDE 49

jetstack.io

apiVersion: etcd.improbable.io/v1alpha1 kind: EtcdCluster metadata: name: my-etcd spec: replicas: 3 5

slide-50
SLIDE 50

jetstack.io

$ kubectl scale etcdcluster my-etcd --replicas 5

slide-51
SLIDE 51

jetstack.io

my-etcd-0.my-etcd.foo.svc my-etcd-1.my-etcd.foo.svc my-etcd-2.my-etcd.foo.svc my-etcd-3.my-etcd.foo.svc my-etcd-0.my-etcd.foo.svc my-etcd-1.my-etcd.foo.svc my-etcd-2.my-etcd.foo.svc

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-52
SLIDE 52

jetstack.io

my-etcd foo my-etcd

CLUSTER PEER

my-etcd-0

PEER

my-etcd-1

PEER

my-etcd-2

PEER

my-etcd-3

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0 Custom Resource

slide-53
SLIDE 53

jetstack.io

apiVersion: etcd.improbable.io/v1alpha1 kind: EtcdPeer metadata: name: my-etcd-3 namespace: foo spec: clusterName: my-etcd bootstrap: initialClusterState: Existing static: initialCluster:

  • name: my-etcd-0

host: my-etcd-0.my-etcd.foo.svc

  • name: my-etcd-1

host: my-etcd-1.my-etcd.foo.svc

  • name: my-etcd-2

host: my-etcd-2.my-etcd.foo.svc

  • name: my-etcd-3

host: my-etcd-3.my-etcd.foo.svc

slide-54
SLIDE 54

jetstack.io

my-etcd foo my-etcd

CLUSTER PEER

my-etcd-0 my-etcd-0 my-etcd-0

PEER

my-etcd-2 my-etcd-2 my-etcd-2

PEER

my-etcd-3 my-etcd-3 my-etcd-3

PEER

my-etcd-1 my-etcd-1 my-etcd-1

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0 Custom Resource

slide-55
SLIDE 55

jetstack.io

Scale down

slide-56
SLIDE 56

jetstack.io

$ kubectl scale etcdcluster my-etcd --replicas 1

slide-57
SLIDE 57

jetstack.io

my-etcd-0.my-etcd.foo.svc my-etcd-1.my-etcd.foo.svc my-etcd-2.my-etcd.foo.svc my-etcd-0.my-etcd.foo.svc my-etcd-1.my-etcd.foo.svc my-etcd-2.my-etcd.foo.svc

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-58
SLIDE 58

jetstack.io

my-etcd foo my-etcd

CLUSTER PEER

my-etcd-0

PEER

my-etcd-1

PEER

my-etcd-2

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0 Custom Resource

slide-59
SLIDE 59

jetstack.io

my-etcd foo my-etcd

CLUSTER PEER

my-etcd-0

PEER

my-etcd-1

PEER

my-etcd-2 finalizer

Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0 Custom Resource

slide-60
SLIDE 60

jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

my-etcd foo my-etcd

CLUSTER PEER

my-etcd-0

Custom Resource

slide-61
SLIDE 61

jetstack.io

Other features

  • Version upgrade
  • Backup
  • Restore
slide-62
SLIDE 62

jetstack.io

Testing

slide-63
SLIDE 63

jetstack.io Go Gopher — Renee French — CC-BY-3.0 Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

slide-64
SLIDE 64

jetstack.io Go Gopher — Renee French — CC-BY-3.0 Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

mocked!

slide-65
SLIDE 65

jetstack.io

slide-66
SLIDE 66

jetstack.io

$ kind create cluster 1 2 $ docker build . Load images & deploy operator 3

Go Gopher — Renee French — CC-BY-3.0 Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

Deploy an EtcdCluster and assert on behaviour 4

Custom Resource

slide-67
SLIDE 67

jetstack.io

What did we learn?

slide-68
SLIDE 68

jetstack.io

Operators provide value for applications with complex run-books.

slide-69
SLIDE 69

jetstack.io

Operators expose application specific APIs that work with existing tooling.

slide-70
SLIDE 70

jetstack.io

You can build an Operator in any stack, but Go was right for us.

slide-71
SLIDE 71

jetstack.io

You can end-to-end test Operators on your laptop with kind.

slide-72
SLIDE 72

jetstack.io github.com/improbable-eng/etcd-cluster-operator

Questions?

slide-73
SLIDE 73

james.laverack@jetstack.io @JamesLaverack @JetstackHQ

Thank you.

jetstack.io We’re hiring! — jetstack.io/careers