Highly available cross-region deployments with Kubernetes Bastian - - PowerPoint PPT Presentation

▶
highly available cross region deployments with kubernetes
SMART_READER_LITE
LIVE PREVIEW

Highly available cross-region deployments with Kubernetes Bastian - - PowerPoint PPT Presentation

Sep 28, 2022 984 likes •2.4k views

Highly available cross-region deployments with Kubernetes Bastian Hofmann @BastianHofmann Container orchestration platform Deploy, run and scale your services in isolated containers No vendor lock in Standardized APIs Runs on Your laptop

slide-1
SLIDE 1

@BastianHofmann

Highly available cross-region deployments with Kubernetes

Bastian Hofmann

slide-2
SLIDE 2
slide-3
SLIDE 3

Container orchestration platform

slide-4
SLIDE 4

Deploy, run and scale your services in isolated containers

slide-5
SLIDE 5

No vendor lock in

slide-6
SLIDE 6

Standardized APIs

slide-7
SLIDE 7

Runs on

slide-8
SLIDE 8

Your laptop

slide-9
SLIDE 9

Bare metal

slide-10
SLIDE 10

Cloud Providers

slide-11
SLIDE 11

And if you don't want to install and maintain Kubernetes yourself

slide-12
SLIDE 12

Managed Kubernetes

slide-13
SLIDE 13
slide-14
SLIDE 14

Self-healing built in

slide-15
SLIDE 15

But what happens when a complete datacenter is not available

slide-16
SLIDE 16

Multi-Region Kubernetes Setups

slide-17
SLIDE 17

Why

slide-18
SLIDE 18

Availability

slide-19
SLIDE 19

Scalability

slide-20
SLIDE 20

More Points-of-Presence

slide-21
SLIDE 21

Reduce dependencies on one single cloud provider

slide-22
SLIDE 22

Because of the standardised API across providers Kubernetes can help

slide-23
SLIDE 23

Features

slide-24
SLIDE 24

Scheduling by Node Labels

slide-25
SLIDE 25

failure- domain.beta.kubernetes.io/ region=dbl failure- domain.beta.kubernetes.io/ zone=dbl1

slide-26
SLIDE 26

Regions vs availability zone

slide-27
SLIDE 27

failure- domain.beta.kubernetes.io/ region=dbl failure- domain.beta.kubernetes.io/ zone=dbl1

slide-28
SLIDE 28

apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: template: spec: containers:

  • image: nginx

name: nginx nodeSelector: failure-domain.beta.kubernetes.io/region: dbl

slide-29
SLIDE 29

Affinities

slide-30
SLIDE 30

spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution:

  • topologyKey: "failure-domain.beta.kubernetes.io/

region" labelSelector: matchLabels: app: nginx

slide-31
SLIDE 31

Service discovery with built in DNS

slide-32
SLIDE 32

apiVersion: v1 kind: Service metadata: name: nginx spec: type: ClusterIP ports:

  • port: 80

targetPort: 80 selector: app: nginx

slide-33
SLIDE 33

External LoadBalancers

slide-34
SLIDE 34

apiVersion: v1 kind: Service metadata: name: nginx spec: type: LoadBalancer ports:

  • port: 80

targetPort: 80 selector: app: nginx

slide-35
SLIDE 35

external-dns

slide-36
SLIDE 36

StorageClasses

slide-37
SLIDE 37

Some storage providers support dynamic volume provisioning

slide-38
SLIDE 38

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: standard provisioner: kubernetes.io/gce-pd parameters: type: pd-standard volumeBindingMode: WaitForFirstConsumer allowedTopologies:

  • matchLabelExpressions:
  • key: failure-domain.beta.kubernetes.io/zone

values:

  • us-central1-a
  • us-central1-b
slide-39
SLIDE 39

Challenges

slide-40
SLIDE 40

Especially over different regions

slide-41
SLIDE 41

External load balancing and traffic management

slide-42
SLIDE 42

Internal load balancing

slide-43
SLIDE 43

Connectivity between private networks

slide-44
SLIDE 44

Kubernetes DNS across multiple clusters

slide-45
SLIDE 45

Latencies

slide-46
SLIDE 46

Beskow, Paul & Vik, Knut-Helge & Halvorsen, Pål & Griwodz, Carsten. (2009). The partial migration of game state and dynamic server selection to reduce latency. Multimedia Tools Appl..

  • 45. 83-107. 10.1007/s11042-009-0287-7.
slide-47
SLIDE 47

Synchronising Deployments across multiple clusters

slide-48
SLIDE 48

Storage

slide-49
SLIDE 49

Data replication

slide-50
SLIDE 50

Split-brain problem

slide-51
SLIDE 51

Possible setups

slide-52
SLIDE 52

One cluster across AZs in one region

slide-53
SLIDE 53

One cluster across regions

slide-54
SLIDE 54

Multiple clusters connected via VPN

slide-55
SLIDE 55

One cluster setup

slide-56
SLIDE 56

One Kubernetes cluster across multiple availability zones

slide-57
SLIDE 57

Zone 1 Zone 2

Kubernetes Master Components Nginx Nginx Nginx Service Curl

slide-58
SLIDE 58

Easiest option

slide-59
SLIDE 59

Low latencies

slide-60
SLIDE 60

You can use one network

slide-61
SLIDE 61

All pods and services can talk with each other

slide-62
SLIDE 62

One cluster => one internal DNS

slide-63
SLIDE 63

Service Discovery and internal load balancing works

slide-64
SLIDE 64

External LoadBalancers work

slide-65
SLIDE 65

Zone 1 Zone 2

Kubernetes Master Components Nginx Nginx Nginx Service Curl LB

slide-66
SLIDE 66

You can mount storage across AZ

slide-67
SLIDE 67

One cluster state => One deployment

slide-68
SLIDE 68

But what if the whole region is down?

slide-69
SLIDE 69

One Kubernetes cluster across multiple regions and VPN connection between networks

slide-70
SLIDE 70

Region 1 Region 2

WireGuard VPN Kubernetes Master Components Nginx Nginx Nginx Service Curl

slide-71
SLIDE 71

Latencies get higher by distance

slide-72
SLIDE 72

You need a VPN to connect the networks

slide-73
SLIDE 73

WireGuard as a VPN

slide-74
SLIDE 74

https:/ /github.com/squat/kilo

slide-75
SLIDE 75

Kilo gets all Nodes from Kubernetes

slide-76
SLIDE 76

Discovers regions and zones by Kubernetes Node labels

slide-77
SLIDE 77

Configures WireGuard

slide-78
SLIDE 78

Routing tables

slide-79
SLIDE 79

All pods and services can talk with each other

slide-80
SLIDE 80

One cluster => one internal DNS

slide-81
SLIDE 81

Service Discovery and internal load balancing works

slide-82
SLIDE 82

You need a solution for external load balancing

slide-83
SLIDE 83

Region 1 Region 2

WireGuard VPN Kubernetes Master Components Nginx Nginx Nginx Service Curl LB LB

slide-84
SLIDE 84

You have to replicate storage yourself

slide-85
SLIDE 85

Not every Storage provider supports dynamic volume provisioning

slide-86
SLIDE 86

One cluster state => One deployment

slide-87
SLIDE 87

Demo

slide-88
SLIDE 88

Multi cluster setup

slide-89
SLIDE 89

Connecting multiple clusters with a VPN

slide-90
SLIDE 90

Cluster 1 Cluster 2

WireGuard VPN Kubernetes Master Components Kubernetes Master Components Nginx Nginx Nginx Service Nginx Service Curl

slide-91
SLIDE 91

WireGuard as a VPN

slide-92
SLIDE 92

https:/ /github.com/squat/kilo

slide-93
SLIDE 93

All pods and services can talk with each other

slide-94
SLIDE 94

Separate clusters => separate internal DNS

slide-95
SLIDE 95

For service discovery configure each internal DNS to resolve to other clusters

slide-96
SLIDE 96

cluster.region2:53 { forward . 10.10.11.10 } .:53 { kubernetes cluster.local cluster.region1 in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } forward . /etc/resolv.conf loop loadbalance }

slide-97
SLIDE 97

cluster.region1:53 { forward . 10.10.10.10 } .:53 { kubernetes cluster.local cluster.region2 in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } forward . /etc/resolv.conf loop loadbalance }

slide-98
SLIDE 98

Internal load balancing works

slide-99
SLIDE 99

You have to replicate storage yourself

slide-100
SLIDE 100

Every cluster has their own StorageClass that works on all nodes

slide-101
SLIDE 101

Separate clusters have separate state

slide-102
SLIDE 102

Management of deployments across clusters

slide-103
SLIDE 103

Kubefed

https://github.com/kubernetes-sigs/kubefed

slide-104
SLIDE 104

FederatedNamespaces, FederatedDeployments, FederatedConfigMaps, FederatedServices, ...

slide-105
SLIDE 105

Cluster aware controller that manages resources in all connected clusters

slide-106
SLIDE 106

Cluster 1 Cluster 2

Kubernetes Master Components Kubernetes Master Components Kubefed Controller

slide-107
SLIDE 107

Cluster 1 Cluster 2

Kubernetes Master Components Kubernetes Master Components Kubefed Controller

FederatedService FederatedDeployment

slide-108
SLIDE 108

Cluster 1 Cluster 2

Kubernetes Master Components Kubernetes Master Components Nginx Service Nginx Service Kubefed Controller Kubefed Controller Nginx Nginx

slide-109
SLIDE 109

Demo

slide-110
SLIDE 110

Solution for external traffic

slide-111
SLIDE 111

Cluster 1 Cluster 2

WireGuard VPN Kubernetes Master Components Kubernetes Master Components Nginx Nginx Nginx Service Nginx Service Curl LB LB

slide-112
SLIDE 112

Multi cluster external-dns

slide-113
SLIDE 113
slide-114
SLIDE 114

service.namespace.domain.svc.example.com service.namespace.domain.svc.region1.example.com service.namespace.domain.svc.region2.example.com

slide-115
SLIDE 115

Demo

slide-116
SLIDE 116

More options

slide-117
SLIDE 117

Multiple clusters connected via Service Mesh (Istio)

slide-118
SLIDE 118

Conclusion

slide-119
SLIDE 119

Kubernetes makes it easier to create multi region setups

slide-120
SLIDE 120

There are still challenges you have to

  • vercome
slide-121
SLIDE 121

Often just physics

slide-122
SLIDE 122

Federation Tooling is just getting started

slide-123
SLIDE 123

Test it 30 days For free Visit us at our booth

slide-124
SLIDE 124
slide-125
SLIDE 125

https:/ /github.com/bashofmann/ kubernetes-multicluster-demos

slide-126
SLIDE 126

mail@bastianhofmann.de https:/ /twitter.com/BastianHofmann

slide-127
SLIDE 127

Connecting multiple clusters with a Service Mesh Gateway

slide-128
SLIDE 128
slide-129
SLIDE 129
slide-130
SLIDE 130

Cluster 1 Cluster 2

Istio Gateway Istio Control Plane Kubernetes Master Components Kubernetes Master Components Istio Nginx Nginx Nginx Service Nginx Service Istio Gateway Curl

slide-131
SLIDE 131

No VPN necessary

slide-132
SLIDE 132

Fairly easy to set up

slide-133
SLIDE 133

Pods from different clusters communicate over public Ips

slide-134
SLIDE 134

Traffic encrypted and authenticated with mutual TLS

slide-135
SLIDE 135

Communication is only possible through Istio proxies

slide-136
SLIDE 136

Easy service discovery

slide-137
SLIDE 137

Flexible, location aware traffic management

slide-138
SLIDE 138

Telemetry and tracing included

slide-139
SLIDE 139

Demo