MULTITENANCY IN KUBERNETES WHAT COMPANIES CARE ABOUT Velocity - - PowerPoint PPT Presentation

MULTITENANCY IN KUBERNETES

WHAT COMPANIES CARE ABOUT

Velocity

2

Cost

Hello!

I AM KATHARINA PROBST

I’m a Senior Engineering Manager at Google. You can find me at www.linkedin.com/in/katharina.probst

3

WHY MULTITENANCY

4

KUBERNETES AT A GLANCE

5

master user CLI/API/UI node node node NODES

ONE USER, ONE CLUSTER

6

master user CLI/API/UI node node node NODES

MULTIPLE USERS, MULTIPLE CLUSTERS

7

master user CLI/API/UI node node node NODES master user CLI/API/UI node node node NODES

SPRAWL OF MANY CLUSTERS

8

HOW DOES THIS SCALE FINANCIALLY?

9

HOW DOES THIS SCALE OPERATIONALLY?

10

...

Namespace 2 Namespace 1

...

MANY USERS, ONE CLUSTER

11

master User 2 CLI/API/UI User 3 User 1 User n Namespace n

MULTITENANCY USERS

12

CLUSTER ADMIN

• Set up cluster and
• namespaces. Set up

resource limits.

• Ensure consistency

across namespaces in the cluster.

• Operate the clusters

(e.g., respond to incidents).

CLUSTER USER

• Start/stop/manage their
• wn app(s) in their own

namespace(s).

• Understand their

namespace(s)’ resource limits.

• Don’t trample on other

tenants.

NAMESPACE ADMIN

Admin rights to specific namespace(s)

MODELS OF MULTITENANCY

13

HARD MULTITENANCY

• Zero-trust tenants
• Not yet widely used in production
• Ongoing work in kubernetes community to strengthen

14

WHAT WILL HARD MULTITENANCY TAKE?

15

WHAT WILL HARD MULTITENANCY TAKE?

• Tenants can’t

▫ DoS/impact access to others’ resources ▫ See each others’ stufg (e.g., by intercepting network traffjc or accessing stored data)

16

WHAT WILL HARD MULTITENANCY TAKE?

• Tenants can’t

▫ DoS/impact access to others’ resources ▫ See each others’ stufg (e.g., by intercepting network traffjc or accessing stored data)

• Resource objects don’t collide, e.g., custom controllers/CRDs

17

WHAT WILL HARD MULTITENANCY TAKE?

• Tenants can’t

▫ DoS/impact access to others’ resources ▫ See each others’ stufg (e.g., by intercepting network traffjc or accessing stored data)

• Resource objects don’t collide, e.g., custom controllers/CRDs
• Control plane (master) resources are shared fairly

MODELS OF MULTITENANCY

18

SOFT MULTITENANCY

• Tenants are more trusted
• Often used within one enterprise
• Difgerent teams → difgerent namespaces
• Used in practice and in production
• Often used in combination with infrastructure/platform built on

top by centralized team for consistent security, networking, etc.

MULTITENANCY PRIMITIVES

19

ACCESS CONTROL

Use policies to ensure that tenants can access only what they should have access to

FAIR SHARING

Enforce limits per tenant [Better developed on data plane; control plane WIP]

ISOLATION

Ensure tenants cannot access each others’ workloads, secrets, etc. (security isolation)

ACCESS CONTROL - RBAC

20

ClusterRole A pre-set of capabilities, cluster-wide Role Like ClusterRole, but namespace-scoped ClusterRoleBinding Give permissions defined in a ClusterRole RoleBinding Like ClusterRoleBinding, but namespace-scoped

• Use RBAC for controlling access to Secrets etc.
• Pod Security Policy (e.g., access to volume types, privileged): enable

fine-grained authorization of pod creation and update

• Network Policy (ingress, egress): control which pods can talk to each other
• Make Custom Resource Definitions namespace-scoped
• Sandboxes, e.g., gVisor

▫ Ensure security isolation of pods ▫ Ensure that information is not leaked between untrusted tenants

21

ISOLATION

• Resource Quotas (with Limit Range defaults) for CPU, memory, object

counts

• Pod Priority, Quality of Service Classes
• Node

▫ Taints & Tolerations: allow a node to repel a set of pods ▫ Labels and node selectors (less flexible than Affjnity)

• Pod

▫ Affjnity ▫ Pod Anti-affjnity

22

FAIR SHARING - DATA PLANE (NODES)

23

DATA PLANE VS. CONTROL PLANE MULTITENANCY

Much of what we’ve talked about is sharing the data plane (nodes)

24

CONTROL PLANE MULTITENANCY

...

Namespace 2 Namespace 1

...

MANY USERS, ONE CLUSTER

25

master User 2 CLI/API/UI User 3 User 1 User n Namespace n

Not able to manage multiple clusters → 1:1 mapping between master and cluster, though cluster may have multiple tenants

26

MULTITENANT API SERVER?

...

Namespace 2 Namespace 1

...

MANY USERS, ONE CLUSTER

27

master User 2 CLI/API/UI User 3 User 1 User n Namespace n

All tenants share master (incl. Secrets, ConfigMap), but RBAC helps Little protection against individual tenants DoSing each other

28

MULTITENANT API SERVER?

Max inflight requests: mechanism for protecting API server against CPU and memory overloads Current problem to address: Tenants can crowd each other out (accidentally or

• n purpose)

29

BETTER API SERVER FAIRNESS (UNDERWAY)

Proposal generalizes … max-in-flight request handler in the apiserver to make more distinctions among requests and provide prioritization and fairness among the categories of requests.”

30

BETTER API SERVER FAIRNESS (UNDERWAY)

BETTER API SERVER FAIRNESS (UNDERWAY)

Multiple priority levels, each has queues Within each priority level, queues compete evenly

31

WHAT COMPANIES CARE ABOUT

Velocity

32

Cost

• Use multitenancy for improved resource effjciency, cost, and operations
• Difgerent models of multitenancy

▫ Soft ▫ Hard

• Hard multitenancy is still work in progress
• Soft multitenancy is already used in production by various companies

▫ But is often coupled with a shared internal platform to gain consistency of networking, security, etc. across teams

33

KEY TAKE-AWAYS

QUESTIONS?

• Project plan for multitenancy

▫ Building CRD for more automatic management of namespaces

• API Machinery KEP for improved resource sharing
• Cluster multi-tenancy for a good overview

35

Links for additional details