MULTITENANCY IN KUBERNETES WHAT COMPANIES CARE ABOUT Velocity - PowerPoint PPT Presentation

MULTITENANCY IN KUBERNETES

WHAT COMPANIES CARE ABOUT Velocity Cost 2

Hello! I AM KATHARINA PROBST I’m a Senior Engineering Manager at Google. You can find me at www.linkedin.com/in/katharina.probst 3

WHY MULTITENANCY 4

KUBERNETES AT A GLANCE node master user CLI/API/UI node node NODES 5

ONE USER, ONE CLUSTER node master user CLI/API/UI node node NODES 6

MULTIPLE USERS, MULTIPLE CLUSTERS node node master master user CLI/API/UI node user CLI/API/UI node node node NODES NODES 7

SPRAWL OF MANY CLUSTERS kubelet kubelet CLI/API/ user master kubelet UI CLI/API/ master user kubelet UI kubelet kubelet kubelet CLI/API/ user master kubelet UI NODES kubelet NODES kubelet NODES CLI/API/ user master kubelet UI kubelet kubelet NODES CLI/API/ user master kubelet UI kubelet kubelet kubelet CLI/API/ user master kubelet UI CLI/API/ master user kubelet UI kubelet NODES kubelet NODES kubelet NODES CLI/API/ user master kubelet UI kubelet NODES 8

HOW DOES THIS SCALE FINANCIALLY? kubelet kubelet CLI/API/ user master kubelet UI CLI/API/ master user kubelet UI kubelet kubelet kubelet CLI/API/ user master kubelet UI NODES kubelet NODES kubelet NODES CLI/API/ user master kubelet UI kubelet kubelet NODES CLI/API/ user master kubelet UI kubelet kubelet kubelet CLI/API/ user master kubelet UI CLI/API/ master user kubelet UI kubelet NODES kubelet NODES kubelet NODES CLI/API/ user master kubelet UI kubelet NODES 9

HOW DOES THIS SCALE OPERATIONALLY? kubelet kubelet CLI/API/ user master kubelet UI CLI/API/ master user kubelet UI kubelet kubelet kubelet CLI/API/ user master kubelet UI NODES kubelet NODES kubelet NODES CLI/API/ user master kubelet UI kubelet kubelet NODES CLI/API/ user master kubelet UI kubelet kubelet kubelet CLI/API/ user master kubelet UI CLI/API/ master user kubelet UI kubelet NODES kubelet NODES kubelet NODES CLI/API/ user master kubelet UI kubelet NODES 10

MANY USERS, ONE CLUSTER User 1 Namespace 1 User 2 master CLI/API/UI User 3 ... Namespace 2 ... User n Namespace n 11

MULTITENANCY USERS NAMESPACE CLUSTER CLUSTER USER ADMIN ADMIN Admin rights to specific Set up cluster and Start/stop/manage their ◦ ◦ namespace(s) namespaces. Set up own app(s) in their own resource limits. namespace(s). Ensure consistency Understand their ◦ ◦ across namespaces in namespace(s)’ resource the cluster. limits. Operate the clusters Don’t trample on other ◦ ◦ (e.g., respond to tenants. incidents). 12

MODELS OF MULTITENANCY HARD MULTITENANCY Zero-trust tenants ◦ Not yet widely used in production ◦ Ongoing work in kubernetes community to strengthen ◦ 13

WHAT WILL HARD MULTITENANCY TAKE? 14

WHAT WILL HARD MULTITENANCY TAKE? Tenants can’t ◦ DoS/impact access to others’ resources ▫ See each others’ stufg (e.g., by intercepting network traffjc or ▫ accessing stored data) 15

WHAT WILL HARD MULTITENANCY TAKE? Tenants can’t ◦ DoS/impact access to others’ resources ▫ See each others’ stufg (e.g., by intercepting network traffjc or ▫ accessing stored data) Resource objects don’t collide, e.g., custom controllers/CRDs ◦ 16

WHAT WILL HARD MULTITENANCY TAKE? Tenants can’t ◦ DoS/impact access to others’ resources ▫ See each others’ stufg (e.g., by intercepting network traffjc or ▫ accessing stored data) Resource objects don’t collide, e.g., custom controllers/CRDs ◦ Control plane (master) resources are shared fairly ◦ 17

MODELS OF MULTITENANCY SOFT MULTITENANCY Tenants are more trusted ◦ Often used within one enterprise ◦ Difgerent teams → difgerent namespaces ◦ Used in practice and in production ◦ Often used in combination with infrastructure/platform built on ◦ top by centralized team for consistent security, networking, etc. 18

MULTITENANCY PRIMITIVES ACCESS ISOLATION FAIR SHARING CONTROL Ensure tenants cannot Enforce limits per tenant Use policies to ensure access each others’ that tenants can [Better developed on data workloads, secrets, etc. access only what they plane; control plane WIP] (security isolation) should have access to 19

ACCESS CONTROL - RBAC ClusterRole A pre-set of capabilities, cluster-wide Role Like ClusterRole, but namespace-scoped ClusterRoleBinding Give permissions defined in a ClusterRole RoleBinding Like ClusterRoleBinding, but namespace-scoped 20

ISOLATION Use RBAC for controlling access to Secrets etc. ◦ Pod Security Policy (e.g., access to volume types, privileged): enable ◦ fine-grained authorization of pod creation and update Network Policy (ingress, egress): control which pods can talk to each other ◦ Make Custom Resource Definitions namespace-scoped ◦ Sandboxes, e.g., gVisor ◦ Ensure security isolation of pods ▫ Ensure that information is not leaked between untrusted tenants ▫ 21

FAIR SHARING - DATA PLANE (NODES) Resource Quotas (with Limit Range defaults) for CPU, memory, object ◦ counts Pod Priority, Quality of Service Classes ◦ Node ◦ Taints & Tolerations: allow a node to repel a set of pods ▫ Labels and node selectors (less flexible than Affjnity) ▫ Pod ◦ Affjnity ▫ Pod Anti-affjnity ▫ 22

DATA PLANE VS. CONTROL PLANE MULTITENANCY 23

CONTROL PLANE MULTITENANCY Much of what we’ve talked about is sharing the data plane (nodes) 24

MANY USERS, ONE CLUSTER User 1 Namespace 1 User 2 master CLI/API/UI User 3 ... Namespace 2 ... User n Namespace n 25

MULTITENANT API SERVER? Not able to manage multiple clusters → 1:1 mapping between master and cluster , though cluster may have multiple tenants 26

MANY USERS, ONE CLUSTER User 1 Namespace 1 User 2 master CLI/API/UI User 3 ... Namespace 2 ... User n Namespace n 27

MULTITENANT API SERVER? All tenants share master (incl. Secrets, ConfigMap), but RBAC helps Little protection against individual tenants DoSing each other 28

BETTER API SERVER FAIRNESS (UNDERWAY) Max inflight requests: mechanism for protecting API server against CPU and memory overloads Current problem to address: Tenants can crowd each other out (accidentally or on purpose) 29

BETTER API SERVER FAIRNESS (UNDERWAY) Proposal generalizes … max-in-flight request handler in the apiserver to make more distinctions among requests and provide prioritization and fairness among the categories of requests.” 30

BETTER API SERVER FAIRNESS (UNDERWAY) Multiple priority levels, each has queues Within each priority level, queues compete evenly 31

WHAT COMPANIES CARE ABOUT Velocity Cost 32

KEY TAKE-AWAYS Use multitenancy for improved resource effjciency, cost, and operations ◦ Difgerent models of multitenancy ◦ Soft ▫ Hard ▫ Hard multitenancy is still work in progress ◦ Soft multitenancy is already used in production by various companies ◦ But is often coupled with a shared internal platform to gain ▫ consistency of networking, security, etc. across teams 33

QUESTIONS?

Links for additional details Project plan for multitenancy ◦ Building CRD for more automatic management of namespaces ▫ API Machinery KEP for improved resource sharing ◦ Cluster multi-tenancy for a good overview ◦ 35