DevOps: Where is My PodPod Hello! I am smalltown MaiCoin Site - - PowerPoint PPT Presentation

DevOps: Where is My PodPod

Hello!

I am smalltown

MaiCoin Site Reliability Engineer Taipei HashiCorp UG Organizer AWS UG Taiwan Staff

• GUI Driven
• Ticket Based
• Hand Crafted
• Reserved
• Scale-Up
• Smart Hardware
• Proprietary
• Waterfall Ops
• ...
• API Driven
• Self Service
• Automated
• On Demand
• Scale-Out
• Smart Apps
• Open Source
• Agile DevOps
• ...

Pets vs Cattle

Kubernetes = Cattle Pattern

After Using Kubernetes?

Livestock Industry Requires Expertise

System

Feeding Breeding Animal Health Range of Species Product

The Same Thing Happened in K8S

Pod is Pending

Node Not Ready

App Not Redundancy Out of Resource Pod Not in Right Node Interfere W/ Each Other

Yes, You are Involved in Livestock Industry Now!

真的變成 “碼農” 了...

Cluster Pattern Resource Management Pod Arrangement

Cluster Pattern Resource Management Pod Arrangement

How to Arrange Application Workload?

• If There are 3 Applications, 3 Environments (Alpha, Beta,

Production) ...

• Run All Application Instances on a Single Cluster?
• A Separate Cluster for Each Application Instance?
• A Combination of the Above?

One Large Shared Cluster

👎 Efficient Resource Usage 👎 Cheap 👎 Efficient Administration 👏 Single Point of Failure 👏 No Hard Security Isolation 👏 No Hard Multi-Tenancy 👏 Many Users 👏 Clusters Can't Grow Infinitely Large

Alpha Beta Prod Alpha Beta Prod Alpha Beta Prod

Many Small Single-Use Clusters

👎 Reduced Blast Radius 👎 Isolation 👎 Few Users 👏 Inefficient Resource Usage 👏 Expensive 👏 Complex Administration

Alpha Beta Prod Alpha Beta Prod Alpha Beta Prod

Cluster per Application

👎 Cluster Can be Customised for an App 👏 Different Environments in the Same Cluster

Alpha Beta Prod Alpha Beta Prod Alpha Beta Prod

Cluster per Environment

👎 Isolation of the Prod Environment 👎 Cluster can be Customised for an Environment 👎 Lock Down Access to Prod Cluster 👏 Lack of Isolation Between Apps 👏 App Requirements are Not Localised

Alpha Beta Prod

Which One is Better?

• Depends on Your Use Case
• Trade-Off the Pros and Cons of the Different Approaches
• The Choice is Not Limited to the Above Examples
• It can be Any Combination of Them!

Ref

Multiple (Availability) Zones

• Multiple, Isolated Locations Within Each Region
• Protect your Application Against (Availability) Zone

Disruption

Network Latency

• Take AWS for Example, Inter-AZ Network Latency Depends
• n Different Region, General Below 10 ms
• Does It Matter?

Persistent Volume

• High Efficiency Storage and Pod Need to Stay in the Same

(Availability) Zone

• What is the Problem?

Extra Cost

• AWS/Azure/GCP Regional Data Transfer is Charged at $

0.01/GB

• Large Amount of Data Transfer will Lead to Huge Cost

(GitLab)

Cluster Pattern Resource Management Pod Arrangement

How to Put Pod in the Right Node

• Dedicated Nodes
• Nodes with Special Hardware
• Taint based Evictions

Node Selector

apiVersion: v1 kind: Pod ... spec: containers:

• name: cattle

image: cattle imagePullPolicy: IfNotPresent nodeSelector: land: grass

land:grass land:grass ❤

Node Affinity - Required

apiVersion: v1 kind: Pod metadata: name: with-node-affinity spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms:

• matchExpressions:
• key: kubernetes.io/land
• perator: In

values:

• pasture-1
• pasture-2

...

kubernetes.io/land: pasture-1 http://kubernetes.io/land: pasture-1 or pasture-2 ❤

Node Affinity - Preferred

╮（╯＿╰）╭ apiVersion: v1 kind: Pod metadata: name: with-node-affinity spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution:

• weight: 1

preference: matchExpressions:

• key: kubernetes.io/land
• perator: In

values:

• pasture-1
• pasture-2

...

http://kubernetes.io/land: pasture-1 or pasture-1 😣 kubernetes.io/land: pasture-3

Taint

apiVersion: v1 kind: Pod metadata: name: cattle labels: env: test spec: containers:

• name: cattle

image: cattle imagePullPolicy: IfNotPresent

land=mud:NoSchedule

Toleration

... spec: containers:

• name: pig

image: pig imagePullPolicy: IfNotPresent tolerations:

• key: "land"
• perator: "Equal"

value: "mud" effect: "NoSchedule"

land=mud:NoSchedule

Inter-Pod Affinity

apiVersion: v1 kind: Pod metadata: name: with-pod-affinity spec: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution:

• labelSelector:

matchExpressions:

• key: species
• perator: In

values:

• cattle

topologyKey: failure-domain.beta.kubernetes.io/land land:grass land:grass land:mud land:mud

Inter-Pod Anti-Affinity

apiVersion: v1 kind: Pod metadata: name: with-pod-affinity spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution:

• labelSelector:

matchExpressions:

• key: species
• perator: In

values:

• cattle

topologyKey: failure-domain.beta.kubernetes.io/land land:grass land:mud

Why Need PodTopologySpread?

apiVersion: v1 kind: Pod metadata: name: with-pod-affinity spec: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution:

• labelSelector:

matchExpressions:

• key: species
• perator: In

values:

• cattle

topologyKey: failure-domain.beta.kubernetes.io/land land:grass land:grass land:mud land:mud

spec: topologySpreadConstraints:

• maxSkew: 1

topologyKey: land whenUnsatisfiable: DoNotSchedule labelSelector: species: cattle

How PodTopologySpread Work?

land:grass land:grass skew=3 ❌ skew=0 ✅

Cluster Pattern Resource Management Pod Arrangement

Why Need Resource Management?

• Avoid Out of Control Application Affect Others
• Application Support Scale Out Ability
• Easy to Plan Cluster Overall Capability
• Ensure The Most Important Application Survival and Safety
• ...

Everyone Knows Resource Request & Limit

Ref

• Default Memory Requests and Limits for a Namespace
• Default CPU Requests and Limits for a Namespace
• Minimum and Maximum Memory Constraints for a

Namespace

• Minimum and Maximum CPU Constraints for a Namespace
• Memory and CPU Quotas for a Namespace
• Pod Quota for a Namespace

When K8S Users Ignore You 😇

But Do You Know Pod QoS?

• Guaranteed: Every Container in the Pod Must Have a

Memory/CPU Limit and a Memory/CPU Request, and They Must be the Same

• Burstable: Not Meet the Criteria for QoS Class

Guaranteed, and At Least one Container in the Pod has a Memory or CPU Request

• BestEffort: Not Have Any Memory or CPU Limits or

Requests

When Out of Resource...

• BestEffort Pods
• Burstable Pods Whose Resource Usage Exceeds Its Request
• Burstable Pods Whose Resource Usage is Beneath Its

Request

• Guaranteed Pods

฀฀

Pod Disruptions

• Voluntary and Involuntary Disruptions
• Dealing with Disruptions

○ Ensure Pod Requests Appropriate Resources ○ Replicate Your Application ○ Spread Applications Across Racks (Using Anti-Affinity)

• r Across Zones (if Using a Multi-Zone Cluster)

Perform a Disruptive Action on All the Nodes

• Accept Downtime
• Failover to Another Complete Replica Cluster
• Use Pod Disruption Budget

Pod Disruption Budget (1/6)

PDB = At Least 2 of The 3 Pods to be Available at All Times

Pod Disruption Budget (2/6)

PDB = At Least 2 of The 3 Pods to be Available at All Times

Pod Disruption Budget (3/6)

PDB = At Least 2 of The 3 Pods to be Available at All Times

Pod Disruption Budget (4/6)

PDB = At Least 2 of The 3 Pods to be Available at All Times

Pod Disruption Budget (5/6)

PDB = At Least 2 of The 3 Pods to be Available at All Times

Pod Disruption Budget (6/6)

PDB = At Least 2 of The 3 Pods to be Available at All Times

Pod Priority and Preemption

apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority-nonpreempting value: 1000000 preemptionPolicy: Never/PreemptLowerPriority globalDefault: false/true description: "Pod Priority and Preemption"

฀฀

THANKS!

ANY QUESTIONS? You can find me at my office:

• Frontend Engineer
• Backend Engineer