Developing Kubernetes Services at Airbnb Scale @MELANIECEBULA What - PowerPoint PPT Presentation

@MELANIECEBULA kubernetes config files Production Canary Dev Deployment Deployment Deployment kubectl apply kubernetes Production Canary Dev kubernetes cluster ConfigMap ConfigMap ConfigMap verbose Dev repetitive by Production Canary Service Service namespace Service P

@MELANIECEBULA k tool KUBECTL WRAPPER Production Canary Dev Deployment Deployment Deployment kubectl apply Production Canary Dev kubernetes cluster ConfigMap ConfigMap ConfigMap calls kubectl commands (incl. Dev Production Canary plugins) Service Service Service

@MELANIECEBULA k tool OPINIONATED KUBECTL

@MELANIECEBULA • Runs in the project home directory: 
 $ cd /path/to/bonk $ k status • Environment variables for arguments: k tool standardized $ k status ENV=staging USES ENV VARS namespaces! • Prints the command that it will execute: 
 $ k status ENV=staging kubectl get pods --namespace=bonk-staging

@MELANIECEBULA • k generate generates kubernetes fj les • k build performs project build, docker build and k tool docker push with tags • k deploy creates namespace, applies/replaces SIMPLIFIES BUILDS AND DEPLOYS kubernetes fj les, sleeps and checks deployment status • can chain commands; ex: k all

@MELANIECEBULA • defaults to random pod, main container: $ k ssh ENV=staging k tool • specify particular pod, speci fj c container: A DEBUGGING TOOL $ k logs ENV=staging POD=… CONTAINER=bonk • automates debugging with k diagnose ENV=staging

@MELANIECEBULA • defaults to random pod, main container: call kubectl diagnose $ k ssh ENV=staging k tool • specify particular pod, speci fj c container: A DEBUGGING TOOL $ k logs ENV=staging POD=… CONTAINER=bonk • automates debugging with k diagnose ENV=staging

@MELANIECEBULA What are kubectl plugins?

@MELANIECEBULA What are kubectl plugins?

@MELANIECEBULA k diagnose SETUP deploy bonk service with failing command new pod in CrashLoopBackoff

@MELANIECEBULA k diagnose MANUALLY 1. use “get pods - o=yaml” and look for problems 2. grab logs for unready container

@MELANIECEBULA k diagnose MANUALLY 3. get k8s events related to this pod

@MELANIECEBULA kubectl podevents KUBECTL PLUGIN kubectl podevents plugin

@MELANIECEBULA // defines CLI command and flags kubectl diagnose var Namespace string USES COBRA GO CLI var rootCmd = &cobra.Command{ Use: “kubectl diagnose —namespace<namespace>" Short: “diagnoses a namespace with pods in CrashLoopBackOff” Run: func(cmd *cobra.Command, arg[]string) { // Fill in with program logic } } func Execute() { rootCmd.Flags().StringVarP(&Namespace, "namespace", "n", “”) rootCmd.MarkFlagRequired("namespace") if err := rootCmd.Execute(); err != nil { fmt.Println(err) os.Exit(1) }

@MELANIECEBULA // get pods (assume Namespace is defined) kubectl diagnose kubeconfig := filepath.Join(os.Getenv("HOME"), ".kube","config") USES K8S CLIENT-GO config, err := clientcmd.BuildConfigFromFlags("", kubeconfig) if err != nil { … } clientset, err := kubernetes.NewForConfig(config) if err != nil { … } pods, err := uses k8s client-go clientset.CoreV1().Pods(Namespace).List(metav1.ListOptions{}) and Namespace fmt.Printf("There are %d pods in the namespace %s\n", param to get pods len(pods.Items), Namespace) for _, pod := range pod.Items { podName := pod.Name for _, c := range pod.Status.ContainerStatuses { if c.Ready != true { // print c.LastTerminatedState and c.State } }

@MELANIECEBULA // get pods (assume Namespace is defined) kubectl diagnose kubeconfig := filepath.Join(os.Getenv("HOME"), ".kube","config") USES K8S CLIENT-GO config, err := clientcmd.BuildConfigFromFlags("", kubeconfig) if err != nil { … } clientset, err := kubernetes.NewForConfig(config) if err != nil { … } pods, err := clientset.CoreV1().Pods(Namespace).List(metav1.ListOptions{}) fmt.Printf("There are %d pods in the namespace %s\n", len(pods.Items), Namespace) for _, pod := range pod.Items { podName := pod.Name for _, c := range pod.Status.ContainerStatuses { if c.Ready != true { prints info for all unready containers // print c.LastTerminatedState and c.State } }

@MELANIECEBULA // get pod events for namespace and pod kubectl diagnose cmd := exec.Command("kubectl", "podevents", Namespace, podName) USES OS/EXEC (WHEN LAZY) var out bytes.Buffer podevents kubectl var stderr bytes.Buffer plugin cmd.Stdout = &out cmd.Stderr = &stderr err := cmd.Run() if err != nil { fmt.Println(fmt.Sprint(err) + ": " + stderr.String()) log.Fatal(err) } else { fmt.Println("Events: \n" + out.String()) } } // also grab logs cmd = exec.Command("kubectl", "logs", podname, fmt.Sprintf("-- namespace=%s", Namespace), "-c", "bonk")

@MELANIECEBULA kubectl diagnose GO KUBECTL PLUGIN

@MELANIECEBULA kubectl diagnose GO KUBECTL PLUGIN 1. unready container info

@MELANIECEBULA kubectl diagnose GO KUBECTL PLUGIN 1. unready container info 2. kubectl podevents

@MELANIECEBULA kubectl diagnose GO KUBECTL PLUGIN 1. unready container info 2. kubectl podevents 3. pod logs for unready containers

Takeaways • Create an opinionated kubectl wrapper • Automate common k8s workflows with kubectl plugins

CI/CD

Each step in our CI /CD jobs are RUN steps in a build Dockerfile

runs k commands

DEPLOY PROCESS

@MELANIECEBULA A single deploy process for every change Develop Merge Deploy Write code and config Open a PR and merge Deploy all code and under your project your code to master config changes

@MELANIECEBULA A single deploy process for every change Project Deployment AWS Storage ownership kubectl kubectl apply apply Service kubernetes cluster ConfigMap Alerts Docs Discovery API Gateway Service Dashboards Secrets Routes

@MELANIECEBULA How do we apply k8s configuration? • kubectl apply all files Deployment • in some cases where apply “kubectl fails, replace files without apply” force kubernetes cluster • always restart pods on ConfigMap deploy to pick up changes • return atomic success or failure state by sleeping and checking status Service

@MELANIECEBULA How do you always restart pods on deploy? We add a date label Deployment to the pod spec, which convinces k8s kubectl kubectl to relaunch all pods apply apply kubernetes cluster ConfigMap Service

@MELANIECEBULA How do we apply custom configuration?

@MELANIECEBULA How do we apply custom configuration? kubectl kubectl apply apply kubernetes cluster aws.yml AWS AWS AWS CRD Controller webhook

@MELANIECEBULA How do we apply custom configuration? 1. Create a custom kubectl kubectl apply apply resource definition for aws.yml kubernetes cluster aws.yml AWS AWS AWS CRD Controller webhook

@MELANIECEBULA How do we apply custom configuration? 2. Create a controller kubectl kubectl that calls a web hook apply apply when aws.yml is kubernetes cluster applied aws.yml AWS AWS AWS CRD Controller webhook

@MELANIECEBULA How do we apply custom configuration? 3. Create a web hook kubectl kubectl that updates a apply apply custom resource kubernetes cluster aws.yml AWS AWS AWS CRD Controller webhook

@MELANIECEBULA How do we apply custom configuration? 4. AWS lambda exposes web hook to be called AWS AWS AWS lambda AWS CRD Controller webhook

Takeaways • Code and configuration should be deployed with the same process • Use custom resources and custom controllers to integrate k8s with your infra

VALIDATION

@MELANIECEBULA • enforce best practices • at build time with validation scripts Configuration • at deploy time with admission controller SHOULD BE VALIDATED

@MELANIECEBULA How do we validate configuration at build time?

@MELANIECEBULA How do we validate configuration at build time? project.yml validation script global job dispatcher kube project docs validation validation script build build jobs bonk CI jobs aws .yml validation script global jobs repo

@MELANIECEBULA How do we validate configuration at build time? project.yml validation script global job dispatcher kube project docs validation validation script build build jobs 1. Define global job in bonk CI jobs aws .yml global jobs repo validation script global jobs repo

@MELANIECEBULA How do we validate configuration at build time? project.yml validation script global job dispatcher kube project docs validation validation script build build jobs 2. job dispatcher always dispatches bonk CI jobs aws .yml global jobs to validation script projects global jobs repo