Secure Kubernetes Container Workloads with Production-Grade - PowerPoint PPT Presentation

Secure Kubernetes Container Workloads with Production-Grade Networking Cynthia Thomas Irena Berezovsky Tim Hockin

CIA IT operations have top secret apps for their agents, most of which require isolation Antoni is in Ops and wants to help CIA - embrace DevOps Berta is a Dev eager to design - efficiently and deliver excellent apps Antoni Berta

The world before Neutron: can I plug in your cable? 1. New project defined: Developer needs an environment 2. Dev asks SysAdmin for some resources 3. SysAdmin installs Server OS and asks Network people for a VLAN (ewww!) 4. Network people ask Security team to open a port in a firewall rule ∞. Someone plugs into wrong port or wrong requirements: start over!

Antoni & Berta were doing it the hard way The CIA IT takes weeks, even months to deliver isolated resources for the various projects at CIA. Servers and VLANs and firewalls, oh my!

Neutron for higher layer network services ● OpenStack core project since Folsom ● Tenant and Admin solution agnostic API ● Pluggable Framework ● Provides extensible API to build rich topologies (vendor extensions) ● Advanced Services Support, i.e. LBaaS, VPNaaS, FWaaS

OpenStack Networking on the fly at CIA ● OpenStack has reduced the time to deliver compute resources Operator (networking, security, etc) versus App ● Security policies allows the Developer CIA to keep business units separate ● Each department admin can manage its own resources

Can we do better? ● Spawning a VM is slow expensive App App App ● There is a lot of management overhead ● VMs are not portable: run on a specific Libs Libs Libs Hypervisor Guest Guest Guest OS OS OS If only there was a way to virtualize an OS to enable multiple workloads to run on a single Hypervisor OS... OS Hardware

Along came Docker Containers are an alternative to VMs ● Bundle your app & deps, but not the OS container container container Faster and lower overhead than VMs App App App ● O(milliseconds) to spawn Libs Libs Libs Developer-focused ● Enables fast iteration, less non-app concerns OS Ridiculously simple UX Hardware It’s the technology of the decade!

CIA developers demand containers Launch in milliseconds! Dev-Prod parity, on my laptop! MUST HAVE! But it is very chaotic -- they need need help managing it all...

Kubernetes changed everything The Kubernetes API is app-centric ● De-emphasizes infrastructure and operations concerns ● Those can exist, but are not the primary focus ● Integrate with existing infrastructure and ops, but don’t replace it Networking is infrastructure, security is ops We still need to address the concerns of ops!

Kubernetes network model Assumes a single, shared network space ● No noun for Network (yet?) Network plugins decide what technology ● veth, VXLAN, OVS, etc. All connectivity is enabled by default Implicitly single tenant ● Also reflected in services like DNS Compare to Docker model: ● Noun for Network ● App-centric networks

Tension between dev and ops “No, no, no! I can’t have my apps be reachable by Carlos’s team!”

Namespaces: Kubernetes Scopes for named objects within a cluster ● Pods, Services, etc. are all namespaced Logical grouping of related things ● Could be 1 user ● ...or 1 app ● ...or 1 tier of an app No relationship to nodes or networks ● All Namespaces exist on all nodes ● Network is not segmented Seems like an obvious hook for networking

Kubernetes NetworkPolicy API to lock down the network ● Describe the graph of your app ● Specify which connectivity to allow Applies per Namespace ● Exists alongside the apps it decribes ● Default-deny plus explicit allow rules Network infrastructure can enforce it ● Many vendors have implementations Does not cover egress (yet?)

Kubernetes NetworkPolicy apiVersion: extensions/v1beta1 apiVersion: extensions/v1beta1 kind: NetworkPolicy kind: NetworkPolicy metadata: metadata: name: front-to-mid name: mid-to-db namespace: cia-spy-app namespace: cia-spy-app spec: spec: podSelector: podSelector: matchLabels: matchLabels: role: middleware role: db ingress: ingress: - ports: - ports: - protocol: TCP - protocol: TCP port: 6379 port: 3306 from: from: - podSelector: - podSelector: matchLabels: matchLabels: role: frontend role: middleware

Kubernetes NetworkPolicy apiVersion: extensions/v1beta1 apiVersion: extensions/v1beta1 kind: NetworkPolicy kind: NetworkPolicy metadata: metadata: name: front-to-mid name: mid-to-db namespace: cia-spy-app namespace: cia-spy-app spec: spec: podSelector: podSelector: matchLabels: matchLabels: role: middleware role: db ingress: ingress: - ports: - ports: - protocol: TCP - protocol: TCP port: 6379 port: 3306 from: from: - podSelector: - podSelector: matchLabels: matchLabels: role: frontend role: middleware

Kubernetes NetworkPolicy apiVersion: extensions/v1beta1 apiVersion: extensions/v1beta1 kind: NetworkPolicy kind: NetworkPolicy metadata: metadata: name: front-to-mid name: mid-to-db namespace: cia-spy-app namespace: cia-spy-app spec: spec: podSelector: podSelector: matchLabels: matchLabels: role: middleware role: db ingress: ingress: - ports: - ports: - protocol: TCP - protocol: TCP port: 6379 port: 3306 from: from: - podSelector: - podSelector: matchLabels: matchLabels: role: frontend role: middleware

Kubernetes NetworkPolicy apiVersion: extensions/v1beta1 apiVersion: extensions/v1beta1 kind: NetworkPolicy kind: NetworkPolicy metadata: metadata: name: front-to-mid name: mid-to-db namespace: cia-spy-app namespace: cia-spy-app spec: spec: podSelector: podSelector: matchLabels: matchLabels: role: middleware role: db ingress: ingress: - ports: - ports: - protocol: TCP - protocol: TCP port: 6379 port: 3306 from: from: - podSelector: - podSelector: matchLabels: matchLabels: role: frontend role: middleware

Where Neutron is ahead of k8s Neutron Kubernetes Multi Tenant environment Single Tenant Rich network topologies with overlapping IPs Flat, shared network with IP per pod Security Groups, Port Security (ARP Spoofing) Network Policy (ingress only) Port Quality of Service - Admin and Tenant facing API Primarily application-centric API

Containers Challenges ● A lot of new products/companies have emerged in the container orchestration and integration ecosystem ● With multi-host, multi orchestration environment, networking becomes critical ● Run containers in VMs for better isolation and security ● Multi-tenancy, host or cluster per tenant ● VMs and Containers share the same network

What is Kuryr? ● Neutron as a production-ready networking abstraction that containers need ● Kuryr translates container orchestration events into Neutron entities, performs API calls and manages the response to the orchestrator

Kuryr as a translator between k8s and Neutron ● Map container networking abstractions to the Neutron API Kubernetes Neutron Namespace Network, Subnet ● Allow Container, BM and VM networking under the same API Pod Port ● Implements all the common code for Service Load Balancer Neutron vendors, allowing them to External IP Floating IP provide advanced container Network Policy Security Groups networking by just having a binding script

Example: CIA Security Antoni can satisfy CIA security requirements with Kubernetes & Kuryr: ● Kubernetes made it easy for the app devs to express the application in terms of required deployment ● With NetworkPolicy devs can specify the intended application connectivity ● With Kuryr mapping of Kubernetes requests to Neutron constructs, and NetworkPolicy realization by Neutron security groups, true isolation and security is achieved via the Kubernetes API

Kubernetes + Kuryr + MidoNet: Scalable Neutron Neutron plugin scaling with ease, and flexible API for fine-grain security policies ● Event-based design (receives events from k8s-api) ● Compatible with Kubernetes >=1.2 ● API watcher + CNI driver ● Asynchronous event-loop based on asyncio python 3.4 library ● No kube-proxy

How is MidoNet a Scalable Neutron Solution?

Example: CIA MidoNet deployment Antoni uses MidoNet for his OpenStack Neutron plugin for production-grade networking ● He knows he can scale compute for the CIA with confidence with an HA solution ● MidoNet Manager provides Antoni with every single network flow and each security policy applied

Example: CIA MidoNet deployment with kuryr-k8s ● Antoni is trying the kuryr-k8s tech preview solution with MidoNet: https://docs.midonet.org/ ● Today, he can launch a script to automatically deploy the k8s-master and k8s-worker to try MidoNet with k8s ● Antoni wants native API calls for pods and VMs while using the same operator tools

MidoNet-enhanced Security Neutron Security Groups ● white-list of allowed traffic ● port-level firewall MidoNet implements SG+: ● low-level constructs called chains and rules ● richer feature set for matching/filtering and actions