Lessons Learnt from Running a Container Native Cloud Xu Wang - - PowerPoint PPT Presentation

Reinvent IaaS with Container!

Lessons Learnt from Running a Container Native Cloud

Xu Wang (@gnawux) CTO & Cofounder, Hyper.sh

Reinvent IaaS with Container!

Agenda

• Hyper.sh: a Container Native Cloud
• Under the Hood: How We Build a Container Native Cloud
• Innovation from the User Side: App Patterns in Hyper.sh
• Solve the Challenges Introduced by Containers

Reinvent IaaS with Container!

How we define Container Native Cloud

• Container is a First-Class citizen in the cloud
• Container level APIs
• Launch container without virtual machine (cluster)
• Container-Centric Resources
• Floating IP, Security Group…
• Container-Based Services
• Service (Load Balancer), Cron, Function…
• Charged by Container
• Charged based on container life cycle rather than VMs

Reinvent IaaS with Container!

Why Container Native Cloud

“All problems in computer science can be solved by another level of indirection, except of course for the problem of too many indirections.”

• ---David Wheeler

Reinvent IaaS with Container!

Why Container Native Cloud (cont.)

• Overhead:
• Resources
• Ops

https://www.openstack.org/videos/boston-2017/kubernetes-on-openstack-on-kubernetes-the-infrastructure-club-sandwich

Reinvent IaaS with Container!

Hyper.sh: a Container Native Cloud

Reinvent IaaS with Container!

Agenda

• Hyper.sh: a Container Native Cloud
• Under the Hood: How We Build a Container Native Cloud
• Innovation from the User Side: App Patterns in Hyper.sh
• Solve the Challenges Introduced by Containers

Reinvent IaaS with Container!

How we build hyper.sh

apirouter

Distributed graph engine Hyperd / runV

Neutron Cinder

Kube- API

node node node

controller scheduler

compose engine cron job engine service engine Hyper func engine High level app engines stackube (hypernetes)

• penstack/stackube

keystone kubernetes

Reinvent IaaS with Container!

Runtime: Secure à Simple

• Container Isolated by hypervisor
• Secure as VM, 
Fast as Container

Reinvent IaaS with Container!

runV: Secure as VM, Fast as Container

Reinvent IaaS with Container!

runV: Project Summary

• https://github.com/hyperhq/runV
• Version 1.0 (Sep 2017)
• Compatible with OCI runtime spec 1.0
• Full Support of Kubernetes CRI Semantics
• Supported arch: x86_64, aarch64, power, s390x
• Supported hypervisor: kvm/qemu, xen (fv), xenpv, kvmtool
• Contributors: HyperHQ, Huawei, ZJU, Intel, Alibaba, IBM
• Work with Hyperd, Containerd, & CRI-O

Reinvent IaaS with Container!

Stackube: K8S as the Scheduler of OpenStack

Reinvent IaaS with Container!

What’s in Stackube Repo?

• Keystone RBAC plugin
• Neutron CNI plugin
• With a k8s Network object controller
• Neutron based multi-tenant k8s service proxy
• k8s Cinder plugin with block device mode
• Deployment scripts and guide
• Documentations

Reinvent IaaS with Container!

Project Status

• https://github.com/openstack/stackube
• 1.0-beta released
• Done: Migrate existing code from Hypernetes to Stackube
• Done: Keystone RBAC plugin
• Done: k8s Network and tenant object controller
• Done: Neutron CNI plugin
• Done: Neutron based multi-tenant k8s service proxy
• Done: k8s Cinder plugin with block device mode
• Done: Multi-tenant k8s DNS addon
• Done: containerize all plugins above
• Done: multi-node deployment scripts

Reinvent IaaS with Container!

Agenda

• Hyper.sh: a Container Native Cloud
• Under the Hood: How We Build a Container Native Cloud
• Innovation from the User Side: App Patterns in Hyper.sh
• Solve the Challenges Introduced by Containers

Reinvent IaaS with Container!

CI on Hyper.sh

• CI integration with Hyper
• Buildbot on Hyper
• Jenkins on Hyper (Hykins)
• Drone on Hyper
• Agility
• Zero Preparing Time
• Zero Cleanup/Waste Time
• No resource pool is required

anymore

Reinvent IaaS with Container!

Parallel Containers & Timed Containers

• Spawn containers as Jobs come
• Rather than queue and schedule in a VM cluster
• Or periodically spawn containers
• Lower Management Cost
• More Responsive

Reinvent IaaS with Container!

Disaster Recovery

• Containers on Hyper.sh as a backup
• Start backup containers in seconds once server in private cluster down
• Result
• Much faster than start a VM cluster and launch Apps
• For most Apps, no hot standby any more

Reinvent IaaS with Container!

Summary: the Value for Customers

• Eliminate the Visible Resource Pool
• Shorter Preparation Time, More friendly for Parallel Jobs
• Spawn on Demand, Minimize Standby Resources
• Independent Lifecycle for Each Container, Simplify

Management

Reinvent IaaS with Container!

Agenda

• Hyper.sh: a Container Native Cloud
• Under the Hood: How We Build a Container Native Cloud
• Innovation from the User Side: App Patterns in Hyper.sh
• Solve the Challenges Introduced by Containers

Reinvent IaaS with Container!

Challenges from Concurrent Launch

• The fact
• Much more containers than traditional VM
• Shorter living time of a container
• The challenges
• Metric the server load – Not only by static resource
• Consumption introduced by create and cleanup resources
• The solutions
• Scheduler plugin, not only based on static resource usage
• Optimize procedure, reduce the workload of message queue
• Cache and delay delete

Reinvent IaaS with Container!

Challenges from High Density

• The fact
• More VM (containers) on a single host
• More ports on a single host
• The challenges
• ovs agent overhead
• open-vswitch device lookup consumption
• The solutions
• Avoid unnecessary ports
• Reduce/Eliminate the agent consumption
• Optimize ovs port lookup
• Newer version upstream introduced many fixes

Reinvent IaaS with Container!

Thank You!