SLIDE 1
CUSTOMER
CUSTOMER
WHO? Sten Spans Schuberg Philis @sspans (github, etc) CUSTOMER - - PowerPoint PPT Presentation
WHO? Sten Spans Schuberg Philis @sspans (github, etc) CUSTOMER - - PowerPoint PPT Presentation
CUSTOMER WHO? Sten Spans Schuberg Philis @sspans (github, etc) CUSTOMER TOPIC Going from 100 to 10000 systems Orchestrating a Zone Not Google-scale CUSTOMER WHY? New Zone Rethink principles Automate Comments on Centos7/KVM Conceptual
SLIDE 2
SLIDE 3
CUSTOMER
WHY? New Zone Rethink principles Automate Comments on Centos7/KVM Conceptual or Technical?
SLIDE 4
CUSTOMER
WHAT?
SLIDE 5
CUSTOMER
SUDO MAKE CLOUD Networking Hypervisors Storage Orchestration
SLIDE 6
CUSTOMER
TOYS
Source: https:/ /www.flickr.com/photos/rfc1036/406675831/
SLIDE 7
CUSTOMER
STAFF
SLIDE 8
CUSTOMER
GOAL
SLIDE 9
CUSTOMER
GOAL
SLIDE 10
CUSTOMER
CLOUDY
https:/ /www.flickr.com/photos/versageek/493800514
SLIDE 11
CUSTOMER
MISTAKES Artisinal / Pets Network not Scalable / Redundant Stretching Failure-domains Other technical downsides Lack of Automation
SLIDE 12
CUSTOMER
WHAT IS ARTISINAL? People tracking MAC addresses Tweaking settings for each system Multiple sources of truth Validation / Acceptance test Naming - individual servers
SLIDE 13
CUSTOMER
NAMING? Impacts automation Impacts labeling Impacts replacements Go for location-based identities!
SLIDE 14
CUSTOMER
NETWORKING? Large layer2 domains Sharing networks between zones Manual configuration Not redundant (enough)? Or more failures due to redundancy?
SLIDE 15
CUSTOMER
FAILURE DOMAINS Do you really want twin-datacenter? Clustering is complicated… Way more complicated failures… Have you actually tested failures?
SLIDE 16
CUSTOMER
GOAL Manage zone as one unit Capture design / logic in config-management Versioned Iterations Think about naming Think about how you identify hosts Simplify…
SLIDE 17
CUSTOMER
GOAL Stop managing individual servers (cattle) Stop being Artisanal Start scaling Start Orchestrating Think Terraform/CloudFormation/Heat
SLIDE 18
CUSTOMER
BUILDING BLOCKS Isolated Networking Isolated Pods Worry-free Storage Optional: Dedicated SDN Clusters Fully orchestrated zones
SLIDE 19
CUSTOMER
BOOTSTRAP NETWORK CORE Core Switches LoM switch Hypervisors SDN?
SLIDE 20
CUSTOMER
CORE SWITCHES Linux based Bootstrap via DHCP/HTTP
Chef/Ansible/Puppet supported!
Capture design in cookbooks/playbooks
Can run additional services
SLIDE 21
CUSTOMER
SDN
Cluster per (availability) Zone
Failure Domain Features vs. Lock-in Complicated? Expensive? Accept tunnels between zones Customers will accept trade-offs!
SLIDE 22
CUSTOMER
BOOTSTRAP A POD TOR Switch Pair LoM switch Hypervisors Storage
SLIDE 23
CUSTOMER
TOR SWITCHES Linux Based Bootstrap via DHCP/HTTP
Chef/Ansible/Puppet supported!
Capture design in cookbooks/playbooks
Can run DHCP/DNS per Pod Move pod services into the Pod
SLIDE 24
CUSTOMER
LOM SWITCHES
Can bootstrap via ToR switch
Config via ToR Manage iLO’s via DHCP Hooks Would love a linux box here too
SLIDE 25
CUSTOMER
HYPERVISORS Linux Based
Automated Firmware Updates
Bootstrap via DHCP/HTTP HTTP Bootstrap via Chef TFTP Proxy on ToR Location based DHCP (Option 82)
SLIDE 26
CUSTOMER
HYPERVISOR HARDWARE Machines are extremely scalable Calculate cost per VM Waiting for 25G Ethernet Has anybody solved EFI PXE? Please?
SLIDE 27
CUSTOMER
PROVISIONING Bootstrap via DHCP/HTTP
Nekopan - Golang webserver
Interfaces with Chef (or ansible/puppet)
SLIDE 28
CUSTOMER
STORAGE Stable NFS – For now… API Driven No fancy replication / clustering
SLIDE 29
CUSTOMER
DONE? Lets add all of this to cloudstack…
SLIDE 30
CUSTOMER
CLOUDSTACK SDN providers need work cloudstack-setup-agent is … horrible Routervm/SystemVM Small networking issues And I bet there is more…
SLIDE 31
CUSTOMER
THE HORROR:
SLIDE 32
CUSTOMER
WHAT IS GOING ON? All Ubuntu is the same… Fedora == Redhat 6 Centos == Redhat 5 Or you may have Redhat 7
Really? WTF?
SLIDE 33
CUSTOMER
RESULTS ON CENTOS 7 Selinux is disabled (revert broken) Firewall changes don’t work for firewalld Cgroup changes are not that cool really
Workarounds for old bugs results in breakage on newer systems
So I reinstalled the box
SLIDE 34
CUSTOMER
CENTOS 7 STATUS Selinux seems to work Labeled NFS is still bleeding edge No need to mess with cgroups Firewalld is pretty nice really Cloudstack should perhaps audit the config But please don’t change it…
SLIDE 35
CUSTOMER