OpenStack Operations Quick Ramp-up and Survival Guide Joshua Guan, Operations Engineer, IBM Bluemix Private Cloud, @joshuakwan Fan He, Architect, IBM Bluemix Private Cloud, @fancyhe
Joshua Guan, Operations Lead Fan He, Cloud Architect IBM Bluemix Private Cloud IBM Bluemix Private Cloud
A Little Bit Background … • Bluemix Private Cloud is IBM’s private cloud as service based on OpenStack • Bluemix Private Cloud landed in China to support IBM’s Cloud business there. • We were building an OpenStack Operations Team from scratch
Agenda • Define an OpenStack Operations Team • Operating Model • Processes • Tooling • Teaming • Tooling Integration • Cliché: OpenStack upgrade, HA, Live Migration
Operating OpenStack is like … You thought you would work like this And, Welcome to the real world
Define an OpenStack Operations Team Operating Model Processes Tooling Teaming • How the cloud services are • Operation Tiers • Monitoring • Roles and Responsibilities offered • Escalation Levels • Collaboration • Shift Model • What is the SLA • Incident Management • Cloud Management • Collaboration with Business • Change Management • Knowledge Base Partners, Data Centers and • Shifts • Security backend teams, etc. • Onboard & Offboard • Customer Support • …
Operating Model Customers use consume complies OpenStack Service Support Entry Points Service Level Agreement Offering route operates Business Partner collaborate/escalate OpenStack Operations Data Center collaborate/escalate Development Team
Processes • Roles Operation Security • Responsibilities Tiers Tier Role Responsibilities Escalation Shifts 1 Support First line of defense Flows 2 Operations Deploy, upgrade, admin 3 OpenStack Engineering Build the product Change Incident Management Management 3 Network Engineering Undercloud networks
Processes • How tickets/alerts/incidents Operation Security go between different tiers Tiers customer Escalation Shifts Flows Tier 1 Change Incident Tier 2 Management Management Tier 3 Tier 3 Tier 3
Processes Definition Example Operation Priority Level P0, P1, P2 Security Tiers Incident OpenStack node failure, Data center Definition network interruption Management RFO, Outage Track Activities Escalation Shifts Flows Response time Immediate, 15min, 1hr Update interval Every 30min Change Incident Communicatio Customer ticket, email, statuspage.io Management Management n method Escalcation to 1hr leadership
Processes • Different types of changes • How the change will be rolled Operation Security out Tiers • When the change will be rolled out Escalation Shifts Flows • Review and approval • Customer communication Change Incident Management Management
Processes Time Operation Security Tiers at-work at-work at-work at-work at-work at-work Escalation Shifts Flows on-call primary on-call primary on-call primary Change Incident Management Management on-call secondary on-call secondary on-call secondary
Processes • Security Compliance Activities • Health Check Operation Security • Patch Reporting Tiers • Vulnerability Scanning • Continuous Business Need Escalation Shifts Flows Change Incident Management Management
Tooling • Monitoring Monitoring • Alerting • Log Aggregation Customer Collaboration Support • Dashboard OpenStack Operations Cloud Security Management Knowledge Base
Tooling • Chat Monitoring • File Sharing • Project Kanban Customer Collaboration Support • Shift Management OpenStack Operations Cloud Security Management Knowledge Base
Tooling • CMDB Monitoring • Asset Management • Change Management Customer Collaboration Support • Incident Management OpenStack Operations Cloud Security Management Knowledge Base
Tooling • Internal Wiki/Runbooks Monitoring • Product Documents for Customers Customer Collaboration Support OpenStack Operations Cloud Security Management Knowledge Base
Tooling • Access Management Monitoring • Security Compliance Management • Health Checking Customer Collaboration Support • Patching Reporting • Vulnerability Scanning OpenStack Operations Cloud Security Management Knowledge Base
Tooling • Ticketing System Monitoring • Customer Chat • Customer Satisfaction Customer Collaboration Support • Cloud Level Maintenance Communication OpenStack • Site Level Maintenance Operations Communication Cloud Security Management Knowledge Base
Teaming Service Level Service Shift Agreement Availability Model
Teaming Time Operators on shift • 24x7 Availability • Spread the pain Triage • Eliminate interruptions as at-work at-work at-work at-work at-work at-work possible at-work at-work at-work at-work SME On-call 1 SME On-call 2 SME On-call 3 on-call primary on-call primary on-call primary primary primary primary on-call secondary on-call secondary on-call secondary secondary secondary secondary
Tooling Integration • A lot of screens to watch • A lot of systems to work on • A lot of interruptions • Use your tools to “kill” them
Tooling Integration As a good start: Kill ”context switch” – work on a single platform
Tooling Integration As a good start: Kill ”context switch” – work on a single platform
Tooling Integration What’s next: Kill ”all interruptions” – workflow automation across platforms
Cliché – Where BOOOOOM Happens • Implementations & Operations: Change management • The Practices of Upgrade • The Story of HA • The Myth of Live Migration
Change management • “Infrastructure as Code” • Incoming change requests • Customer initiated requirements • Internal enhancements roll out • Compliance • Change planning for Consistency • Priorities • Dependencies
OpenStack Upgrade • Prerequisites: deployment automation • Consistency – cloud configurations in CMDB • Idempotency – code to run OpenStack upgrade • Upgrade process design • Upgrade orchestration • Repeatable success & minimum disruption Reference: Upgrading OpenStack: A Best Practices Guide
Let’s talk about High Availability…. • Architecture decisions for HA • Eliminate SPOF; Non-disruptive upgrade; Load Balancing; … • Inherent availability = MTTF / (MTTF + MTTR) • HA’s “dark side” for cloud operations • Recovery with HA resetting • Complexity’s impact on recovery time • Mitigation plan • Built-in monitoring for HA mechanism • Recovery automation
Live Migration? • Does ”nova live-migrate” work? • Manage customer expectations • Abuse prevention • Limited appropriate scenarios • Automation with caution • Integration with pre & post- verification routine Reference: Live Migration is a Perk, not a Panacea @kiwik http://kiwik.github.io/openstack/2015/05/23/Nova-Live-Migration-Workflow/
The Open Cloud: Delivering Solutions with Choice October 26 th CCIB Room 116 Kickoff with Todd Moore 11:25 IBM Vice President, Open Technology En Enterprise OpenStack for Beginners 11:30 Perspectives Pe Shamail Tahir • Tyler Britten The Open Cloud: A Platform of Possibilities 12:15 Jesse Proudman • Azmir Mohamed Don’t Just Take Our Word for It: Use Cases from Materna & AT&T 2:15 4:30 Join Brad Topol Armin von Dolenga (Materna) • Jacob Caspi (AT&T) and the Interop Challenge Vendors for refreshments Part 1 - Designing Effective Microservices 3:05 Manuel Silveyra Part 2 - Deploying Infrastructure Foundations 3:55 Microservices on Shaun Murikami • Andrew Bodine the Open Cloud Part 3 - Delivering Application Microservices 5:05 Daniel Krook Part 4 – Directing Deployments with DevOps 5:55 Megan Kostick • Michael Brewer • Manuel Silveyra
Thank You
Recommend
More recommend
