Orchestrating an OpenStack DevOps Cloud for R&D to Achieve - PowerPoint PPT Presentation

Orchestrating an OpenStack DevOps Cloud for R&D to Achieve Continuous Delivery Ting Zou & Tanay Nagjee – November 7, 2013

Introductions Ting Zou Director, Cloud Computing R&D Data Center Huawei USA Tanay Nagjee Solutions Engineer @ Electric Cloud Former ElectricCommander Engineer 2

This is Huawei A privately-owned Global Company An Innovative Industry Contributor An ICT Industry Leader

A Privately-owned Global Company  A global company providing information and communications technology (ICT) solutions.  P roducts and solutions have been deployed in 140+ countries, serving 1/3 of the world’s population.  A privately-owned company founded in 1987, Shenzhen

Sustainable Growth 2012 Revenue amounted to $35.4 billion, a YoY increase of 8%; net profit reached $2.47 billion  2012 USD 4.8 billion R&D investment  1H 2013, Revenue amounted to USD $18.5B, a YoY increase of 10.8%; Expected net profit margin of  7-8% in 2013. By Region (2012) By BG (2012) Americas Consumer China 5.1 bn 7.8 bn 11.8 bn 15% 22% 33% Enterprise 5% 1.9 bn 35% 73% 17% EMEA Asia Pacific Carrier Network 12.5bn 6.0 bn 25.7 bn Currency: USD Currency: USD 5

Continuous Innovation Investment 41,948 patents in China, 12,453 international   2012 R&D investment: USD 4.8 PCTs, 14,494 patents in outside of China. billion, 13% of 2012 revenues US$300 million in patent royalties each year.   70,000 R&D employees 150+ standards organizations   16 R&D centers worldwide 30,000+ standards proposals   28 joint innovation centers 4.8Bn Accumulated R&D investments: USD 19 3.8Bn 2.7Bn billion 2Bn 1.5Bn 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Problem overview

What are we solving? Large scale R&D environment with complicated tools requires multi- thousands of CPU cores available on demand. 120 300 10 9 Actual Time Engineering Expectation Current Build Time Expected Build Time 5 Million LOC Build 10 Million LOC code coverage testing report （ unit: minute ） （ unit ： minute ） 8

What are we solving? Environment/tools provisioning is very time-consuming; lab asset utilization is low. 480 10 500 400 300 2 200 20 100 0 Current Provision time Expected Provision Current expected environment provision Time Compile/Build environment Provision Software Testing environment provision （ unit ： minute ） （ unit ： hour ） 9

What are we solving? R&D data grows rapidly; requires the flexibility of dynamically expanding PB storage capability. 2 500 2500 2000 1300 1000G 1500 500G 1000 500 0 2011 年 2012 年 daily execution data generated by daily build Millions LoC build static analysis/ build Data Generated by the Tools in Auto Testing/CI coverage to generate PB data annually 10

What are we solving? Integration between development and operation; DevOps enabled continuous delivery required.  R & D requires multi-layer and multi- platform infrastructure  Virtualization and cloud computing DevOps  Allow developers more control over the converged infrastructure production environment  Automatic provision and  Concise definition of R&D processes, configuration management technology automate as much as possible  Bridging the gap between development Cloud and operations  Globally distributed R&D teams 11

Solution overview

Virtualization/Cloud sounds like a good solution, but...  To virtualize 3k 2-CPUs servers, commercial SW license + 1 year support is a big investment: Servers CPUs Standard Enterprise Enterprise Plus 3000 6000 8 million US$ 20 million US$ 25 million US$  Shall we leverage low-cost(free) open source cloud OS to reduce R&D TCO? Source: http://www.qyjohn.net

Choose OpenStack as the Provider of IAAS Cobbler Chef Puppet etc In-House Dev Module iLCM (Lab Configuration Manager) Swift Controller Controller Controller Swift Swift-Proxy Quantum(A) Quantum(A) Quantum(A) Swift-Proxy Controller Node Controller Node #1 Controller Node #2 Storage Node #3 Private Switch (Network) vSwitch External Switch (Network)  8 Huawei RH2285(dual 6-cores CPUs, 96GB MEM, 8TB storage) servers used in PoC  Openstack is provisioned and managed by Huawei iLCM(intelligent Lab Configuration Manager) with backend integration with cobbler/chef/puppet open source tools  Leverage the OpenStack open source community as much as we can (Devstack, Mirantis, RPC, Dell crossbar etc) plus in house developed script modules, will contribute back to the community once it is ready.

Single Portal for Cloud Admin and Users  Cloud Admin Portal for users with different level of privileges, integrated with LDAP at the backend  Cloud resource usage monitoring (Roadmap)  Cloud VM provision and software configuration management  Cloud Network, Storage and Security configuration management (Roadmap)

Cloud is free now, what else can be free? Proje oject ct Mgmt Redmine dmine Trac ac Bugzilla gzilla Provision ovision Monitor nitor JIRA RA Chef ef Nagios gios Sourc GIT GIT Subversio bversion Mercurial rcurial SCM SCM urceFo Github Puppe ppet thub eForge Collabora llaboratio tio CFEngine Engine OpenGrok enGrok Gerrit rrit Reviewboa viewboard rd ge n Design sign Developme velopment nt Testing sting Others hers Hadoop doop Code de analysis lysis Jenkins Cpplint plint CppCheck pCheck Requi quirem rement nt nkins Hive ve 、 Analysys alysys Func nc Testing ting Pef Testing Pef ting HBase ase PcLint Lint valgrind lgrind CI CI memcached mcached IOMeter Meter Selenium lenium Unit it Testing ting Design sign RabbitMQ bbitMQ Travis Jmeter eter gUnit nit,cpp ,cppUnit Unit modeling deling ActiveMQ tiveMQ avis-CI Page ge Speed ed Automatio tomation n Framework mework CI Mobit bitest est gMock ock,JMo ,JMock ck Cloud oud Platform tform OpenStack enStack CloudStac oudStack

HUDOS (Huawei Unified DevOps System) Technologies

Single Portal for R&D engineering  SaaS (instant creation of an environment with complete portfolio of common R&D tools)  Lab topology creation with compute, network, storage etc (Roadmap)  Project R&D data statistics monitoring  Single dashboard portal for engineering to access all the needed resource/tools in R&D process

IaaS and PaaS enable DevOps-Engineering link  IaaS integrated with engineering desktop dashboard  R&D tools PaaS integrated with engineering desktop dashboard

OpenStack integration

Electric Cloud’s Software Delivery System Continuous Delivery Manager Release Trains | Feature Boards | Pipelines | Gates | Dashboards Build Deploy Test Build Test Deploy Apps Automation Automation AutomationC CI CT D Test Build Infrastructure Acceleration Acceleration Provisioning Software Delivery Platform Platform Automation & Acceleration Services Workflow | Resource management I Tools integration | API | Security | Reporting

Before and After Electric Cloud Before Electric After Business Issue Cloud Electric Cloud Impact Develop to Deploy 90 Days 10 Minutes 99.93% 10+ errors/cycle ~0 errors/cycle 99+% Build to Release/Deploy Audit application changes Days Minutes 90+% (who, what, how, why, when) Time to troubleshoot 20 Days Minutes 99+% problems

OpenStack integration 23

OpenStack integration 24

OpenStack integration 25

OpenStack integration 26

OpenStack integration 27

Solution walk-through

Development Scenario HUDOS Reviewer Mike Developer Joe Eclipse Subversion Redmine Modify code Check out sources Mark issue as launch preflight overlay deltas “build & unit test” Eclipse Jenkins Build + Build + test failed; test Launch build + test success? notify developer w/ preflight source Time Savings Redmine Review Board Review Board Mark issue as Create review Review modified “code review” request code Redmine Eclipse Eclipse Review Code rejected; Success Mark issue as success? auto-commit code notify developer “resolved” 29

Redmine: select a bug to fix 30

Eclipse: modify code & launch a preflight 31

Commander: SVN preflight & Jenkins launch 32

Redmine: auto-updated issue status 33

Jenkins: auto-launched build 34

Commander: build succeeds; wait for review 35

Review Board: auto-submitted review request 36

Review Board: accept the changes 37

Commander: the workflow succeeds 38

Eclipse: auto-commit the changes 39

Test Scenario HUDOS Test Engineer Jill Commander Redmine OpenStack Pick issue to verify Mark issue as Provision specified and tests cases “verifying” virtual machines Commander Notification Testing Tests failed; VMs Launch success? automated tests ready to inspect Time Savings OpenStack Redmine Teardown virtual Mark issue as machines “closed” Notification Tests passed; issue closed 40

OpenStack: no dynamically deployed VMs 41

Commander: launch test workflow 42

Commander: provision VMs 43