DIABLO VS HAVANA HOW OPENSTACK HAS MATURED Joe Gordon ABOUT ME - - PowerPoint PPT Presentation
DIABLO VS HAVANA HOW OPENSTACK HAS MATURED Joe Gordon ABOUT ME Developer @ Full time upstream OpenStack developer nova-core ATC since Essex Interested in making OpenStack scale and work contact information jog0 on freenode github.com/jogo
Joe Gordon
Developer @ Full time upstream OpenStack developer nova-core ATC since Essex Interested in making OpenStack scale and work contact information jog0 on freenode github.com/jogo launchpad.net/~jogo
A BRIEF HISTORY OF THE INTEGRATED RELEASE
Swift Nova Nova-volume Nova-network Identity Glance Keystone*
Swift Nova Nova-volume Nova-network Identity Glance Keystone Horizon
Swift Nova Nova-volume Nova-network Glance Keystone Horizon Quantum (Neutron) Cinder
Swift Nova Nova-network Glance Keystone Horizon Quantum (Neutron) Cinder
Total number of OpenStack services: 32
Swift (4) Nova (6) Nova-network Glance (2) Keystone (1) Horizon (1) Neutron (6) Cinder (4) Ceilometer (4) Heat (4)
Swift Nova Nova-network Glance Keystone Horizon Neutron (Quantum) Cinder Ceilometer Heat
Swift Nova Nova-network Glance Keystone Cinder
NEW SERVICES
Heat Ceilometer More in pipeline Icehouse Trove Incubation Ironic Marconi Savanna
Expand existing APIs Split out projects Extensions
| Hosts | Host administration | Volumes | Volumes support | Quotas | Quotas management support | SimpleTenantUsage | Simple tenant usage extension | FlavorExtraData | Provide additional data for flavors | SecurityGroups | Security group support | FlavorExtraSpecs | Instance type (flavor) extra specs | Createserverext | Extended support to the Create Server v1.1 API | Multinic | Multiple network support | VSAs | Virtual Storage Arrays support | VolumeTypes | Volume types support | Keypairs | Keypair Support | Rescue | Instance rescue mode | VirtualInterfaces | Virtual interface support | Floating_ips | Floating IPs support
AdminActions Agents Aggregates AssistedVolumeSnapshots AttachInterfaces AvailabilityZone BareMetalExtStatus BareMetalNodes BlockDeviceMappingV2Boot CellCapacities Cells Certificates Cloudpipe CloudpipeUpdate ConfigDrive ConsoleOutput Consoles Coverage DeferredDelete DiskConfig Evacuate ExtendedAvailabilityZone ExtendedFloatingIps ExtendedIps ExtendedIpsMac ExtendedQuotas ExtendedServerAttributes ExtendedServices ExtendedStatus ExtendedVIFNet ExtendedVolumes FixedIPs FlavorAccess FlavorDisabled FlavorManage FlavorRxtx FlavorSwap FloatingIpDns FloatingIpPools FloatingIps FloatingIpsBulk Fping HideServerAddresses Hypervisors ImageSize InstanceActions Migrations MultipleCreate NetworkAssociationSupport Networks OSInstanceUsageAuditLog OSTenantNetworks QuotaClasses SchedulerHints SecurityGroupDefaultRules ServerDiagnostics ServerPassword ServerStartStop ServerUsage Services Shelve UsedLimits UsedLimitsForAdmin UserData UserQuotas VolumeAttachmentUpdate
Drivers Testing and bugs Performance and scalability Forward facing internal changes Process
Nova: baremetal, docker, hyperv, powervm Cinder: coraid, emc, glusterfs, gpfs, nexenta, rbd, sheepdog, soldfire, ...
Unit tests Integration tests (Tempest) Lots of fixed bugs
PERCENT UNIT TESTS BY LOC
Project Diablo Havana Nova 43% 56% Glance 62% 62% Swift 65% 71% Keystone 41% 45%
Orignally named 'openstack-integration-tests' Just 107 tests* extensions flavors floating ips image metadata images keypairs servers volumes 2,655 lines of python code First release of Tempest
1275* tests 1150 api tests compute, image, identity, network, object_storage,
102 test files 92 cli tests 11 scenario tests 18 thirdparty (boto) tests 4 other Parallel 20-30 minutes to run in gate 24,837 lines of python code
785 compute 216 identity 24 image 17 network 50 object_storage 54 volume
Same APIs as Diablo plus new ones Diablo has been 'end of life'ed Many tests cover new features. Ignore those Tempest not designed to be version independent Got 622 tests to run and 154 (27%) to pass.* 110 of which were in compute
Diablo has poor parameter validation Invalid key names Min/max count for multiple creates Changed return codes for negative cases Return empty list and 200 instead of 400 when invalid filter value in nova list servers We shouldn't have done this Poor pagination in Diablo JSON only tests in Diablo, Havana tests XML too Many false negatives in results due to slight changes in now APIs and metadata are used
Project Diablo Essex Folsom Grizzly Havana Nova 14 22 30 31 33 Glance 5 15 30 13 8 Keystone 31 11 9 8 Cinder 10 3 12 Neutron 27 14 24 Categories Non-gated features Dependencies Race conditions Performance
Nova-scheduler Active-active No RPC fanout DB Indices Smarter queries Fewer queries service group Heartbeat to DB every 10 seconds 1k DB calls a second if have 10K nodes. Adjustable interval Alternatives memcache and zookeeper
More efficient image work flow Glance, Nova, Cinder Move large files around less PKI Tokens Fewer round trips to keystone
CASE STUDY FROM HP PUBLIC CLOUD
Diablo 40,836 LOC changed to 239 files in nova 2,265 LOC to manage DB +100s of hours of DB tuning to make it work with more then just a few nodes Havana DB just worked 1,167 LOC changed in 35 files Most of that is in patches in review today Thanks to Thom Leggett for these statistics
Split out projects Oslo Road to live and continuous upgrades Iterate on APIs
Never break trunk Master branch is always green Developers are never blocked Transparency Automate everything Egalitarian Be strict. Reduce burden on reviewers
Automated Gate 87 Contributors in September 2011 (release of Diablo) No Gate No Git or Gerrit Bzr and Launchpad
346 Contributors in October 2013 (release of Havana) Scale development process Integrated Gate 9 Projects 30+ separate services 1 Merge pipeline New tools for an asynchronous world Tests not pass fail, rather classified with a race percentage Fix that race condition we hit 0.5% of the time Debugging is hard elastic-recheck
WHAT CHALLENGES ARE WE FACING TODAY?
More testing Live rolling upgrades Limiting factor to the whole system is reviewers (and not just core) Improving quality while growing the project as a whole Scalability
Powered by reveal.js