Dirty Clouds Done Dirt Cheap Matthew Treinish mtreinish@kortar.org - - PowerPoint PPT Presentation

Dirty Clouds Done Dirt Cheap

Matthew Treinish mtreinish@kortar.org mtreinish on Freenode May 11, 2017 https://github.com/mtreinish/dirty-clouds-done-dirt-cheap

Building a Cloud

1 / 35

Scope of the Project

◮ Pretend to be a sysadmin with no prior OpenStack knowledge ◮ Try to rely only on install docs and google searches ◮ $1500 USD Budget ◮ Build a basic compute cloud:

◮ Keystone ◮ Glance ◮ Nova ◮ Neutron

◮ Install Ocata from tarballs ◮ No automation or pre-existing install scripts

2 / 35

Buying Hardware

◮ Maximize core count per USD ◮ Second priority is amount of RAM per core ◮ Machines don’t need to be fast (that costs money!)

3 / 35

4 / 35

The Servers

Model PowerEdge R610 Processor 2x Intel Xeon E5540 Memory Installed 32GB Total Memory; 8 x 4 GB DDR3 Hard Drives 2x 146GB 10K SAS Hard Drive RAID Controller Dell PowerEdge R610 Perc 6i Ethernet 2x Dual Port Embedded Broadcom NetXtreme ll 5709c Return Policy/Warranty 60 days Money Back Or Exchange

$215 Each!!

5 / 35

6 / 35

Quirks with the servers

◮ Super stripped down:

◮ No management interface ◮ No redundant power supply

◮ 4x8GB of RAM not 8x4GB ◮ Memory installed in wrong slots ◮ Dead RAID controller battery ◮ Came with 15k RPM hard drives not 10k RPM ◮ Came pre-installed with Windows Server 2012 (and default password Apple123)

7 / 35

LackRack

https://wiki.eth0.nl/index.php/LackRack

◮ Use a LACK side table from Ikea ◮ 19 inch width between legs ◮ Can fit 8U ◮ Lots of color choices ◮ $9.99 USD

8 / 35

9 / 35

10 / 35

Installing OpenStack

11 / 35

Installing OpenStack Services

• 1. Download service tarball
• 2. Create service users
• 3. Install binary requirements
• 4. Create service dirs in /etc and /var/lib
• 5. Copy etc/ from tarball into /etc/$Service
• 6. pip install the tarball
• 7. Follow install guide on project configuration and setup

12 / 35

Setting Up Keystone

◮ 2 config options ◮ Install guide doesn’t have

instructions on apache wsgi app setup

◮ Google search found:

https://docs.openstack.org/developer/keystone/apache- httpd.html

13 / 35

Python Requirements aren’t fun

ERROR k e y s t o n e Traceback ( most r e c e n t c a l l l a s t ) : ERROR k e y s t o n e F i l e "/ u s r / l o c a l / b i n / keystone −wsgi−admin" , l i n e 51 , i n <module> ERROR k e y s t o n e a p p l i c a t i o n = i n i t i a l i z e _ a d m i n _ a p p l i c a t i o n ( ) ERROR k e y s t o n e F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / k e y s t o n e / s e r v e r / wsgi . py" , l i n e 132 , i n i n i t i a l i z e _ a d m i n _ a p p l i c a t i o n ERROR k e y s t o n e c o n f i g _ f i l e s=_ g e t _ c o n f i g _ f i l e s ( ) ) . . . ERROR k e y s t o n e F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / p a s t e / d e p l o y / l o a d w s g i . py" , l i n e 640 , i n find_egg_entry_point ERROR k e y s t o n e pkg_resources . r e q u i r e ( s e l f . spec ) ERROR k e y s t o n e F i l e "/ u s r / l i b / python2 . 7 / d i s t −packages / pkg_resources /__init__ . py" , l i n e 943 , i n r e q u i r e ERROR k e y s t o n e needed = s e l f . r e s o l v e ( p a r s e _ r e q u i r e m e n t s ( r e q u i r e m e n t s ) ) ERROR k e y s t o n e F i l e "/ u s r / l i b / python2 . 7 / d i s t −packages / pkg_resources /__init__ . py" , l i n e 834 , i n r e s o l v e ERROR k e y s t o n e r a i s e V e r s i o n C o n f l i c t ( d i s t , req ) . with_context ( dependent_req ) ERROR k e y s t o n e C o n t e x t u a l V e r s i o n C o n f l i c t : ( r e q u e s t s 2 . 1 3 . 0 (/ u s r / l o c a l / l i b / python2 . 7 / d i s t − packages ) , Requirement . p a r s e ( ’ r e q u e s t s !=2.12.2 ,!=2.13.0 , >=2.10.0 ’ ) , s e t ( [ ’ o s l o . p o l i c y ’ ] ) ) 14 / 35

Setting up Glance

◮ Straightforward configuration:

◮ Auth ◮ Image Directories ◮ Database 15 / 35

Don’t forget to create glance store directory

[ req −8f f 7 f 7 d 2 −a4a5 −4558−b5a9−9ecca139675f 20 f283024ffd4bf4841a8d33bdb4f385 6 c3fc6392e0c487e85d57afe5a5ab2b7 − d e f a u l t d e f a u l t ] E r r o r i n s t o r e c o n f i g u r a t i o n . Adding images to s t o r e i s d i s a b l e d . Traceback ( most r e c e n t c a l l l a s t ) : F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / g l a n c e / a p i / v2 / image_data . py" , l i n e 116 , i n upload image . set_data ( data , s i z e ) . . . F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / g l a n c e _ s t o r e / backend . py" , l i n e 426 , i n store_add_to_backend v e r i f i e r = v e r i f i e r ) F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / g l a n c e _ s t o r e / c a p a b i l i t i e s . py" , l i n e 223 , i n

• p_checker

r a i s e

• p_exec_map [ op ](∗∗ kwargs )

StoreAddDisabled : C o n f i g u r a t i o n f o r s t o r e f a i l e d . Adding images to t h i s s t o r e i s d i s a b l e d . 16 / 35

Setting up Nova

◮ Database migrations are slower, took

about 3mins

◮ Don’t forget the placement API, no

docs on apache setup

◮ novnc is problematic from source ◮ Set force_config_drive option to

true

17 / 35

Requirements still aren’t fun

ERROR nova Traceback ( most r e c e n t c a l l l a s t ) : ERROR nova F i l e "/ u s r / l o c a l / b i n /nova−a p i " , l i n e 10 , i n <module> ERROR nova s y s . e x i t ( main ( ) ) ERROR nova F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / nova /cmd/ a p i . py" , l i n e 59 , i n main ERROR nova s e r v e r = s e r v i c e . WSGIService ( api , u s e _ s s l=should_use_ssl ) . . . ERROR nova F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / pkg_resources /__init__ . py" , l i n e 2324 , i n r e q u i r e ERROR nova i t e m s = working_set . r e s o l v e ( reqs , env , i n s t a l l e r , e x t r a s=s e l f . e x t r a s ) ERROR nova F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / pkg_resources /__init__ . py" , l i n e 859 , i n r e s o l v e ERROR nova r a i s e V e r s i o n C o n f l i c t ( d i s t , req ) . with_context ( dependent_req ) ERROR nova C o n t e x t u a l V e r s i o n C o n f l i c t : ( pbr 1 . 1 0 . 0 (/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages ) , Requirement . p a r s e ( ’ pbr >=2.0.0 ’ ) , s e t ( [ ’ o s l o . i 1 8 n ’ , ’ o s l o . l o g ’ , ’ o s l o . c o n t e x t ’ , ’ o s l o . u t i l s ’ ] ) ) 18 / 35

You need a sudoers file

ERROR nova Traceback ( most r e c e n t c a l l l a s t ) : ERROR nova F i l e "/ u s r / l o c a l / b i n /nova−a p i " , l i n e 10 , i n <module> ERROR nova s y s . e x i t ( main ( ) ) ERROR nova F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / nova /cmd/ a p i . py" , l i n e 59 , i n main ERROR nova s e r v e r = s e r v i c e . WSGIService ( api , u s e _ s s l=should_use_ssl ) ERROR nova F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / nova / s e r v i c e . py" , l i n e 309 , i n __init__ ERROR nova s e l f . manager = s e l f . _get_manager ( ) . . . ERROR nova F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / nova / u t i l s . py" , l i n e 180 , i n e x e c u t e ERROR nova r e t u r n p r o c e s s u t i l s . e x e c u t e (∗cmd , ∗∗ kwargs ) ERROR nova F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / o s l o _ c o n c u r r e n c y / p r o c e s s u t i l s . py" , l i n e 400 , i n e x e c u t e ERROR nova cmd=sanitized_cmd ) ERROR nova P r o c e s s E x e c u t i o n E r r o r : Unexpected e r r o r w h i l e r u n n i n g command . ERROR nova Command : sudo nova−rootwrap / e t c / nova / rootwrap . conf i p t a b l e s −save −c ERROR nova E x i t code : 1 ERROR nova Stdout : u ’ ’ ERROR nova S t d e r r : u ’ sudo : no t t y p r e s e n t and no a s k p a s s program s p e c i f i e d \n ’ ERROR nova 19 / 35

Don’t forget to create state directories

ERROR nova Traceback ( most r e c e n t c a l l l a s t ) : ERROR nova F i l e "/ u s r / l o c a l / b i n /nova−a p i " , l i n e 10 , i n <module> ERROR nova s y s . e x i t ( main ( ) ) ERROR nova F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / nova /cmd/ a p i . py" , l i n e 59 , i n main ERROR nova s e r v e r = s e r v i c e . WSGIService ( api , u s e _ s s l=should_use_ssl ) ERROR nova F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / nova / s e r v i c e . py" , l i n e 311 , i n __init__ ERROR nova s e l f . app = s e l f . l o a d e r . load_app ( name ) ERROR nova F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / nova / wsgi . py" , l i n e 497 , i n load_app . . . ERROR nova F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / nova / a p i / openstack /compute / c l o u d p i p e . py" , l i n e 55 , i n s e t u p ERROR nova f i l e u t i l s . e n s u r e _ t r e e (CONF. c r y p t o . keys_path ) ERROR nova F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / o s l o _ u t i l s / f i l e u t i l s . py" , l i n e 40 , i n e n s u r e _ t r e e ERROR nova

• s . makedirs ( path ,

mode ) ERROR nova F i l e "/ u s r / l i b / python2 . 7 / os . py" , l i n e 157 , i n makedirs ERROR nova mkdir ( name , mode ) ERROR nova OSError : [ Errno 13] P e r m i s s i o n d e n i e d : ’ / u s r / l o c a l / l i b / python2 . 7 / d i s t −packages / keys ’ 20 / 35

Networking Configuration

21 / 35

Setting Up Neutron

◮ Too many configuration files ◮ Blindly copying pasting from install

guide

◮ Rootwrap and sudo configuration are

not documented

◮ First time I had to look at packages

and/or devstack

22 / 35

Any guesses what this means

ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent [ req −8c98874d −5436−4912−9db5− f 6 7 5 b f e 4 d f 5 e − − − − −] E r r o r i n agent l o o p . D e v i c e s i n f o : { ’ c u r r e n t ’ : s e t ( [ ’ tap0ab3c3ba −09 ’ ] ) , ’ timestamps ’ : { ’ tap0ab3c3ba −09 ’ : 9} , ’ removed ’ : s e t ( [ ] ) , ’ added ’ : s e t ( [ ’ tap0ab3c3ba −09 ’ ] ) , ’ updated ’ : s e t ( [ ] ) } ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent Traceback ( most r e c e n t c a l l l a s t ) : ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t − packages / neutron / p l u g i n s /ml2/ d r i v e r s / agent /_common_agent . py" , l i n e 453 , i n daemon_loop ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent sync = s e l f . process_network_devices ( d e v i c e _ i n f o ) ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t − packages / o s p r o f i l e r / p r o f i l e r . py" , l i n e 153 , i n wrapper ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent r e t u r n f (∗ args , ∗∗ kwargs ) ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent F i l e "/ u s r / l o c a l / l i b / python2 . 7 / d i s t − packages / neutron / p l u g i n s /ml2/ d r i v e r s / agent /_common_agent . py" , l i n e 203 , i n process_network_devices . . . ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent execute_rootwrap_daemon (cmd , process_input , addl_env ) ) ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent F i l e "<s t r i n g >" , l i n e 2 , i n run_one_command ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent F i l e "/ u s r / l i b / python2 . 7 / m u l t i p r o c e s s i n g / managers . py" , l i n e 774 , i n _callmethod ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent r a i s e convert_to_error ( kind , r e s u l t ) ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent RemoteError : ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent U n s e r i a l i z a b l e message : ( ’# ERROR ’ , V a l u e E r r o r ( ’ I /O o p e r a t i o n

• n

c l o s e d f i l e ’ , ) ) ERROR neutron . p l u g i n s . ml2 . d r i v e r s . agent . _common_agent − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − 23 / 35

Setting up Compute Nodes

◮ Same basic formula ◮ Rinse and repeat 4 times ◮ Don’t forget to run nova discover_hosts after bring it up

24 / 35

25 / 35

Blank Images

DEBUG glance_store . _drivers . f i l e s y s t e m [ req −3163a1a7−4ca9 −47e8 −9444−cd8b865055fb 20 f283024ffd4bf4841a8d33bdb4f385 6 c3fc6392e0c487e85d57afe5a5ab2b7 − d e f a u l t d e f a u l t ] Wrote 0 bytes to / var / l i b / glance / images /e6735636 −43d9−4fb0−a302− f3710386b689 with checksum d41d8cd98f00b204e9800998ecf8427e add / usr / l o c a l / l i b / python2 .7/ d is t −packages / glance_store / _drivers / f i l e s y s t e m . py :706

26 / 35

DHCP Fun

27 / 35

What to do with your budget cloud?

28 / 35

OpenStack Development

◮ Lots of capacity for running devstack ◮ A really good platform to develop and test OpenStack applications ◮ For example I found 4 tempest bugs testing it on the cloud

29 / 35

Cloud Native Compute Workloads

◮ Good for running embarrasingly parallel workloads ◮ Each invidiual machine is slow, but a fair amount of parallel capacity. ◮ My example use case: https://github.com/mtreinish/handbrakecloud

30 / 35

Virtualized Home Infrastructure

◮ Gives you the flexability ◮ Also enables HA migration to public cloud

31 / 35

Installation Pain Points

◮ Python Packaging:

◮ Binary Dependencies ◮ etc files (and any data files) ◮ No dependency solver, always use upper constraints

◮ Debugging OpenStack requires a high level of competence

32 / 35

Making OpenStack Better for Small Deployments

◮ Honestly, it’s not that bad ◮ >=90% of the issues were because of using tarballs ◮ Networking and neutron is too confusing ◮ Work on improving logging and error reporting

33 / 35

Why you don’t want to do this

◮ 5x 1U Servers in your bedroom closet is not pleasant ◮ The power bill (at peak draw it’s about 1.1kW for the rack) ◮ Don’t get to spend $1,328.37 on a weekend vacation

34 / 35

Where to get more information

◮ openstack-dev ML openstack-dev@lists.openstack.org ◮ Ocata install guides https://docs.openstack.org/project-install-guide/ocata/ ◮ Ocata network guides https://docs.openstack.org/ocata/networking-guide/

35 / 35