Avoiding Vendor Lock-In Avoiding Vendor Lock-In Using Apache - - PDF document

▶
avoiding vendor lock in avoiding vendor lock in using
SMART_READER_LITE
LIVE PREVIEW

Avoiding Vendor Lock-In Avoiding Vendor Lock-In Using Apache - - PDF document

Nov 23, 2023 346 likes •611 views

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] Avoiding Vendor Lock-In Avoiding Vendor Lock-In Using Apache Libcloud Using Apache Libcloud Tomaz Muraus Tomaz Muraus tomaz@apache.org tomaz@apache.org Cloud Open 2012,

slide-1
SLIDE 1

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 1/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Avoiding Vendor Lock-In Avoiding Vendor Lock-In Using Apache Libcloud Using Apache Libcloud

Tomaz Muraus Tomaz Muraus tomaz@apache.org tomaz@apache.org Cloud Open 2012, San Diego, CA Cloud Open 2012, San Diego, CA Agenda Agenda Who am I? Who am I? · ·

slide-2
SLIDE 2

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 2/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

What is Libcloud? What is Libcloud? Why? Why? History History Project info Project info Libcloud APIs Libcloud APIs What is currently going on & plans for the future What is currently going on & plans for the future Questions Questions · · · · · · · · · · · · · · Who am I? Who am I? Tomaz Muraus, Tomaz Muraus, @KamiSLO @KamiSLO, , +Tomaz Muraus +Tomaz Muraus Github: Github: github.com/Kami github.com/Kami Libcloud Project chair & committer Libcloud Project chair & committer Engineer at Rackspace Engineer at Rackspace · · · · · · · · · ·

slide-3
SLIDE 3

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 3/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Dude who likes open standards and open source Dude who likes open standards and open source · · What is Libcloud? What is Libcloud? “Libcloud is a Python library which abstracts differences “Libcloud is a Python library which abstracts differences between cloud provider APIs and allows users to manage between cloud provider APIs and allows users to manage their cloud resources (servers, storage, load balancers, DNS) their cloud resources (servers, storage, load balancers, DNS) using a unified and easy to use interface.” using a unified and easy to use interface.”

slide-4
SLIDE 4

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 4/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

What is Libcloud? What is Libcloud? Turns this: Turns this:

1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11

from from

libcloud.compute.types

libcloud.compute.types import import

Provider

Provider from from

libcloud.compute.providers

libcloud.compute.providers import import

get_driver

get_driver Cls Cls = =

get_driver(Provider.RACKSPACE)

get_driver(Provider.RACKSPACE) driver driver = =

Cls(

Cls('username' 'username', , 'api key' 'api key') ) size size = =

driver.list_sizes()[

driver.list_sizes()[0 0] ] images images = =

driver.list_images()[

driver.list_images()[0 0] ] node node = =

driver.create_node(name

driver.create_node(name= ='icanhasaserver' 'icanhasaserver', size , size= =size, size, image image= =image) image)

? ?

slide-5
SLIDE 5

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 5/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

What is Libcloud? What is Libcloud? Into this: Into this:

slide-6
SLIDE 6

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 6/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Why? Why? Cloud interoperability and standards are (mostly) a lie Cloud interoperability and standards are (mostly) a lie Need for standardization Need for standardization Different APIs Different APIs Different response formats (XML, JSON, text) Different response formats (XML, JSON, text) Different authentication methods Different authentication methods Different request signing mechanisms Different request signing mechanisms · · · · · · · · · · · · Why - different response formats (XML) Why - different response formats (XML)

<DescribeInstancesResponse <DescribeInstancesResponse

slide-7
SLIDE 7

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 7/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

xmlns="http://ec2.amazonaws.com/doc/2010-08-31/"> xmlns="http://ec2.amazonaws.com/doc/2010-08-31/"> <requestId>56d0fffa-8819-4658-bdd7-548f143a86d2</requestId> <requestId>56d0fffa-8819-4658-bdd7-548f143a86d2</requestId> <reservationSet> <reservationSet> <item> <item> <reservationId>r-07adf66e</reservationId> <reservationId>r-07adf66e</reservationId> <instancesSet> <instancesSet> <item> <item> <instanceId>i-4382922a</instanceId> <instanceId>i-4382922a</instanceId> <imageId>ami-0d57b264</imageId> <imageId>ami-0d57b264</imageId> <instanceState> <instanceState> <code>0</code> <code>0</code> <name>pending</name> <name>pending</name> </instanceState> </instanceState> <privateDnsName/> <privateDnsName/>

Why - different response formats (text) Why - different response formats (text)

de:0:0:write:requests 466 de:0:0:write:requests 466 rx 760681 rx 760681 vnc:password testpass vnc:password testpass ide:0:0 f0202f1c-0b4f-4cfc-8ae3-e30951d09ef0 ide:0:0 f0202f1c-0b4f-4cfc-8ae3-e30951d09ef0 ide:0:0:read:requests 7467 ide:0:0:read:requests 7467 ide:0:0:read:bytes 165395968 ide:0:0:read:bytes 165395968 vnc:ip 178.22.66.28 vnc:ip 178.22.66.28 boot ide:0:0 boot ide:0:0

slide-8
SLIDE 8

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 8/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

smp 1 smp 1 started 1286568422 started 1286568422 nic:0:model virtio nic:0:model virtio status active status active user 93b34fd9-7986-4b25-8bfd-98a50383605d user 93b34fd9-7986-4b25-8bfd-98a50383605d ide:0:0:media disk ide:0:0:media disk name cloudsigma node name cloudsigma node

Why - different response formats (JSON) Why - different response formats (JSON)

[ [ { { "ips": [ "ips": [ { { "address": "67.214.214.212" "address": "67.214.214.212" } } ], ], "memory": 1073741824, "memory": 1073741824, "id": "99df878c-6e5c-4945-a635-d94da9fd3146", "id": "99df878c-6e5c-4945-a635-d94da9fd3146", "storage": 21474836480, "storage": 21474836480, "hostname": "foo.apitest.blueboxgrid.com", "hostname": "foo.apitest.blueboxgrid.com", "description": "1 GB RAM + 20 GB Disk", "description": "1 GB RAM + 20 GB Disk", "cpu": 0.5, "cpu": 0.5, "status": "running" "status": "running"

slide-9
SLIDE 9

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 9/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

} } ] ]

Why - different authentication methods Why - different authentication methods Shared token / secret Shared token / secret HMAC based HMAC based HTTP basic / digest auth HTTP basic / digest auth X509 certificate-based X509 certificate-based · · · · · · · ·

slide-10
SLIDE 10

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 10/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Apache Libcloud Apache Libcloud Python library (pip install apache-libcloud) Python library (pip install apache-libcloud) Originally developed at Cloudkick in 2009 Originally developed at Cloudkick in 2009 Later this year project joined Apache Incubator Later this year project joined Apache Incubator Graduated to Apache TLP in May 2011 Graduated to Apache TLP in May 2011 Current stable release is Current stable release is 0.11.1 0.11.1 Similar libraries in other languages: Similar libraries in other languages: Fog Fog (Ruby) (Ruby) jclouds jclouds (Java) (Java) deltacloud deltacloud · · · · · · · · · · · · · · · · · ·

slide-11
SLIDE 11

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 11/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Apache Libcloud Apache Libcloud 8 committers 8 committers A decent amount of contributions from the community A decent amount of contributions from the community Mailing list: {users,dev}@libcloud.apache.org Mailing list: {users,dev}@libcloud.apache.org IRC channel: #libcloud on irc.freenode.net IRC channel: #libcloud on irc.freenode.net · · · · · · · · Apache Libcloud - who is using it Apache Libcloud - who is using it Rackspace Rackspace · ·

slide-12
SLIDE 12

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 12/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Server Density Server Density CollabNet CollabNet Salt Stack Salt Stack · · · · · · Libcloud APIs Libcloud APIs Compute Compute Storage Storage DNS DNS Load balancers Load balancers · · · · · · · ·

slide-13
SLIDE 13

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 13/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Libcloud APIs - Supported providers Libcloud APIs - Supported providers

slide-14
SLIDE 14

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 14/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Libcloud APIs - Compute Libcloud APIs - Compute Allows users to manage VMs / cloud servers across more Allows users to manage VMs / cloud servers across more than 25 different providers than 25 different providers Includes limited support for block storage (EBS, etc.) Includes limited support for block storage (EBS, etc.) management management Supported services include: Amazon EC2, Rackspace Supported services include: Amazon EC2, Rackspace Cloud Servers, Cloud Servers, OpenStack, CloudStack, ... OpenStack, CloudStack, ... · · · · · ·

slide-15
SLIDE 15

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 15/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Libcloud APIs - Compute - Base API Libcloud APIs - Compute - Base API list_nodes list_nodes list_images list_images list_sizes list_sizes list_locations list_locations create_node create_node deploy_node deploy_node reboot_node reboot_node destroy_node destroy_node · · · · · · · · · · · · · · · ·

slide-16
SLIDE 16

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 16/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Libcloud APIs - Compute - Example Libcloud APIs - Compute - Example

1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 16 16 17 17 18 18

import import

  • s
  • s

from from

libcloud.compute.types

libcloud.compute.types import import

Provider

Provider from from

libcloud.compute.providers

libcloud.compute.providers import import

get_driver

get_driver from from

libcloud.compute.deployment

libcloud.compute.deployment import import

MultiStepDeployment, \

MultiStepDeployment, \ ScriptDeployment, SSHKeyDeployment ScriptDeployment, SSHKeyDeployment conn conn = =

get_driver(Provider.RACKSPACE)(

get_driver(Provider.RACKSPACE)('username' 'username', , 'key' 'key') ) install_key install_key = =

SSHKeyDeployment(

SSHKeyDeployment(open

  • pen(os.path.expanduser(

(os.path.expanduser("~/.ssh/id_rsa.pub" "~/.ssh/id_rsa.pub" install_puppet install_puppet = =

ScriptDeployment(

ScriptDeployment("apt­get ­y install puppet" "apt­get ­y install puppet") ) msd msd = =

MultiStepDeployment([install_key, install_puppet])

MultiStepDeployment([install_key, install_puppet]) image image = =

conn.list_images()[

conn.list_images()[0 0] ] size size = =

conn.list_sizes()[

conn.list_sizes()[0 0] ] node node = =

conn.deploy_node(name

conn.deploy_node(name= ='test' 'test', image , image= =image, size image, size= =size, size, deploy deploy= =msd) msd)

Libcloud APIs - Storage Libcloud APIs - Storage Allows users to manage cloud storage across 5 different Allows users to manage cloud storage across 5 different · ·

? ?

slide-17
SLIDE 17

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 17/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

providers providers Supported services include: Amazon S3, Rackspace Cloud Supported services include: Amazon S3, Rackspace Cloud Files, Google Files, Google Storage, OpenStack Swift Storage, OpenStack Swift · · Libcloud APIs - Storage - Base API Libcloud APIs - Storage - Base API list_containers list_containers list_container_object list_container_object get_container get_container get_object get_object · · · · · · · ·

slide-18
SLIDE 18

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 18/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

create_container create_container upload_object upload_object upload_object_via_stream upload_object_via_stream download_object download_object download_object_as_stream download_object_as_stream delete_container delete_container delete_object delete_object · · · · · · · · · · · · · · Libcloud APIs - Storage - Example Libcloud APIs - Storage - Example

1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 16 16 17 17 18 18 19 19

import import

subprocess

subprocess from from

libcloud.storage.types

libcloud.storage.types import import

Provider

Provider from from

libcloud.storage.providers

libcloud.storage.providers import import

get_driver

get_driver driver driver = =

get_driver(Provider.CLOUDFILES_US)(

get_driver(Provider.CLOUDFILES_US)('username' 'username', , 'key' 'key') ) directory directory = =

'/home/some/path'

'/home/some/path' cmd cmd = =

'tar cvzpf ­ %s'

'tar cvzpf ­ %s'

%

%

(directory)

(directory) container container = =

driver.create_container(

driver.create_container('backups' 'backups') ) pipe pipe = =

subprocess.Popen(cmd, bufsize

subprocess.Popen(cmd, bufsize= =0 0, shell , shell= =True True, stdout , stdout= =subprocess.PIPE) subprocess.PIPE) return_code return_code = =

pipe.poll()

pipe.poll() while while

return_code

return_code is is

None

None: : # Compress data in our directory and stream it directly to CF # Compress data in our directory and stream it directly to CF container.upload_object_via_stream(iterator container.upload_object_via_stream(iterator= =pipe.stdout, pipe.stdout,

  • bject_name
  • bject_name=

='backup.tar.gz' 'backup.tar.gz') ) return_code return_code = =

pipe.poll()

pipe.poll() print print

'Upload complete'

'Upload complete'

? ?

slide-19
SLIDE 19

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 19/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Libcloud APIs - Load balancers Libcloud APIs - Load balancers Allows users to manage cloud load balancers across 4 Allows users to manage cloud load balancers across 4 different providers different providers Supported services: Rackspace LoadBalancers, CloudStack Supported services: Rackspace LoadBalancers, CloudStack LoadBalancers, ... LoadBalancers, ... · · · ·

slide-20
SLIDE 20

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 20/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Libcloud APIs - Load balancers - Base API Libcloud APIs - Load balancers - Base API list_protocols list_protocols list_balancers list_balancers balancer_list_members balancer_list_members get_balancer get_balancer create_balancer create_balancer destroy_balancer destroy_balancer balancer_attach_member balancer_attach_member balancer_attach_compute_node balancer_attach_compute_node balancer_detach_member balancer_detach_member · · · · · · · · · · · · · · · · · ·

slide-21
SLIDE 21

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 21/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Libcloud APIs - Load balancers - Example Libcloud APIs - Load balancers - Example

1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 16 16 17 17 18 18

Cls Cls = =

get_driver(Provider.RACKSPACE_US)

get_driver(Provider.RACKSPACE_US) driver driver = =

Cls(

Cls('username' 'username', , 'api key' 'api key') ) new_balancer new_balancer = =

driver.create_balancer(name

driver.create_balancer(name= ='test­lb' 'test­lb', , algorithm algorithm= =Algorithm.ROUND_ROBIN, port Algorithm.ROUND_ROBIN, port= =80 80, protocol , protocol= ='http' 'http', , members members= =(Member( (Member(None None, , '192.168.86.1' '192.168.86.1', , 8080 8080), ), Member( Member(None None, , '192.168.86.2' '192.168.86.2', , 8080 8080))) ))) # wait for the balancer to become ready # wait for the balancer to become ready while while

True

True: : balancer balancer = =

driver.get_balancer(balancer_id

driver.get_balancer(balancer_id= =new_balancer. new_balancer.id id) ) if if

balancer.state

balancer.state = == =

State.RUNNING:

State.RUNNING: break break time.sleep( time.sleep(20 20) ) # fetch list of members # fetch list of members members members = =

balancer.list_members()

balancer.list_members() print print

members

members

Libcloud APIs - DNS Libcloud APIs - DNS Allows users to manage DNS across 3 different providers Allows users to manage DNS across 3 different providers · ·

? ?

slide-22
SLIDE 22

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 22/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Supported services: Zerigo DNS, Rackspace DNS, Linode Supported services: Zerigo DNS, Rackspace DNS, Linode DNS DNS · · Libcloud APIs - DNS - Base API Libcloud APIs - DNS - Base API list_record_types list_record_types list_zones list_zones list_records list_records get_zone get_zone · · · · · · · · · · · · · ·

slide-23
SLIDE 23

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 23/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

get_record get_record create_zone create_zone update_zone update_zone create_record create_record update_record update_record delete_zone delete_zone delete_record delete_record · · · · · · · · · · · · · · · · · · · · · · · · · · · · Libcloud APIs - DNS - Example Libcloud APIs - DNS - Example

1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 16 16 17 17 18 18 19 19

from from

pprint

pprint import import

pprint

pprint from from

libcloud.compute.providers

libcloud.compute.providers import import

get_driver as get_compute_driver

get_driver as get_compute_driver from from

libcloud.compute.types

libcloud.compute.types import import

Provider as ComputeProvider

Provider as ComputeProvider from from

libcloud.dns.providers

libcloud.dns.providers import import

get_driver as get_dns_driver

get_driver as get_dns_driver from from

libcloud.dns.types

libcloud.dns.types import import

Provider as DNSProvider, RecordType

Provider as DNSProvider, RecordType compute_driver compute_driver = =

get_dns_driver(DNSProvider.ZERIGO)(

get_dns_driver(DNSProvider.ZERIGO)('username' 'username', , 'api key' 'api key' dns_driver dns_driver = =

get_dns_driver(DNSProvider.ZERIGO)(

get_dns_driver(DNSProvider.ZERIGO)('email' 'email', , 'password' 'password') ) nodes nodes = =

compute_driver.list_nodes()

compute_driver.list_nodes() zone zone = =

dns_driver.create_zone(domain

dns_driver.create_zone(domain= ='mydomain2.com' 'mydomain2.com') ) created created = =

[]

[] for for

node

node in in

nodes:

nodes: print print

'Creating %s record (data=%s) for node %s'

'Creating %s record (data=%s) for node %s'

%

%

(

('A' 'A', ip, name) , ip, name) record record = =

zone.create_record(name

zone.create_record(name= =node.name, node.name, type type= =RecordType.A, RecordType.A, data data= =node.public_ips[ node.public_ips[0 0]) ])

? ?

slide-24
SLIDE 24

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 24/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

20 20 21 21 22 22 23 23

created.append(record) created.append(record) print print

'Done, created %d records'

'Done, created %d records'

%

%

(

(len len(created)) (created)) pprint(created) pprint(created)

What is currently going on & plans for the future What is currently going on & plans for the future Libcloud REST - talk to Libcloud over HTTP Libcloud REST - talk to Libcloud over HTTP Improving pricing data distribution Improving pricing data distribution Revamping the whole “location” concept Revamping the whole “location” concept ... ... · · · · · · · ·

slide-25
SLIDE 25

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 25/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Rackspace (SF) is hiring Rackspace (SF) is hiring Python, Node.js, Lua, Ruby, Java, ... Python, Node.js, Lua, Ruby, Java, ... Chef, Cassandra, ZooKeeper, Scribe, ... Chef, Cassandra, ZooKeeper, Scribe, ... Cloud, Big data, ... Cloud, Big data, ... We <3 open source We <3 open source We <3 cloud We <3 cloud http://rackertalent.com/sanfrancisco/ http://rackertalent.com/sanfrancisco/ http://github.com/racker/ http://github.com/racker/ http://github.com/Rackspace/ http://github.com/Rackspace/ · · · · · · · · · · · · · · · ·

slide-26
SLIDE 26

9/7/12 Avoiding Vendor Lock-in Using Apache Libcloud [www.tomaz.me] 26/26 www.tomaz.me/slides/avoiding_vendor_lock_in_using_apache_libcloud/#1

Thank you & Questions Thank you & Questions

Website: Website: http://libcloud.apache.org http://libcloud.apache.org Docs: Docs: http://libcloud.apache.org/docs/ http://libcloud.apache.org/docs/ Mailing lists: Mailing lists: {users,dev}@libcloud.apache.org {users,dev}@libcloud.apache.org IRC: #libcloud on irc.freenode.net IRC: #libcloud on irc.freenode.net · · · · · · · ·