manager (IM) Introduction General platform to deploy on-demand - - PowerPoint PPT Presentation

▶
manager im introduction
SMART_READER_LITE
LIVE PREVIEW

manager (IM) Introduction General platform to deploy on-demand - - PowerPoint PPT Presentation

Mar 17, 2024 120 likes •347 views

Infrastructure manager (IM) Introduction General platform to deploy on-demand customized virtual computing infrastructures. Web Interface CLI Interface With the precise software RADL configuration required. Infrastructure Manager

slide-1
SLIDE 1

Infrastructure manager (IM)

slide-2
SLIDE 2

Introduction

  • General platform to deploy on-demand customized virtual computing

infrastructures.

  • With the precise software

configuration required.

  • Complex infrastructures.
  • Share Infrastructure descriptions.
  • No need of pre-baked VMIs.
  • The same complex infrastructure

can be deployed both on on- premises and on public Clouds.

VMRC Infrastructure Manager Cloud Selector VMRC Cloud Connector Conf. Manager Ficheros Conf & CTX XML-RPC API REST API Web Interface CLI Interface MV MV MV

...

RADL

VM Master

Cntxt. Agent

Ansible

OpenNebula OpenStack EC2 ...

Ansible

VMRC

...

slide-3
SLIDE 3

Cloud providers

  • It supports a wide range of cloud providers and other computing

back-ends :

  • Public: Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft

Azure, Open Telekom Cloud (OTC).

  • On-premises: OpenNebula,

OpenStack.

  • Federated: EGI FedCloud (OCCI),

FogBow.

  • Containers: Docker, Kubernetes
  • The list above can be easily

extended by plugins.

…

10 nodes Memory > 1GB Ubuntu Linux Java ver. 1.6

slide-4
SLIDE 4

Main features

  • It features DevOps capabilities.
  • Based on Ansible.
  • Provides recipes for common deployments.
  • Also supporting cloud-init scripts.
  • IM works as a service that offers several interfaces:
  • XML-RPC and REST APIs.
  • Command-line application.
  • Web-based GUI.
  • It is distributed under a GNU GPL v3.0 open source license

and its source code is available on GitHub.

https://github.com/grycap/im

slide-5
SLIDE 5

Infrastructure Manager

  • The user can provide an RADL or TOSCA documents as input to the IM,

describing the infrastructure:

  • RADL:
  • Resource and Application Description Language.
  • High level Language to define virtual infrastructures and Specify VM requirements.
  • TOSCA:
  • OASIS Standard
  • Open standard language to model application architectures to be deployed on a

Cloud.

slide-6
SLIDE 6

RADL Document

  • An RADL document has the following general structure:

ansible <ansible_host_id> (<features>) network <network_id> (<features>) system <system_id> (<features>) configure <configure_id> (<Ansible recipes>) contextualize [max_time] ( system <system_id> configure <configure_id> [step <num>] ... ) deploy <system_id> <num> [<cloud_id>]

The keywords ansible, network, system and configure assign some features or recipes to an identity <id>. The features are a list

  • f constraints separated by and, and a constraint is formed

by <feature name> <operator> <value>.

network net (outbound = ‘yes') system node ( cpu.arch = 'x86_64' and cpu.count = 1 and memory.size >= 512M and net_interface.0.connection = 'net' and disk.0.os.name = 'linux‘ and disk.0.image.url = ‘one://onecloud.i3m.upv.es/67’ ) configure node ( @begin

  • tasks:
  • user: name=user1 password=1234

@end ) contextualize ( system node configure node ) deploy node 1

slide-7
SLIDE 7

A Sample tosca template: kepler

tosca_definitions_version: tosca_simple_yaml_1_0 imports:

  • indigo_custom_types: custom_types.yaml

topology_template: node_templates: kepler: type: tosca.nodes.indigo.Kepler requirements:

  • host: kepler_server

kepler_server: type: tosca.nodes.indigo.Compute capabilities: host: properties: num_cpus: 1 mem_size: 1 GB endpoint: properties: network_name: PUBLIC ports: vnc_port: protocol: tcp source: 5900

  • s:

properties: type: linux distribution: ubuntu version: 14.04 image: one://onecloud.i3m.upv.es/67

  • utputs:

instance_ip: value: { get_attribute: [ kepler_server, public_address, 0 ] } instance_creds: value: { get_attribute: [ kepler_server, endpoint, credential, 0 ] }

Network requirements OS requirements

slide-8
SLIDE 8

IM Working Scheme

  • The user specifies the image (or list of images) to use.
  • URI naming convention to abstract from cloud provider:
  • one://server:port/image-id
  • ost://server:port/ami-id
  • aws://region/ami-id
  • <site end-point>/<image-id>
  • The IM can also contact the VMRC to get a list with the most suitable VMIs
  • http://www.grycap.upv.es/vmrc
  • Then, the IM obtains the list of IaaS providers available to the user.
  • From the credentials provided by the user.
  • Finally, it contacts the IaaS provider selected and deploys the

infrastructure.

slide-9
SLIDE 9

Contextualization process

  • 1. SSH connection to the Master VM
  • A GNU/Linux-based VM with a public IP
  • 2. Configure Master VM
  • Install and configure Ansible
  • 3. Launch Contextualization Agent
  • Check SSH from VMs
  • Call Ansible

Conf. Manager VMRC

Get Apps In VMI

RADL

Requested Apps Check SSH

VM VM VM

...

Configure Master

Master VM

Cntxt. Agent

Ansible Check SSH Configure Nodes

4 1 2 3

slide-10
SLIDE 10

Client-side Tools: CLI

usage: client.py [-u|--xmlrpc-url <url>] [-a|--auth_file <filename>] operation op_parameters

  • ptions:
  • -version

show program's version number and exit

  • h, --help

show this help message and exit

  • a AUTH_FILE, --auth_file=AUTH_FILE

File with authentication data

  • u XML-RPC, --xmlrpc-url=XML-RPC

URL of the InfrastructureManager service. Operations: list create <radl_file> destroy <inf_id> getinfo <inf_id> [radl_attribute] getradl <inf_id> getcontmsg <inf_id> getstate <inf_id> getvminfo <inf_id> <vm_id> [radl_attribute] getvmcontmsg <inf_id> <vm_id> addresource <inf_id> <radl_file> [ctxt flag] removeresource <inf_id> <vm_id> [ctxt flag] alter <inf_id> <vm_id> <radl_file> start <inf_id> stop <inf_id> reconfigure <inf_id> [<radl_file>] [vm_list] startvm <inf_id> <vm_id> stopvm <inf_id> <vm_id> sshvm <inf_id> <vm_id> getversion

slide-11
SLIDE 11

Client-SIDE Tools: Web

  • Publicly-available web interface (also open-sourced).
  • Easily deploy infrastructures from a web browser
  • Share RADL/TOSCA documents.
  • http://servproject.i3m.upv.es/im/
slide-12
SLIDE 12

APIs to be consumed by clients

  • XML-RPC API
  • API that follows the XML-RPC specification.
  • REST API
  • IM Service can be accessed through a REST(ful) API
  • More info: http://www.grycap.upv.es/im/documentation.php
slide-13
SLIDE 13

Using IM in EGI FedCloud

  • Get data from AppDB.
  • Get the URI of the correct image to use.
  • Format:
  • <site end-point>/<image-id>
  • Get the instance size.
  • Use your proxy in IM the authentication data.
slide-14
SLIDE 14

A simple node with Ansible

network publica (outbound = 'yes') system front ( instance_type = 'extra-large' and net_interface.0.connection = 'publica' and disk.0.os.name = 'linux' and disk.0.image.url = 'https://fc-one.i3m.upv.es:11443/uuid_image_for_egi_centos_6_centos6kvm_im154_170' ) configure front ( @begin

  • name: Install some packages

action: yum pkg=pkg1,pkg2 state=installed @end )

Data from AppDB

slide-15
SLIDE 15

A simple node with cloud-init

network publica (outbound = 'yes') system front ( instance_type = 'extra-large' and net_interface.0.connection = 'publica' and disk.0.os.name = 'linux' and disk.0.image.url = 'https://fc-one.i3m.upv.es:11443/uuid_image_for_egi_centos_6_centos6kvm_im154_170' and ) configure front ( @begin packages:

  • pkg1
  • pkg2

@end ) contextualize ( system front configure front with cloud_init )

slide-16
SLIDE 16

VMOps Dashboard

  • Part of this functionality has been integrated in the EGI

VMOps Dashboard.

  • IM is the OCCI communication layer to create VM topologies.
  • TOSCA compatible.
  • https://wiki.egi.eu/wiki/Federated_Cloud_AppDB_VMOps_Dashboard
  • https://dashboard.appdb.egi.eu/vmops
slide-17
SLIDE 17

Where is the IM used?

  • In the INDIGO-DataCloud project:
  • IM is a key component of

the architecture:

  • Used at the PaaS Core to provide

deployment of infrastructures to Cloud sites external to INDIGO- DataCloud (including public Cloud sites).

  • Used at IaaS to provide TOSCA-based

deployment of infrastructures for OpenNebula sites.

Deployment of Customized Virtual Infrastructures using INDIGO-DataCloud

Orchestrator Service Brokering/ Policy Service QoS/SLA Service Managed Services/ Applications (MSA) Service Monitoring Service Data Management Services Infrastructure Manager

TOSCA

IAM Service Grid HPC Clusters

SAGA POSIX / WebDav SAGA POSIX / WebDav

EUDAT

CDMI

GUI-based Portlets Repository TOSCA- compliant Templates

  • 1. Access
  • 2. Authenticate
  • 3. Select /

Customize 9.a Delegate Deploy

  • 5. Status
  • 6. Prioritize
  • 7. Agree

8.a Transfer (Optional) 8.b. Transfer (Optional)

  • 4. Deploy

9.b.1 Delegate Deploy External Cloud

Cloud APIs CDMI

Partner IaaS OpenNebula

IM CDMI

OpenStack

HEAT TOSCA

9.b.2 Provision & Configure

...

Virtual Infrastructure for User Application/Service

  • 10. Deliver VMs

...

slide-18
SLIDE 18

Where is the IM used?

  • IM is a key component in EC3, a tool to deploy virtual hybrid elastic clusters that

is integrated in the EGI Access service (for the long-tail of science).

  • EC3: http://www.grycap.upv.es/ec3/
  • EGI Access: https://access.egi.eu/start
slide-19
SLIDE 19

EC3

  • The IM provisions the front-end node
  • f a virtual cluster where the number
  • f nodes can grow and shrink

(depending on the workload).

EC3 Client Infrastructure Manager (IM) VMRC Worker node Worker node Worker node Infrastructure Manager (IM) CLUES LRMS

Cloud Connector

Search VM Image Ask for a front- end node RADL file

WEB SHELL

Front-end

Administrator user Cluster user contextualizer contextualizer Deploy &

Contextualize

front-end LRMS client LRMS client LRMS client

. . .

Cloud provider (i.e. EGI FedCloud, OpenNebula, AWS…)

2 1 3 4 5 6

Submit jobs to the LRMS Elastically manage the size

  • f the cluster

Specific user SW (Galaxy, Docker,…)

  • Cost and energy-

efficient cluster- based computing.

slide-20
SLIDE 20

More Information

  • Video demos in YouTube:
  • IM demo using EGI FedCloud:
  • https://www.youtube.com/watch?v=qC_1kUnqp2A
  • More videos on the GRyCAP Channel:
  • https://www.youtube.com/channel/UCQD6RJBs57Giz4Xm8dhDczQ
  • IM image in Docker Hub:
  • https://hub.docker.com/r/grycap/im/
  • IM Web interface:
  • http://servproject.i3m.upv.es/im
  • Source Code in GitHub:
  • https://github.com/grycap/im
  • IM Web:
  • http://www.grycap.upv.es/im
slide-21
SLIDE 21

Contact

Miguel Caballer – micafer1@upv.es Germán Moltó – gmolto@dsic.upv.es Ignacio Blanquer – iblanque@dsic.upv.es

Instituto de Instrumentación para Imagen Molecular Universitat Politècnica de València Camino de Vera s/n 46022, Valencia SPAIN