CERN and Science Clouds in Europe with TOSCA, OpenStack Heat and the - - PowerPoint PPT Presentation

▶
cern and science clouds in europe with tosca openstack
SMART_READER_LITE
LIVE PREVIEW

CERN and Science Clouds in Europe with TOSCA, OpenStack Heat and the - - PowerPoint PPT Presentation

Dec 13, 2023 303 likes •572 views

CERN and Science Clouds in Europe with TOSCA, OpenStack Heat and the Heat Translator Matt Rutkowski Mathieu Velten Sahdev Zala Ricardo Rocha Brad Topol mrutkows@us.ibm.com mathieu.velten@cern.ch spzala@us.ibm.com Ricardo.Rocha@cern.ch

slide-1
SLIDE 1

CERN and Science Clouds in Europe with TOSCA, OpenStack Heat and the Heat Translator

Ricardo Rocha

Ricardo.Rocha@cern.ch

Brad Topol

btopol@us.ibm.com

Matt Rutkowski

mrutkows@us.ibm.com

Sahdev Zala

spzala@us.ibm.com @sp_zala

Mathieu Velten

mathieu.velten@cern.ch @mrutkowski @bradtopol

slide-2
SLIDE 2

Introduction

  • Indigo DataCloud Project
  • Goal is to provide a sustainable European infrastructure for science, spanning multiple computer centers and existing

public clouds.

  • Challenges
  • Need to effectively exploit heterogeneous cloud infrastructures (OpenStack, OpenNebula, proprietary clouds) as a

single cloud

  • Need a standard orchestration model that works across multiple cloud infrastructures
  • Project Approach
  • Leverage TOSCA as a standard based approach for modeling cloud stacks and applications
  • Support for TOSCA is growing in several cloud communities
  • Leverage OpenStack components as a means of deploying TOSCA cloud stacks
  • OpenStack Heat Project provides robust tools to convert alternative orchestration languages to OpenStack Heat’s

HOT templates

  • Heat-Translator: Provides a framework for translation orchestration models to Hot
  • TOSCA-Parser: Production quality tool for parsing TOSCA templates
  • In this presentation we will cover
  • TOSCA-Parser and Heat-Translator
  • Project Indigo and CERN’s initiative
  • Demo concrete examples of using TOSCA to model a complex infrastructure covering multiplesites
  • Latest on TOSCA specification
slide-3
SLIDE 3

Mitaka: TOSCA-Parser

  • What is the TOSCA-Parser?
  • A parser for TOSCA Simple and Network Functions Virtualization (NFV) YAML based

specifications.

  • Produces in-memory graph of TOSCA nodes and relationship among them
  • Subproject of OpenStack Heat
  • Two point releases – 0.4.0 and 0.3.0
  • 0.4.0
  • Release date: Feb 04, 2016
  • Introduced support for TOSCA Simple Profile for NFV v1.0
  • Support for TOSCA Groups and TOSCA Policies
  • Support for TOSCA namespace, nested properties, bug fixes etc.
  • 0.3.0
  • Release date: Dec 02, 2015
  • Full validation of TOSCA templates
  • Support for nested imports in service template and type definition
  • Shell entry point, bug fixes etc.
  • New features available on master
  • Support for LoadBalancer node type
  • Support for range data type etc.
  • PyPi releases are available at https://pypi.python.org/pypi/tosca-parser

Node Types Relationships Capabilities Interfaces Validation Tests App Server Mongo Server Node App Mongo DB Mongo DBMS Node. js HostedOn HostedOn HostedOn Groups Policies Custom Types

slide-4
SLIDE 4

Mitaka: Heat-Translator

  • What is the Heat-Translator?
  • An OpenStack project to map and translate non-Heat (e.g. TOSCA)

templates to Heat Orchestration Template (HOT)

  • Subproject of OpenStack Heat
  • Two point releases – 0.4.0 and 0.3.0
  • 0.4.0
  • Release date: March 02, 2016
  • Support for TOSCA NFV templates translation
  • Introduced support for Policies translation
  • Completed OpenStack Client plug-in work with new test suites
  • Handle Nova server specific key_name property, bug fixes etc.
  • 0.3.0
  • Release date: Dec 17, 2015
  • Querying Nova flavors to find match to TOSCA constraints
  • New CLI option --validate-only to only validate template without

translation

  • New CLI option --output-fileto save translated output to file etc.
  • New features available on master
  • New CLI option --deploy to automatically deploy translated template
  • Support for Ansible and Puppet scripts in translation etc.
  • PyPi releases are available at, https://pypi.python.org/pypi/heat-translator

TOSCA Heat Orchestration Template (HOT)

slide-5
SLIDE 5

TOSCA OpenStack Integration is Maturing Rapidly

TOSCA-Parser

Heat-Translator

Tacker

TOSCA Plugin TOSCA Integration

apps.openstack.org

OSC Plugin

Translate Deploy

slide-6
SLIDE 6

CERN and the Large Hadron Collider

  • 21 member states
  • Many other involved in other ways
  • Large Hadron Collider (LHC)
  • 27km long
  • 100m underground
  • Particle collisions at specialized detectors
  • ~600 million times / sec
  • Generating ~30PB of data / year
  • CERN Cloud in numbers
  • ~6000 nodes
  • ~150k cores
  • 2000 users
  • 16000 VMs running
slide-7
SLIDE 7

Indigo DataCloud

  • Towards a sustainable European SW infrastructure for science
  • Funded under the European Union Horizon 2020 program
  • Multiple partners, heterogeneous infrastructure (OpenStack, OpenNebula)
  • INFN, CERN, ATOS, UPV, CSIC, DESY, UPV, LIP, T-Systems, CNRS, Reply, …
  • Main strengths
  • Support for multi-disciplinary scientific communities
  • Exploits available, general solutions
  • Low learning curve, rely on popular software tools
  • Hybrid, distributed environment
  • Credits
  • German Molto (UPV), Miguel Caballer (UPV), Marica Antonacci (INFN), Alvaro Lopez Garcia

(CSIC), Giacinto Donvito (INFN)

  • And many others...

https://www.indigo-datacloud.eu

slide-8
SLIDE 8

Why TOSCA?

  • First step was to evaluate available options
  • HEAT, CloudFormation
  • Both tied to specific implementations
  • TOSCA appeared as a viable common denominator
  • Topology definition
  • End user applications
  • Existing codebase (TOSCA parser, HEAT translator)
  • Reusable in other contexts too
  • Growing support in different communities
slide-9
SLIDE 9

Current Use Cases

  • Deploy simple VM with a given image
  • Elastic cluster for batch processing (infrastructure)
  • Supported systems: torque, slurm, condor
  • Deploy a Mesos cluster (infrastructure)
  • Galaxy portal (end user)
  • Data intensive application for biomedical research
  • Indigo specific jobs (infrastructure / end user)
  • Package and deploy applications in docker containers, on

Mesos/Marathon/Chronos

slide-10
SLIDE 10

Basic example: Indigo Compute Node

tosca_definitions_version: tosca_simple_yaml_1_0 imports:

  • indigo_custom_types: custom_types.yaml

description: > TOSCA test for launching compute node with a specified image and getting as an output the IP and SSH credentials to access topology_template: node_templates:

  • utputs:

node_ip: value: { get_attribute: [ simple_node, public_address, 0 ] } simple_node: type: tosca.nodes.indigo.Compute properties: public_ip: yes capabilities: scalable: properties: count: 1 host: properties: num_cpus: 1 mem_size: 1 GB

  • s:

properties: type: linux distribution: ubuntu version: 12.04

slide-11
SLIDE 11

Basic example: Derived Types

tosca.nodes.indigo.Compute: derived_from: tosca.nodes.indigo.MonitoredCompute tosca.nodes.indigo.MonitoredCompute: derived_from: tosca.nodes.Compute properties: zabbix_server: type: string required: no default: 90.147.170.165 zabbix_server_port: type: PortDef required: no default: 32314 zabbix_server_metadata: type: string required: no default: Linux 668c875e-9a39-4dc0-a710-17c41376c1e0 interfaces: Standard: create: implementation: zabbix_agent_install.yml configure: implementation: zabbix_agent_configure.yml inputs: zabbix_server: { get_property: [ SELF, zabbix_server ] } zabbix_server_port: { get_property: [ SELF, zabbix_server_port ] } zabbix_server_metadata: { get_property: [ SELF, zabbix_server_metadata ] } start: implementation: zabbix_agent_start.yml

slide-12
SLIDE 12

Basic example: HOT

heat_template_version: 2013-05-23 description: > TOSCA test for launching compute node with a specified image and getting as an output the IP and SSH credentials to access parameters: {} resources:

  • utputs:

node_ip: value: get_attr:

  • simple_node
  • networks
  • private

simple_node: type: OS::Nova::Server properties: flavor: m1.small image: ubuntu-12.04-software-config-os-init public_ip: true user_data_format: SOFTWARE_CONFIG simple_node_create_config: ... simple_node_create_deploy: ... simple_node_configure_config: type: OS::Heat::SoftwareConfig properties: config: get_file: zabbix_agent_configure.yml group: ansible simple_node_configure_deploy: type: OS::Heat::SoftwareDeployment properties: config: get_resource: simple_node_configure_config input_values: zabbix_server: 90.147.170.165 zabbix_server_metadata: Linux 668c875e-9a39-4d... zabbix_server_port: 32314 server: get_resource: simple_node

slide-13
SLIDE 13

Basic example: Ansible Integration

interfaces: Standard: create: implementation: zabbix_agent_install.yml configure: Implementation: zabbix_agent_configure.yml inputs: zabbix_server: { get_property: [ SELF, zabbix_server ] } zabbix_server_port: { get_property: [ SELF, zabbix_server_port ] } zabbix_server_metadata: { get_property: [ SELF, zabbix_server_metadata ] } start: implementation: zabbix_agent_start.yml

  • hosts: localhost

connection: local tasks: ...

  • name: Apt install

Zabbix agent apt: name=zabbix-agent when: ansible_os_family == "Debian" tasks:

  • lineinfile: |

dest=/etc/zabbix/zabbix_agentd.conf regexp="{{ item.var }}=" line="{{ item.var }}={{ item.value }}" with_items:

  • { var: "ServerActive",

value: "{{zabbix_server}}:{{zabbix_server_port}}"}

slide-14
SLIDE 14

Basic example: Ansible Integration (Galaxy Roles)

interfaces: Standard: configure: implementation: zabbix_agent_configure.yml inputs: zabbix_server: { get_property: [ SELF, zabbix_server ] } zabbix_server_port: { get_property: [ SELF, zabbix_server_port ] } zabbix_server_metadata: { get_property: [ SELF, zabbix_server_metadata ] }

  • hosts: localhost

connection: local roles:

  • role: indigo-dc.zabbix-agent

zabbix_agent_server: "{{ zabbix_server }}" zabbix_agent_server_port: "{{ zabbix_server_port }}" zabbix_agent_metadata: "{{ zabbix_server_metadata }}"

slide-15
SLIDE 15

Multiple deployment & config options

  • Supported in the HEAT Translator
  • .yaml is Ansible
  • .pp is puppet
  • .(anything else) is scripts (python, bash, …)
  • This is reflected on the resulting HOT templates
  • Extending the translator with others should not be hard
  • We did it for Ansible
slide-16
SLIDE 16

Complex example: Batch Cluster

  • A user facing portal manages jobs running on a batch cluster
  • The cluster may spawn multiple sites
  • Sites may be running different cloud flavors
  • Mainly OpenStack and OpenNebula
  • One single frontend for the batch system
  • At least for now
slide-17
SLIDE 17

Complex example: Batch Cluster

node_templates: elastic_cluster_front_end: type: tosca.nodes.indigo.ElasticCluster properties: deployment_id: orchestrator_deployment_id requirements:

  • lrms: torque_front_end
  • wn: wn_node

torque_front_end: type: tosca.nodes.indigo.LRMS.FrontEnd.Torque requirements:

  • host: torque_server

wn_node: type: tosca.nodes.indigo.LRMS.WorkerNode.Torque capabilities: wn: properties: max_instances: 5 min_instances: 0 requirements:

  • host: torque_wn

torque_wn: type: tosca.nodes.indigo.Compute properties: public_ip: no capabilities: scalable: properties: count: 0 host: properties: num_cpus: 1 mem_size: 1 GB

  • s:

properties: type: linux distribution: ubuntu version: 14.04 torque_server: type: tosca.nodes.indigo.Compute properties: public_ip: yes capabilities: host: properties: num_cpus: 1 mem_size: 1 GB

slide-18
SLIDE 18

Demos

  • Basic compute node

https://www.youtube.com/watch?v=TaEXhcjP7x0

  • Batch cluster

https://www.youtube.com/watch?v=IyKp3cXxm4c

slide-19
SLIDE 19

Indigo TOSCA Status

  • First use cases already covered by existing tools
  • Indigo contributions upstream
  • TOSCA Parser: UPV 2nd contributor in number of commits (after IBM :-))
  • Heat Translator: CERN 2nd contributor in number of commits (after IBM :-))
  • Using the heat-translator CLI, the common openstack client, and in some

cases other APIs

  • Coming Next
  • Expand deployment to more sites within the project
  • Experiment with end user applications definitions in TOSCA
  • Continue contributing upstream to both projects
slide-20
SLIDE 20

Newton dev plan: TOSCA-Parser and Heat-Translator

  • Two to three point releases of both projects
  • New features as need arise from work on project Indigo by CERN and other members
  • Enhance Network Functions Virtualization (NFV) support with complex and real life NFV

templates continuing collaboration with OpenStack Tacker project team

  • Enhancement to both projects per OPNFV need
  • Work on completing implementing remaining TOSCA Simple Profile v1.0 specification features
  • TOSCA container parsing and translation support
  • Intrinsic functions, navigation functions
  • Enhanced artifacts support
  • Support for credential data type
  • etc.
  • Grow TOSCA templates eco system
  • Support to provide inputs via a configuration file
  • Horizon plug-in to translate and deploy TOSCA templates from dashboard
slide-21
SLIDE 21

TOSCA Simple Profile v1.1

  • Target: Public Draft release ~June 2016
  • Target “Final” 3Q 2016 -after 30-day public comment/review period

New Features

  • Metadata (completed)
  • now supported in all Types (Node, Relationship, Capability, Data, etc.) (c
  • Group Type (completed)
  • Expanded Group Type to allow management of member resources (i.e., Lifecycle)
  • Has its own Capabilities and Requirements
  • Policy Definition (completed)
  • Event-Condition-Action model (completed)
  • Includes Event Filters and Triggers
  • Cluster Type (75% completed)
  • Add support for Cluster normative type; based upon new Group Type
  • Will support new normative LoadBalancer , Scalable and Router Capability Types
  • Data Clusters (e.g., Cassandra, MongoDB, etc.) – In-Progress
  • Workflow(50% completed)
  • Intermix declarative with Imperative (e.g., Ansible, Ant, Bash)
  • Preserve investment in existing scripts for complex installations / configurations
slide-22
SLIDE 22

TOSCA Policy Definition:

<policy_name>: type: <policy_type_name> description: <policy_description> properties: <property_definitions> # allowed targets for policy association targets: [ <list_of_valid_target_resources> ] triggers: <trigger_symbolic_name_1>:

event: <event_type_name>

target_filter: node: <node_template_name> | <node_type> # (optional) reference to a related node # via a requirement requirement: <requirement_name> # (optional) Capability within node to monitor capability: <capability_name> # Describes an attribute-relative test that # causes the trigger’s action to be invoked.

condition:

<constraint_clause> action: # implementation-specific

  • peration

name <operation_name>: description: <optional description> inputs: <list_of_parameters> implementation: <script> | <service_name>

...

Event

  • Name of a normative TOSCA Event Type
  • That describes an event based upon a

Resource “state” change.

  • Or a change in one or more of the

resources attribute value.

Condition

Identifies:

  • the resource (Node) in the TOSCA

model to monitor.

  • Optionally, identify a Capability of the

identified node.

  • Describe the attribute(state) of the

resource to evaluate (condition)

TOSCA Policy Definition - Event, Condition, Action model

1..N Triggers can be declared Action

Describes:

  • An Operation (name) to invoke when

the condition is met

  • within the declared Implementation
  • Optionally, pass in Input parametersto

the operation along with any well- defined strategy values.

slide-23
SLIDE 23

TOSCA Work Group Activities

  • Interoperability
  • Conformance test suite for v1.0 (Service Templates)
  • Work underway in new GitHub repo., announcement (target ~May 2016)
  • Network Function Virtualization (NFV)
  • Working directly with OpenStack Tacker Project
  • Publishing v1.0 Public Draft 3 – All Normative Types / Models designed with Tacker contribs.
  • Define and Compose: Network Services, VNFs, Links, with Forwarding Graphs & Paths, etc.
  • Expanded scope to include Software-Defined Network (SDN) use cases
  • Container
  • Finish new Cluster capability definitions, Data Cluster use cases. for Simple Profile v1.1
  • Instance Model
  • New Group - will produce new schema for an Instance Model (reuse existing schema)
  • API potentially enabling capture, export and management of deployed application
  • Monitoring
  • Goal: Create normative event types for basic operational events
  • Focus on events types for Health, Scaling & Performance
  • Support basic “Red-Yellow-Green” and Percentage-based monitoring for dashboards
slide-24
SLIDE 24

vBrownBag TechTalk

Deploy TOSCA NFV Workloads into OpenStack Cloud

Tuesday, April 26 at 2:15 PM Austin Convention Center -4th floor - Room 13b Sahdev Zala (IBM) PTL, TOSCA-Parser and Heat-Translator Bob Haddleton (Nokia)Core reviewer, OpenStack Tacker http://openstack.prov12n.com/vbrownbag-techtalk-schedule-tuesday/

slide-25
SLIDE 25

THANK YOU!!