Integration of Enterprise Monitoring Product, Senlin and Mistral - - PowerPoint PPT Presentation

Integration of Enterprise Monitoring Product, Senlin and Mistral for Auto-healing

Xinhui Li(VMware), Wolf Peter(Nokia), Qiming Teng(IBM)

Host OpenStack VM Application

• Physical nodes
• Physical network
• Physical storage
• Hypervisor
• Host OS
• …
• Compute Controller
• Network Controller
• Database
• Message Queue
• Storage
• …
• Service Resiliency
• Quality of Service
• Cost
• Transparency
• Data Integrity
• ...
• Virtual Machine
• Incl. Container
• Virtual Network
• Virtual Storage
• VM Mobility
• Ease of Management
• ...

Motivation

Auto-Healing by Senlin

Nova Server

• s-collect-config

app monitor Ceilometer/Aodh Alarm Senlin-engine Receiver

MQ

nova Cluster “Restart” notification

MQ

2

Application Events

2

VM Events

3

A native signal / alarm

1

A notion of VM groups for

• VM/Application redundancy
• HA policies

4

Recover Action:

• reboot
• rebuild
• migrate
• remote-restart

2

Polling

Gap to an industry-level Solution

 Monitoring/Alarm/Notification is the foundation of Day2

• peration.

 reliable and scalable monitoring  across different sources

 Support users’ own custom workflow

 Long-run task  Error handling and flexible workflow

 Integrate with VRops and Mistral

4

Agenda

Senlin overview Mistral as recover action Integration with vRops for alert and notification Demo

5

Senlin – Resource Pool Manager

Scaling Load-Balancing Health Deletion ...... Region Placement Zone Placement Affinity

Senlin Engine Senlin Engine Senlin API

Senlin CLI OSC plug-in Senlin Dashboard Python/Java Bindings

heat stacks nova heat VM servers docker containers

• penstacksdk

docker-py

How to Use Senlin - Cluster

7  Rich primitives and operations

 Cluster membership management  Cluster scale/capacity management

 Scale in/out  Resize

 Policy management

 Auto-Scaling  Aut-Healing  Placement  Load-balancing

 Create Standby/Active Cluster by move node among different clusters

 Support Blue/Green mode

Senlin Polices

 Placement policy

8

type: senlin.policy.affinity version: 1.0 properties: servergroup: name: my_server_group policies: affinity availability_zone: nova enable_drs_extension: false health_policy: type: OS::Senlin::Policy properties: type: senlin.policy.health-1.0 bindings:

• cluster: {get_resource: cluster}

properties: detection: type: VM_LIFECYCLE_EVENTS recovery: actions:

• REBUILD

 Health Policy

Agenda

Senlin overview Mistral as recover action Integration with vRops for alert and notification Demo

9

Workflow as Recover Action

10

Agenda

Senlin overview Mistral as recover action Integration with vRops for alert and notification Demo

11

12

vRops - Policy driven Performance, Capacity, Configuration and Compliance analytics

13

Adapter Instance Status list Solution Status

Managing Data Collection from Different Sources

Sample Heat Template – vROps Alert Resources

14

vROps-notification

Sample Heat Template – Hook Back to OpenStack

CONFIDENTIAL

15

Demo

16

Heat Template

Senlin cluster 1

vm vm vm

Senlin cluster 2

vm vm

Custom group: type: name value: cluster-name rule: contains Notifications Recover: Mistral based resize receiver

VIO OpenStack

Future Work

 Event correlation

 Event collection across layers  Correlate events to find root cause and react

 Sequence of Actions

 Workflow comprised of more than one nova/heat action

Q & A