Monasca: Project Onboarding Witold Bedyk (Fujitsu) Johannes Graler - - PowerPoint PPT Presentation

▶
monasca project onboarding
SMART_READER_LITE
LIVE PREVIEW

Monasca: Project Onboarding Witold Bedyk (Fujitsu) Johannes Graler - - PowerPoint PPT Presentation

Mar 24, 2024 226 likes •939 views

Monasca: Project Onboarding Witold Bedyk (Fujitsu) Johannes Graler (SUSE) witold.bedyk@est.fujitsu.com johannes.grassler@suse.com IRC: witek IRC: jgrassler Preliminaries Slides and Transcript Compiled slides (recommended)

slide-1
SLIDE 1

Monasca: Project Onboarding

Witold Bedyk (Fujitsu) witold.bedyk@est.fujitsu.com IRC: witek Johannes Graßler (SUSE) johannes.grassler@suse.com IRC: jgrassler

slide-2
SLIDE 2

Preliminaries

slide-3
SLIDE 3

3

Slides and Transcript

  • Compiled slides (recommended)

‒ http://btw23.de/johannes/talks/monasca-onboarding.tar.bz2

  • odpdown sources

‒ https://github.com/jgrassler/talks/tree/master/monasca-onboarding

slide-4
SLIDE 4

4

This Session

  • What it is:

‒ Primer on Monasca ‒ Overview of Monasca repositories and architecture ‒ Introduction to the specifics of Monasca development ‒ How can you contribute?

  • What it is not

‒ General introduction to OpenStack development ‒ Refer to Code & Documentation Contributor Guide for that.

slide-5
SLIDE 5

5

What is Monasca?

  • Monitoring and Logging as a Service

‒ Highly scalable ‒ Fault tolerant ‒ High Performance ‒ Multi-tenant

slide-6
SLIDE 6

6

What is Monasca? (cont.)

  • Features:

‒ Metrics with dimensions (key/value pairs) as metadata ‒ Real-time alerting ‒ Pluggable notification engine ‒ Flexible aggregation engine

slide-7
SLIDE 7

7

Sources of Documentation

  • https://docs.openstack.org/monasca-api
  • https://wiki.openstack.org/wiki/Monasca
  • http://monasca.io/
slide-8
SLIDE 8

8

Main Contributors

  • Fujitsu
  • HPE
  • OP5
  • StackHPC
  • SUSE
slide-9
SLIDE 9

Architecture and Development

slide-10
SLIDE 10

10

Metrics API (mon

  • nasc

sca- a-api pi)

slide-11
SLIDE 11

11

Metrics API (mon

  • nasc

sca- a-api pi)

slide-12
SLIDE 12

12

Metrics API (mon

  • nasc

sca- a-api pi)

  • Repository

‒ https://github.com/openstack/monasca-api

  • Purpose

‒ Receives metrics from agents ‒ Makes metrics available for visualization/processing ‒ Interface for modifying configuration database (alarms, notifications, ...)

  • Development Information

‒ Most important documentation repository for Monasca: source for https://docs.openstack.org/monasca-api ‒ API reference: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md ‒ Contains data model for configuration database (monasca_api/common/repositories) ‒ Contains database migrations for configuration database (being added in OpenStack Rocky) ‒ Deprecated Java implementation: ignore when contributing

slide-13
SLIDE 13

13

Creating Database Migrations

  • Generate skeleton revision

$ cd /opt/stack/monasca-api/monasca_api/db/ alembic revision Generating /opt/stack/monasca-api/monasca_api/db/alembic/versio ns/8ae1c3750508_.py ... done

  • Edit revision

$EDITOR alembic/versions/8ae1c3750508_.py

slide-14
SLIDE 14

14

Metrics API (mon

  • nasc

sca- a-api pi)

slide-15
SLIDE 15

15

Monasca Agent (mo mona nasca ca-a

  • age

gent)

slide-16
SLIDE 16

16

Monasca Agent (mo mona nasca ca-a

  • age

gent)

  • Repository

‒ https://github.com/openstack/monasca-agent

  • Purpose

‒ Collect metrics on monitored systems and forward them to monasca-api ‒ Easily extensible by adding custom plugins

  • Development Information

‒ Check plugins (for collecting metrics) in monasca_agent/collector/checks_d ‒ Detection plugins (for detecting/configuring checks with monasca-setup) in

monasca_setup/detection/plugins

‒ Please create both if you add a new check. ‒ Detailed documentation available in README

slide-17
SLIDE 17

17

Monasca Agent (mo mona nasca ca-a

  • age

gent)

slide-18
SLIDE 18

18

Monasca Client (py pyth thon- n-mo monas asca cacli lient nt)

slide-19
SLIDE 19

19

Monasca Client (py pyth thon- n-mo monas asca cacli lient nt)

  • Repository

‒ https://github.com/openstack/python-monascaclient

  • Purpose

‒ Python client library and CLI client for the Monasca Metrics API ‒ Used by users to retrieve metrics/manipulate alarms and by all components that

communicate with the Metrics API

  • Development Information

‒ If you extend the Metrics API, you will have to implement the client side of that

extension in python-monascaclient.

slide-20
SLIDE 20

20

Monasca Client (py pyth thon- n-mo monas asca cacli lient nt)

slide-21
SLIDE 21

21

Horizon Plugin (mon

  • nas

asca ca-ui ui)

slide-22
SLIDE 22

22

Horizon Plugin (mon

  • nas

asca ca-ui ui)

  • Repository

‒ https://github.com/openstack/monasca-ui

  • Purpose

‒ Configuration of alarms/thresholds ‒ Visualizing alarms ‒ Provide links to metrics and log dashboards

slide-23
SLIDE 23

23

Horizon Plugin (mon

  • nas

asca ca-ui ui)

slide-24
SLIDE 24

24

Message Queue

slide-25
SLIDE 25

25

Message Queue: Interconnects Monasca Components

  • Repository

‒ N/A (third party component; Apache Kafka)

  • Purpose

‒ Shuttle metrics, notifications and log entries back and forth between components

slide-26
SLIDE 26

26

Notification Engine (mo monas asca ca-no noti tifj fjca catio ion)

slide-27
SLIDE 27

27

Notification Engine (mo monas asca ca-no noti tifj fjca catio ion)

  • Repository

‒ https://github.com/openstack/monasca-notification

  • Purpose

‒ Sends notifications if triggered by alarm ‒ Supports E-Mail, Webhooks and various chat protocols

  • Development Information

‒ Plugin based ‒ Plugins in monasca_notifjcation/plugins/ ‒ Plugins must inherit from

monasca_notifjcation.plugins.abstract_notifjer.AbstractNotifjer

‒ Plugins must be registered in configuration file

slide-28
SLIDE 28

28

Notification Engine (mo monas asca ca-no noti tifj fjca catio ion)

slide-29
SLIDE 29

29

Threshold Engine (mona nasc sca-t

  • thr

hresh sh)

slide-30
SLIDE 30

30

Threshold Engine (mona nasc sca-t

  • thr

hresh sh)

  • Repository

‒ https://github.com/openstack/monasca-thresh

  • Purpose

‒ Listen in on metrics and check them against alarm thresholds ‒ Produces messages for monasca-notifjcation if thresholds exceeded

  • Development Information

‒ Implemented in Java ‒ Contributions may entail changes to monasca-common ‒ Uses Apache Storm for processing metrics

slide-31
SLIDE 31

31

Threshold Engine (mona nasc sca-t

  • thr

hresh sh)

slide-32
SLIDE 32

32

Transform Engine (mon

  • nasc

sca- a-tr trans nsfo form)

slide-33
SLIDE 33

33

Transform Engine (mon

  • nasc

sca- a-tr trans nsfo form)

  • Repository

‒ https://github.com/openstack/monasca-transform

  • Purpose

‒ Republish transformed (usually aggregated) metrics as synthetic new metrics

slide-34
SLIDE 34

34

Transform Engine (mon

  • nasc

sca- a-tr trans nsfo form)

slide-35
SLIDE 35

35

Persister (mona nasc sca-p

  • per

ersis ister er)

slide-36
SLIDE 36

36

Persister (mona nasc sca-p

  • per

ersis ister er)

  • Repository

‒ https://github.com/openstack/monasca-persister

  • Purpose

‒ Consumes metrics from message queue ‒ Stores metrics in time series database

  • Development Information

‒ Two implementations: Java and Python ‒ Contributions may entail changes to monasca-common

slide-37
SLIDE 37

37

Persister (mona nasc sca-p

  • per

ersis ister er)

slide-38
SLIDE 38

38

Time Series Database for Measurements

slide-39
SLIDE 39

39

Time Series Database for Measurements

  • Repository

‒ N/A (third party component; can be Cassandra, InfluxDB or Vertica)

  • Purpose

‒ Store metrics

  • Development Information

‒ To support a new type of time series database, you will need to add code to

monasca-common, monasca-api and monasca-persister.

slide-40
SLIDE 40

Monasca Logging Architecture

slide-41
SLIDE 41

41

Monasca Log API (mo monas asca ca-lo log- g-api pi)

slide-42
SLIDE 42

42

Monasca Log API (mo monas asca ca-lo log- g-api pi)

  • Repository

‒ https://github.com/openstack/monasca-log-api

  • Purpose

‒ Receives log messages from agents

  • Development Information

‒ Repository contains logging specific parts of documentation ‒ Contributions may entail changes to monasca-common

slide-43
SLIDE 43

43

Monasca Log API (mo monas asca ca-lo log- g-api pi)

slide-44
SLIDE 44

44

Log Agents

slide-45
SLIDE 45

45

Log Agents

  • Repository: N/A
  • Purpose: Send logs
  • Not part of Monasca: logstash, beaver or fluentd with Monasca output

plugin.

slide-46
SLIDE 46

46

Log Agents

slide-47
SLIDE 47

47

Monasca Logging

slide-48
SLIDE 48

48

Log Metrics

slide-49
SLIDE 49

49

Log Transformer

slide-50
SLIDE 50

50

Log Persister

slide-51
SLIDE 51

51

Elasticsearch

slide-52
SLIDE 52

52

Kibana

slide-53
SLIDE 53

53

mona nasc sca- a-kib iban ana-p

  • plug

ugin in

  • Repository

‒ https://github.com/openstack/monasca-kibana-plugin

slide-54
SLIDE 54

54

Monasca Logging

slide-55
SLIDE 55

55

Tutorial

  • Interactive Jupyter notebook
  • Demonstrates main Monasca functionalities
  • https://github.com/witekest/monasca-bootcamp/
slide-56
SLIDE 56

Development Environment

slide-57
SLIDE 57

57

Devstack Setup for Monasca

  • local.conf for default (Python based) Monasca stack

enable_plugin monasca-api \ git://git.openstack.org/openstack/monasca-api

  • local.conf setting for Java based persister

MONASCA_PERSISTER_IMPLEMENTATION_LANG=java

slide-58
SLIDE 58

58

Devstack Setup with Vagrant

# cd monasca-api/devstack # vagrant up

slide-59
SLIDE 59

59

monasca-docker

  • Containerized Monasca deployed with Docker Compose

# git clone https://github.com/monasca/monasca- docker # cd monasca-docker # docker-compose up

slide-60
SLIDE 60

60

Running unit tests

# cd $REPO # tox # tox -e py27,py35 # tox -e pep8

slide-61
SLIDE 61

61

Running Integration (Tempest) Tests in Devstack

  • Add monasca-tempest-plugin to local.conf

enable_plugin monasca-tempest-plugin \ https://git.openstack.org/openstack/monasca- tempest-plugin

  • Run tests

# cd /opt/stack/tempest # tempest run -r monasca_tempest_tests.tests.api # tempest run -r monasca_tempest_tests.tests.log_api

slide-62
SLIDE 62

62

Running Tempest Tests with mon

  • nas

asca- a-do dock cker

  • Add section to docker-compose.yaml:

tempest-tests: image: monasca/tempest-tests:latest environment: KEYSTONE_SERVER: "keystone" STAY_ALIVE_ON_FAILURE: "true" MONASCA_WAIT_FOR_API: "true"

  • Run tests

# docker-compose up -d tempest-tests

slide-63
SLIDE 63

Become part of our community

slide-64
SLIDE 64

64

Why Contribute?

  • Pluggable
  • Modular
  • Customizable
  • Small and friendly community
slide-65
SLIDE 65

65

How to contribute?

  • Contributor Guide
  • We use StoryBoard!

‒ Bugs ‒ Feature requests

  • Specifications repository

‒ openstack/monasca-specs

slide-66
SLIDE 66

66

Work to do

  • Project priorities

‒ http://specs.openstack.org/openstack/monasca-specs/priorities/rocky-priorities.html

  • Important Tasks and Reviews

‒ https://storyboard.openstack.org/#!/board/60

slide-67
SLIDE 67

67

Where can you help?

  • Reviews
  • Bugfixes and Bug Reports

https://storyboard.openstack.org/#!/worklist/213

  • Community wide goals
  • Installers
  • Documentation
slide-68
SLIDE 68

Questions?

slide-69
SLIDE 69

Thank You!

slide-70
SLIDE 70

70

Slides and Transcript: Thank you!

https://bit.ly/2IfT7c7