Towards Omnia: a Monitoring Factory for Quality-Aware DevOps Apr 27 - - PowerPoint PPT Presentation

▶
towards omnia a monitoring factory for quality aware
SMART_READER_LITE
LIVE PREVIEW

Towards Omnia: a Monitoring Factory for Quality-Aware DevOps Apr 27 - - PowerPoint PPT Presentation

Feb 15, 2024 155 likes •444 views

Towards Omnia: a Monitoring Factory for Quality-Aware DevOps Apr 27 th , 2017 Marco MIGLIERINA Damian A. TAMBURRI Dev- to Ops: We are moving faster and faster 2 Figures from: http://martinfowler.com/articles/microservices.html and

slide-1
SLIDE 1

Towards Omnia: a Monitoring Factory for Quality-Aware DevOps

Apr 27th, 2017

Marco MIGLIERINA Damian A. TAMBURRI

slide-2
SLIDE 2

Dev- to Ops: We are moving faster and faster

2

Figures from: http://martinfowler.com/articles/microservices.html and https://www.slideshare.net/CiscoDevNet/enabing-devops-in-an-sdn-world

slide-3
SLIDE 3

Observability is essential

3

slide-4
SLIDE 4

Monitoring lagging behind

4

…back in 2011

slide-5
SLIDE 5

Since then… proliferation of tools and solutions

5 144 practitioners surveyed

  • M. Miglierina. Monitoring Modern Distributed Software Applications: Challenges And Solutions. PhD thesis, Politecnico di Milano, 2017.
slide-6
SLIDE 6

Main perceived drawback in monitoring?

6

  • 1. Lack of standards
  • 2. Too many tools
  • 3. Lack of usability
  • M. Miglierina. Monitoring Modern Distributed Software Applications: Challenges And Soloutions. PhD thesis, Politecnico di Milano, 2017.
slide-7
SLIDE 7

Omnia

  • Main objectives
  • reduce learning curve and entry cost to monitoring
  • an attempt of standardization
  • How?
  • One, interoperable, self-service monitoring interface for devs
  • A simple monitoring factory for ops

7

DEVS

Monitoring Stack

OPS

use manage

slide-8
SLIDE 8

Reference team organization

Prod Mgr UX Dev QA DB Admin Product Team Product Team Product Team

A P I

Sys Admin Net Admin SAN Admin Platform Team

8

  • Product teams
  • each responsible of its microservice
  • independent workflows
  • Platform team
  • provide infrastructure support
  • cross functional wrt product teams

Team organzation at Netflix: https://www.nginx.com/blog/adopting-microservices- at-netflix-lessons-for-team-and-process-design/

slide-9
SLIDE 9

The Omnia components

9

Product Team Product Team Product Team Omnia Libs

  • mnia.yml

Product Team Platform Team

  • mnia.admin.yml

Omnia CLI Monitoring Stack Monitoring Interface Monitoring factory Omnia Vocabulary

slide-10
SLIDE 10

Classical Approach (1)

10

automated action manual action Product Team Alfa Service Alfa Dashboard X TDB X Platform Team maintains Agents X learn, configure, deploy Alerting X

The Platform Team decides to use the monitoring stack X

slide-11
SLIDE 11

Classical Approach (2)

11

automated action manual action Product Team Alfa Service Alfa v0.1 TDB X Lib Dashboard X TDB X Platform Team learn, configure learn, code, release Agents X Alerting X learn, configure

Product Teams have to learn how to use X and instrument their code

slide-12
SLIDE 12

Classical Approach (3)

12

automated action manual action Product Team Alfa Service Alfa v0.1 TDB X Lib Dashboard X TDB X Platform Team pushData Agents X pushData Alerting X sendAlerts view

Monitoring stack X is now up and running

slide-13
SLIDE 13

Classical Approach (4)

13

automated action manual action Product Team Alfa Service Alfa v0.1 TDB X Lib Platform Team Agents Y learn, configure, deploy Monitoring Tool Y

The Platform Team decides to migrate to monitoring stack Y

slide-14
SLIDE 14

Classical Approach (5)

14

automated action manual action Product Team Alfa Platform Team Agents Y Monitoring Tool Y Service Alfa v0.2 AgY Lib learn, configure learn, code, release

Product Teams have to learn how to use Y and re- instrument their code

slide-15
SLIDE 15

Classical Approach (6)

15

automated action manual action Product Team Alfa Platform Team Agents Y Monitoring Tool Y Service Alfa v0.2 AgY Lib view pullData pushData sendAlerts

Monitoring stack Y is now up and running

slide-16
SLIDE 16

Omnia-based approach (1)

16

automated action manual action

Product Team Alfa Service Alfa v0.1 Omnia Lib Platform Team learn, code, release

  • mnia.yml
  • mnia.admin.yml

v1 learn, code

By using Omnia a single learning step is required for all teams

slide-17
SLIDE 17

Omnia-based approach (2)

17

automated action manual action

Product Team Alfa Service Alfa v0.1 Omnia Lib Dashboard X TDB X Platform Team Agents X Alerting X Omnia CLI

  • mnia.yml
  • mnia.admin.yml

v1 use parse parse configure, deploy

Omnia deploys and configures the monitoring stack X from code

slide-18
SLIDE 18

Omnia-based approach (3)

18

automated action manual action

Product Team Alfa Service Alfa v0.1 Omnia Lib Dashboard X TDB X Platform Team view pushData Agents X pushData Alerting X sendAlerts Omnia CLI

  • mnia.yml
  • mnia.admin.yml

v1

Monitoring stack X is now up and running

slide-19
SLIDE 19

Omnia-based approach (4)

19

automated action manual action

Product Team Alfa Service Alfa v0.1 Omnia Lib Dashboard X TDB X Platform Team Agents X Alerting X Omnia CLI

  • mnia.yml
  • mnia.admin.yml

v2 code

Migration to a new monitoring stack is a simple code change made by the Platform Team

slide-20
SLIDE 20

Omnia-based approach (5)

20

automated action manual action

Product Team Alfa Service Alfa v0.1 Omnia Lib Platform Team Omnia CLI

  • mnia.yml
  • mnia.admin.yml

v2 Monitoring Tool Y Agents Y use parse configure, deploy parse

Omnia takes care of reconfiguring and redeploying the stack

slide-21
SLIDE 21

Omnia-based approach (6)

21

automated action manual action

Product Team Alfa Service Alfa v0.1 Omnia Lib Platform Team Omnia CLI

  • mnia.yml
  • mnia.admin.yml

v2 Monitoring Tool Y Agents Y view sendAlerts pushData pullData

Monitoring stack Y is now up and running The Product Teams did not have to change a line

slide-22
SLIDE 22

Omnia Libs

  • Requirements:
  • independent of the monitoring stack
  • minimize instrumentation overhead
  • Example:

22 Omnia Lib for Spring Boot

slide-23
SLIDE 23

Under the hood

  • Convention over configuration:
  • statsd (w/ influxdb tag ext.) protocol
  • automated meta-data decoration
  • default endpoint
  • Example:

23

Spring Boot Omnia Lib

POST “payments,host=vm1,service=payservice 34” http://collector:8125 payments 34

slide-24
SLIDE 24

Omnia.yml: monitoring config as code

  • Requirements
  • independent of the monitoring stack
  • versionable with the code
  • Example:

24 dashboard.tmpl Go Template Library dashboard

slide-25
SLIDE 25

The Omnia vocabulary

  • and for metrics

25

  • Shared vocabulary for resources
slide-26
SLIDE 26

Monitoring infra as code

26

  • mnia.admin.yml example:

v1 v2 Reusing existing monitoring tools Reusing existing adapters Reusing existing provisioning tools

slide-27
SLIDE 27

Conclusion

  • Contributions
  • Reduce entry cost and learning curve to monitoring
  • Application of DevOps practices to monitoring
  • Attempt of standardization
  • Threats to validity
  • A common interface may simplify tools characteristics
  • However:
  • initial approach to monitoring has simple requirements
  • it could push tool vendors to implement missing features
  • the approach could be applied to existing or new tools
  • Future work
  • Extensive evaluation using real world examples

27