Styx Exporting Data from Prometheus, for Science! About me - PowerPoint PPT Presentation

Apr 15, 2023 •1.22k likes •1.42k views

Styx Exporting Data from Prometheus, for Science! About me Matthias Loibl I am software developer and computer science student at TU Berlin Employee at JustWatch (SRE, Go) I love working on Distributed Systems with Go , Docker ,

Styx Exporting Data from Prometheus, for Science!
About me Matthias Loibl I am software developer and computer science student at TU Berlin ● Employee at JustWatch (SRE, Go) ● ● I love working on Distributed Systems with Go , Docker , Kubernetes , and Prometheus Creator of gopass ●
Why do we need Styx?
What is pluto? ● IMAP server on planetary-scale , research prototype at TU Berlin ● Synchronize mailbox state across replicas world-wide Consistency via Conflict-free Replicated Data Types (CRDTs) ● Write a paper ●
How we run pluto ● Running on federated Kubernetes clusters in EU & US ● Each cluster is running its own Prometheus Metrics tell us how fast we're consistent ○
Why Prometheus? ● Prometheus awesome for insights to our software ○ pluto is written in Go → easy to integrate metrics Monitor our infrastructure ● ● View graphs with grafana in real time
Why not just Grafana? Why Styx? ● Screenshots aren't professional for scientific papers ● Export the data from Prometheus as a backup Edit the graphs in a friendly manner later on → gnuplot & matplotlib ●
Styx ● Written in Go, thus single binary go get github.com/go-pluto/styx ○ Optional dependencies ● ○ gnuplot matplotlib ○ Talks to Prometheus via API endpoint /api/v1/query_range ● Styx is also one of the pluto moons, thus the name ●
CSV $ styx 'go_goroutines' $ styx --duration 5m 'go_goroutines' $ styx --prometheus http://prom.example.com 'go_goroutines' Exports simple .csv file to further utilization in Excel, Google Spreadsheet…
gnuplot ● Code Generation! ● Generate a .gnuplot file with all gnuplot commands and the data. Self contained and reproducible graphs outside of Prometheus ○
matplotlib ● Code Generation! ● Generate a .py file with all matplotlib commands and the data. Self contained and reproducible graphs outside of Prometheus ○
DEMO
More Prometheus Tools
Prom-Metric-Viewer https://github.com/metalmatze/prom-metric-viewer
Exporters ● justwatchcom/elasticsearch_exporter ● justwatchcom/sql_exporter metalmatze/digitalocean_exporter ● metalmatze/githubql_exporter ● ● metalmatze/transmission-exporter
Contact Twitter @metalmatze GitHub metalmatze Website matthiasloibl.com
Thanks github.com/go-pluto/styx

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

HADES AND THE RIVER STYX By:John Dougherty & Ahmar Ross HADES AND THE RIVER STYX Page:2 The

HADES AND THE RIVER STYX By:John Dougherty & Ahmar Ross HADES AND THE RIVER STYX Page:2 The river Styx separated the living and the dead.Whoever died their soul went to the underworld with hades. HADES AND THE RIVER Page:3 When your

206 views • 5 slides

HADES AND THE RIVER STYX. Lejon Holmes and Avontae dent.

HADES AND THE RIVER STYX. Lejon Holmes and Avontae dent. ---------------------------------------------- Payment or no payment. If your family had not buried you with a coin, to use as payment, you were stuck on the land of living.

400 views • 5 slides

STYX: Stream Processing with Trustworthy Cloud-based Execution Julian Stephen, Savvas Savvides,

STYX: Stream Processing with Trustworthy Cloud-based Execution Julian Stephen, Savvas Savvides, Vinaitheerthan Sundaram, Masoud Saeida Ardekani, Patrick Eugster October 6, 2016 Purdue University Table of contents 1. Overview 2. Ensuring

754 views • 56 slides

dmabuf Nikhil Devshatwar Texas Instruments, India 2 About author Embedded Linux developer

1 Exporting virtual memory as dmabuf Nikhil Devshatwar Texas Instruments, India 2 About author Embedded Linux developer Texas Instruments (Bangalore, India) Key work areas Video subsystem drivers Camera drivers Base

1.09k views • 17 slides

US Policy Spillover (?) Chinas Accession to the WTO and Rising Exports to the EU Karsten Mau

US Policy Spillover (?) Chinas Accession to the WTO and Rising Exports to the EU Karsten Mau Brown Bag Lunch Seminar in Economic Theory Bielefeld, 19 June 2015 Research Question and Hypothesis Why did Chinas EU exports rise so fast after

525 views • 26 slides

Exp xport Div iversif ification, Margin ins and Economic ic Growth at In Industrial Le

Exp xport Div iversif ification, Margin ins and Economic ic Growth at In Industrial Le Level: Evid idence fr from Th Thaila land Juthathip Jongwanich Faculty of Economics and Cluster of International Competitiveness Thammasat

289 views • 24 slides

Establishing the Nuclear Safety Infrastructure in the United Arab Emirates Ian Grant Director of

Establishing the Nuclear Safety Infrastructure in the United Arab Emirates Ian Grant Director of Nuclear Safety September 2010 Presentation Outline Introduction UAE Policy on Peaceful Uses of Nuclear Energy UAE Legislative and

500 views • 33 slides

IPFIX Mediation: Framework IPFIX IETF-77 March 23, 2010 draft-ietf-ipfix-mediators-framework-05

IPFIX Mediation: Framework IPFIX IETF-77 March 23, 2010 draft-ietf-ipfix-mediators-framework-05 Atsushi Kobayashi, Benoit Claise, Gerhard Munz, Keisuke Ishibashi 1 History Submitted -04 version on October 2009. Received comments from

332 views • 8 slides

with WebAssembly Boyan Mihaylov @boyanio boyan.io https://github.com/boyanio/wasm-class/

In-depth hands-on experience with WebAssembly Boyan Mihaylov @boyanio boyan.io https://github.com/boyanio/wasm-class/ @boyanio WebAssembly ( WASM ) is compiler target for programs on the Web @boyanio function add(a, b) { return a + b; }

962 views • 41 slides

Binding ex post facto Patrick D. Elliott x Yasu Sudo y November 12, 2018 LENLS, Keio University

Binding ex post facto Patrick D. Elliott x Yasu Sudo y November 12, 2018 LENLS, Keio University Leibniz-Center for General Linguistics x and University College London y Tiis is joint work with Yasu Sudo (University College London). 1 Overview

891 views • 51 slides

Best of Both Worlds: Ex-Ante and Ex-Post Fairness in Resource Allocation Nisarg Shah University

Best of Both Worlds: Ex-Ante and Ex-Post Fairness in Resource Allocation Nisarg Shah University of Toronto Joint work with Rupert Freeman (MSR NYC, now at UVA) & Rohit Vaish (RPI) Implementing Fractional Allo Im llocations 1 1

295 views • 4 slides

Automated Design of Robust Mechanisms Michael Albert 1 , Vincent Conitzer 1 , Peter Stone 2 1 Duke

Automated Design of Robust Mechanisms Michael Albert 1 , Vincent Conitzer 1 , Peter Stone 2 1 Duke University, 2 University of Texas at Austin 3rd Workshop on Algorithmic Game Theory and Data Science June 26th, 2017 Previously appeared in AAAI17

771 views • 54 slides

s trss rs

s trss rs Prtr s P tr trss st

667 views • 39 slides

Bayesian Games CMPUT 654: Modelling Human Strategic Behaviour S&LB 6.3 Lecture Outline

Bayesian Games CMPUT 654: Modelling Human Strategic Behaviour S&LB 6.3 Lecture Outline 1. Recap 2. Bayesian Game Definitions 3. Strategies and Expected Utility 4. Bayes-Nash Equilibrium Recap: Repeated Games A repeated game

728 views • 23 slides

EC476 Contracts and Organizations, Part III: Lecture 4 Leonardo Felli 32L.G.06 2 February 2015

EC476 Contracts and Organizations, Part III: Lecture 4 Leonardo Felli 32L.G.06 2 February 2015 Benjamin Cardozo 1921 Law never is , but is always about to be. It is realized only when embodied in a judgment, and in being realized,

621 views • 47 slides

Text 1. A text is a sequence of characters 2. Each character is taken from a finite alphabete

Computer Architecture applied computer science 02.03 Representation of non-numerical sets urbino worldwide campus 02 Information theory 02.03 Representation of non-numerical sets Texts Images Signals (Audio/Video)

420 views • 6 slides

Input Input devices Text entry Positional input Input Devices 1 MacBook Wheel (The Onion) -

Input Input devices Text entry Positional input Input Devices 1 MacBook Wheel (The Onion) - https://youtu.be/9BnLbv6QYcA - https://www.student.cs.uwaterloo.ca/~cs349/videos/macbook_wheel.mp4 Input Devices 2 iPod Wheel Input Devices 3

759 views • 32 slides

Introduction to Programming Bertrand Meyer Last revised 1 December 2003 I ntroduction to

1 Introduction to Programming Bertrand Meyer Last revised 1 December 2003 I ntroduction to Programming Lecture 15 Chair of Softw are Engineering 2 Lecture 15: A glimpse into the hardware I ntroduction to Programming Lecture 15

1k views • 20 slides

7. International character sets Default character set: Unicode Characters correspond to

7. International character sets Default character set: Unicode Characters correspond to numbers . Different encodings exist for these numbers. In Web servers HTTP metadata specifies the coding (so called MIME media type), e.g.

126 views • 12 slides

upTEX Unicode version of pTEX with CJK extensions Takuji Tanaka upTEX project

upTEX Unicode version of pTEX with CJK extensions Takuji Tanaka upTEX project Oct 26, 2013 Takuji Tanaka (upTEX project) upTEX Unicode version of pTEX with CJK extensions Oct 26, 2013 1 / 42 Outline /

929 views • 34 slides

Basic Types in C Dalhousie University Winter 2019 Basic Types in C Integer types: Floating

CSCI 2132: Software Development Norbert Zeh Faculty of Computer Science Basic Types in C Dalhousie University Winter 2019 Basic Types in C Integer types: Floating point types: int float (32 bits) Most long (64 bits) double

505 views • 27 slides

Outline Crypto intro Computer Security: Secret Key Crypto Symmetric crypto Bart Jacobs

Crypto intro Crypto intro Symmetric crypto Symmetric crypto Achieving security goals with symmetric crypto Radboud University Nijmegen Achieving security goals with symmetric crypto Radboud University Nijmegen e-Passport example

269 views • 7 slides

Introduction II (extended) Radu Nicolescu Department of Computer Science University of Auckland

A#1R Drive Lambdas Frs Read Err Args Introduction II (extended) Radu Nicolescu Department of Computer Science University of Auckland 19 July 2018 1 / 26 A#1R Drive Lambdas Frs Read Err Args 1 Assignment #1R 2 Driving FP 3 Lambdas

680 views • 37 slides

Multimedia Communications Spring 2006-07 Session Level Advances (SDP RFC 2327, SAP, SIP

CS 584 / CMPE 584 Multimedia Communications Spring 2006-07 Session Level Advances (SDP RFC 2327, SAP, SIP RFC 2543) Shahab Baqai LUMS SDP In the context of a multimedia multicast Internet, a session directory tool is used to:

609 views • 49 slides