Discoverable Metadata for System Monitoring Data S. Leak, A. - - PowerPoint PPT Presentation

▶
discoverable metadata for system monitoring data
SMART_READER_LITE
LIVE PREVIEW

Discoverable Metadata for System Monitoring Data S. Leak, A. - - PowerPoint PPT Presentation

Sep 28, 2023 177 likes •270 views

Discoverable Metadata for System Monitoring Data S. Leak, A. Greiner, A. Gentile, J. Brant Context and Premise Much log data being collected, hard to coordinate who is collecting what, where, how DOE Resilience Project: how to make log

slide-1
SLIDE 1

Discoverable Metadata for System Monitoring Data

  • S. Leak, A. Greiner, A. Gentile, J. Brant
slide-2
SLIDE 2

Context and Premise

  • Much log data being collected, hard to

coordinate who is collecting what, where, how

  • DOE Resilience Project: how to make log data

navigable for investigating fault propagation

  • Idea: Machine-readable metadata to bypass

many of the problems

slide-3
SLIDE 3

Linked data connecting disparate independent metadata into a single global query-able graph

slide-4
SLIDE 4

(Contrived) Example

What might have caused this unexpected job failure around 8am

  • n 2015-03-27?

./ ./logs.py logs.py -q ' q ' SELECT ?title ?contact ?page WHERE { SELECT ?title ?contact ?page WHERE { ?logset logset a a logset:LogSet logset:LogSet . ?logset logset dct:title dct:title ?title . ?title . ?logset logset dcat:contactPoint dcat:contactPoint ?contact . ?contact . ?logset logset dcat:landingPage dcat:landingPage ?page . ?page . ?logset logset dct:temporal dct:temporal/logset:startDate logset:startDate ?start . ?start . ?logset logset dct:temporal dct:temporal/logset:endDate logset:endDate ?end . ?end . FILTER (?start < "2015 FILTER (?start < "2015-03 03-27T07:00:00"^^ 27T07:00:00"^^xsd:dateTime xsd:dateTime) . ) . FILTER (?end > "2015 FILTER (?end > "2015-03 03-27T09:00:00"^^ 27T09:00:00"^^xsd:dateTime xsd:dateTime) . ) . snl:mutrino snl:mutrino (logset:hasPart|logset:affects logset:hasPart|logset:affects)* ?thing )* ?thing . ?logset logset logset:isViewOf logset:isViewOf*/ */dcat:distribution dcat:distribution*/ */logset:subject logset:subject ?thing . ?thing . }

slide-5
SLIDE 5

Query returns a few things

Someone has log data covering that! Better still: annotations (smaller/easier to download)

['" ['"annotations annotations for for Mutrino Mutrino dataset release ... dataset release ... ['" ['"annotations annotations for for Mutrino Mutrino dataset release ... dataset release ... ['" ['"annotations annotations for for Mutrino Mutrino dataset release ... dataset release ... ['" ['"Mutrino Mutrino dataset release SAND2016 dataset release SAND2016-2449 ... 2449 ... ['" ['"Mutrino Mutrino dataset release SAND2016 dataset release SAND2016-2449 ... 2449 ... ['" ['"Mutrino Mutrino dataset release SAND2016 dataset release SAND2016-2449 ... 2449 ...

slide-6
SLIDE 6

Download and query the Annotations

./ ./anno.py anno.py -s 2015 s 2015-03 03-27T07:00:00 27T07:00:00 -e 2015 e 2015-03 03-27T09:00:00 27T09:00:00

  • [772743] by abc on system None

Time: 2015-03-27 07:52:33 to 2015-03-27 07:52:33 Start state: None ; End state: None Description: Machine check events (MCE) logged. This occurs when the system processor detected a hardware error. Manually invoked? False ; System down?: False Components: ["c0-0c1s8n1"] Tags: LogDiver category group: NO Baler pattern ID: 3801 Relevant log files: console

  • [772744] by abc on system None

Time: 2015-03-27 08:15:33 to 2015-03-27 08:15:33 [snip] [snip]

slide-7
SLIDE 7

Progress and Challenges

  • RDF vocabulary and sqlite3 schema for

annotations

  • Some prototype tools for working with them
  • Annotated a sample of logs
slide-8
SLIDE 8

Progress and Challenges

  • Graph queries are really powerful

– .. But can be really slow too!

  • Impedance mismatch between formats

– Can be mitigated, but still laborious

  • Some of the tools for RDF/Linked Data can be

a little clunky