Monitor with the Stack Philipp Krenn @xeraa 1 Infrastructure | - - PowerPoint PPT Presentation

▶
monitor
SMART_READER_LITE
LIVE PREVIEW

Monitor with the Stack Philipp Krenn @xeraa 1 Infrastructure | - - PowerPoint PPT Presentation

Jan 23, 2024 218 likes •1.08k views

Monitor with the Stack Philipp Krenn @xeraa 1 Infrastructure | Developer Advocate 2 Disclaimer This is not a training https://www.elastic.co/training 3 Who Is Using Elasticsearch Logstash and Kibana Beats 4 5 6 7

slide-1
SLIDE 1

Monitor

with the Stack

Philipp Krenn@xeraa

1
slide-2
SLIDE 2

Infrastructure | Developer Advocate

2
slide-3
SLIDE 3

Disclaimer

This is not a training

https://www.elastic.co/training

3
slide-4
SLIDE 4

Who Is Using

Elasticsearch Logstash and Kibana Beats

4
slide-5
SLIDE 5 5
slide-6
SLIDE 6 6
slide-7
SLIDE 7 7
slide-8
SLIDE 8

Starting Point

https://github.com/xeraa/mongodb-monitoring

8
slide-9
SLIDE 9

USB Sticks

9
slide-10
SLIDE 10

Box

Vagrant Ansible Provisioner

10
slide-11
SLIDE 11

Credentials vagrant & vagrant

11
slide-12
SLIDE 12

SSH

$ ssh vagrant@127.0.0.1 -p 2222 -o PreferredAuthentications=password Windows: http://www.putty.org

12
slide-13
SLIDE 13

Ansible

$ cd /elastic-stack/ $ ls

13
slide-14
SLIDE 14 14
slide-15
SLIDE 15

REST

$ curl -XGET -u "elastic:changeme" http://localhost:9200/

15
slide-16
SLIDE 16 16
slide-17
SLIDE 17

Login

http://localhost:5601 elastic & changeme

17
slide-18
SLIDE 18 18
slide-19
SLIDE 19

Filebeat

19
slide-20
SLIDE 20

Filebeat Modules

20
slide-21
SLIDE 21

System Dashboards

21
slide-22
SLIDE 22 22
slide-23
SLIDE 23

MongoDB Logs

/var/log/mongodb/mongod.log

23
slide-24
SLIDE 24

/etc/filebeat/filebeat.yml

filebeat.prospectors:

  • input_type: log

paths:

  • /var/log/mongodb/mongod.log

document_type: mongodb

24
slide-25
SLIDE 25

PS: Multiline Logs

25
slide-26
SLIDE 26

/etc/filebeat/filebeat.yml

filebeat.prospectors:

  • input_type: log

paths:

  • /var/log/java-app/*.log

document_type: java multiline.pattern: '^[[:space:]]' multiline.negate: false multiline.match: after

26
slide-27
SLIDE 27

Test

$ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20

27
slide-28
SLIDE 28

Kibana Discover

Limit Kibana view to the mongodb type

28
slide-29
SLIDE 29 29
slide-30
SLIDE 30

Grok Patterns

https://github.com/logstash-plugins/logstash-patterns- core/blob/master/patterns/grok-patterns

30
slide-31
SLIDE 31

MongoDB Patterns

https://github.com/logstash-plugins/logstash-patterns- core/blob/master/patterns/mongodb

31
slide-32
SLIDE 32

Building Patterns

Grokdebug in Kibana 5.5+ (X-Pack Basic)

https://grokdebug.herokuapp.com

32
slide-33
SLIDE 33 33
slide-34
SLIDE 34

/etc/filebeat/filebeat.yml

  • utput.elasticsearch:

hosts: ["localhost:9200"] username: "{{ elastic_user }}" password: "{{ elastic_password }}" pipeline: "mongodb_log"

34
slide-35
SLIDE 35

Console

Pattern in a single line

PUT _ingest/pipeline/mongodb_log { "description": "Ingest pipeline for MongoDB logs", "processors": [ { "grok": { "field": "message", "patterns": [ "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{MONGO3_SEVERITY:mongodb.severity} %{SPACE}%{MONGO3_COMPONENT:mongodb.component}%{SPACE} (?:[%{DATA:mongodb.context}])?%{SPACE}%{GREEDYDATA:message}" ] } } ] } 35
slide-36
SLIDE 36

Filebeat Restart

$ sudo service filebeat restart

36
slide-37
SLIDE 37

Test

$ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20

37
slide-38
SLIDE 38

Refresh Field List

Management → Index Patterns → filebeat-* → Refresh field list

38
slide-39
SLIDE 39

Visualize

mongodb.component of log events

39
slide-40
SLIDE 40 40
slide-41
SLIDE 41 41
slide-42
SLIDE 42

Fetch the Slow Log

42
slide-43
SLIDE 43

/etc/logstash/conf.d/00-mongodb- input.conf

input { mongodb { uri => 'mongodb://127.0.0.1:27017/POCDB' placeholder_db_dir => '/var/local/logstash-mongodb/' collection => 'system.profile' batch_size => 500 generateId => true } }

43
slide-44
SLIDE 44

/etc/logstash/conf.d/20-elasticsearch-

  • utput.conf
  • utput {

elasticsearch { hosts => ["localhost:9200"] manage_template => false index => "mongodb-%{+YYYY.MM.dd}" document_type => "slowlog" user => "{{ elastic_user }}" password => "{{ elastic_password }}" } }

44
slide-45
SLIDE 45

Logstash Restart

$ sudo service logstash restart

45
slide-46
SLIDE 46

Debug Logstash

$ less /var/log/logstash/logstash-plain.log

46
slide-47
SLIDE 47

Doesn't Work

Getting data from system collections

https://github.com/phutchins/logstash-input-mongodb/issues/8

47
slide-48
SLIDE 48

Extension Idea

Build an exec filter to run slow queries against MongoDB's .explain()

48
slide-49
SLIDE 49

Metricbeat

49
slide-50
SLIDE 50

Metricbeat System

50
slide-51
SLIDE 51

Metricbeat Service

51
slide-52
SLIDE 52

/etc/metricbeat/metricbeat.yml

  • module: mongodb

metricsets: ["dbstats", "status"] hosts: ["localhost:27017"]

52
slide-53
SLIDE 53

Metricbeat Restart

$ sudo service metricbeat restart

53
slide-54
SLIDE 54

Test

$ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20

54
slide-55
SLIDE 55 55
slide-56
SLIDE 56

Visual Builder

mongodb.status.network.in.bytes vs mongodb.status.network.out.bytes

56
slide-57
SLIDE 57 57
slide-58
SLIDE 58

Packetbeat

58
slide-59
SLIDE 59

Protocols

59
slide-60
SLIDE 60

Flows

Application layer: Unsupported / encrypted (TLS) protocols IP / TCP / UDP Number of packets & bytes Retransmissions Temporal flow

60
slide-61
SLIDE 61

/etc/packetbeat/packetbeat.yml

packetbeat.protocols.mongodb: ports: [27017]

61
slide-62
SLIDE 62

Packetbeat Restart

$ sudo service packetbeat restart

62
slide-63
SLIDE 63

Test

$ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20

63
slide-64
SLIDE 64 64
slide-65
SLIDE 65

Heartbeat

65
slide-66
SLIDE 66

Heartbeat

ICMP, TCP, HTTP, HTTPS

66
slide-67
SLIDE 67

/etc/heartbeat/heartbeat.yml

heartbeat.monitors:

  • type: tcp

hosts: ["127.0.0.1:27017"] schedule: '@every 10s'

67
slide-68
SLIDE 68

Heartbeat Restart

$ sudo service heartbeat restart

68
slide-69
SLIDE 69

Test

$ sudo service mongod stop $ sudo service mongod start

69
slide-70
SLIDE 70

Visualize

Up or down and TCP response times

70
slide-71
SLIDE 71 71
slide-72
SLIDE 72 72
slide-73
SLIDE 73

Dashboard

Combining visualizations

73
slide-74
SLIDE 74 74
slide-75
SLIDE 75

Winlogbeat

75
slide-76
SLIDE 76

libbeat

https://github.com/elastic/beats/tree/master/generate/beat

76
slide-77
SLIDE 77 77
slide-78
SLIDE 78

X-Pack

Security Monitoring Graph Reporting Alerting Machine Learning

78
slide-79
SLIDE 79

X-Pack Basic

79
slide-80
SLIDE 80

Conclusion

80
slide-81
SLIDE 81 81
slide-82
SLIDE 82 82
slide-83
SLIDE 83 83
slide-84
SLIDE 84

Thanks!

Questions?

Philipp Krenn@xeraa

84