Monitor your containers with the Elastic Stack Monica Sarbu
Monica Sarbu Team lead, Beats team monica@elastic.co @monicasarbu 3
Monitor your containers with the Elastic Stack
Elastic Stack @monicasarbu 5
Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch
Multiple data types, one place m e t r i c s e r l o g s • D o c k e r • D o c k a c t i o n s • M y S Q L t r a n s • A p a c h e l o g s o g s • R e d i s l • f l o w s m e t r i c s • D o c k e r O • d i s k I c s • R e d i s m e t r i • C P U % •memory % s a c t i o n s • H T T P t r a n •flows g s • M y S Q L l o •Redis transactions •filesystem @monicasarbu 7
Central point for your distributed infrastructure @monicasarbu 8
The Beats 30+ other community Beats shipping @monicasarbu 9
Filebeat 10
Filebeat • Tails log files, without parsing them • “At least once” guarantees, handles backpressure • Extra powers: • Multiline • JSON logs • Filtering 11
Parse log lines with Ingest Node I N G E S T @monicasarbu 12
Parse log lines with Logstash I N G E S T @monicasarbu 13
Filebeat Back pressure handling 14
Why back-pressure is key? @monicasarbu 15
Synchronous sending registry file acked read read stream of log lines batch of messages ack @monicasarbu 16
This means.. • Filebeat adapts its speed automatically to as much as the next stage can process • But: be aware when benchmarking 17
When the next stage is down.. • Filebeat patiently waits • Log lines are not lost • It doesn’t allocate memory, it doesn’t buffer things on disk 18
Filebeat Collect container logs 19
Docker logging drivers https://docs.docker.com/engine/admin/logging/overview/ @monicasarbu 20
Centralize Docker logs: option 1/522 • Use the Docker gelf driver and the Logstash-gelf-input • Pros: • No shipper to install, send directly to Logstash • Cons: • UDP based, no delivery guarantees, no congestion control @monicasarbu 21
Centralize Docker logs: option 2/522 • Use the Docker JSON driver , use Filebeat with the JSON support • Pros: • Simple (default driver) • Easy to add container metadata (name, labels, etc.) • `docker logs` works • Cons: • JSON driver can slow down Docker @monicasarbu 22
Centralize Docker logs: option 3/522 • Use the Docker syslog driver, and a local syslog server , then Filebeat for shipping • Pros: • Good control over the path where the files are written, rotation strategies, etc. • Cons: • you need to manage the syslog server • metadata is serialized as string, needs to be de- serialized again (opportunity for mistakes) • multiline is difficult because data from containers can be mixed @monicasarbu 23
Centralize Docker logs: option 4/522 • Use the Docker journald driver then Filebeat for shipping • Pros: • journald is often already available • convenient support for metadata • `docker logs` works • Cons: • Filebeat doesn’t yet support journald (a Journalbeat exists, however) @monicasarbu 24
Centralize Docker logs: option 5/522 • Mount a volume and have your app write logs into the volume • Pros: • If your app can rotate it’s own logs, it’s very easy to setup • Scales well • Cons: • Difficult to pass metadata @monicasarbu 25
Centralize Docker logs: conclusion • json driver, syslog driver, and shared volume are pretty good options today • journald driver might be better options in the future @monicasarbu 26
Metricbeat new in 5.0 27
One Metricbeat module for each service + Add your own @monicasarbu 28
Metricbeat system module CPU diskIO Mem filesystem network cores load processes @monicasarbu 29
Metricbeat Collect container metrics 30
in progress Querying the Docker API • Dedicated Docker module • Has access to container names and labels • Easy to setup • Offers: • CPU and memory • Docker container information • network (in/out bytes, dropped) • diskIO (reads/writes) • status of containers (# of stopped, running, etc) @monicasarbu 31
Reading cgroup data from /proc/ • Doesn’t require access to the Docker API (can be a security issue) • Works for any container runtime (Docker, rkt, runC, LXD, etc.) • Part of the system module • Automatically enhances process data with cgroup information • Cannot get the container name and labels @monicasarbu 32
Run as a container App1 App2 App3 Host @monicasarbu 33
Elasticsearch as time series DB 34
Elasticsearch BKD trees • Added for Geo-points • faster to index #velo • faster to query • more disk-efficient • more memory efficient @monicasarbu 35
Float values On Disk Usage in kb 80000 • half floats 70000 60000 • scaled floats (using a scaling factor) - great for 50000 things like percentage 40000 points 30000 20000 10000 0 float half float scaled float scaled float (factor = 4000) (factor = 100) Points disk usage (kb) docs_values disk usage (kb) @monicasarbu 36
Why Elasticsearch for time series • Horizontal scalability. Mature and battle tested cluster support. • Flexible aggregations (incl moving averages & Holt Winters) #velo • One system for both logs and metrics • Timelion UI, Grafana • Great ecosystem: e.g. alerting tools @monicasarbu 37
Packetbeat 38
Supported traffic decoders http:// Thrift DNS + ICMP AMQP Add your own @monicasarbu 39
Unknown traffic, use flows •Look into data for which we don’t understand the application layer protocol •TLS •Protocols we don’t yet support •Get data about IP / TCP / UDP layers •number of packets & bytes •retransmissions •inter-arrival time @monicasarbu 40
Packetbeat Monitor traffic exchanged between your containers 41
Monitor outside containers App1 App2 App3 Packetbeat Host traffic exchanged between your containers @monicasarbu 42
Demo: Metricbeat, Filebeat, Packetbeat Multiple data types, one view in Kibana 43
Thank you • github.com/elastic/beats • discuss.elastic.co • @elastic #elasticbeats • #beats on freenode 44
Recommend
More recommend
