Monitor your containers with the Elastic Stack Monica Sarbu Monica - - PowerPoint PPT Presentation

Monitor your containers with the Elastic Stack

Monica Sarbu

3

Monica Sarbu

Team lead, Beats team monica@elastic.co @monicasarbu

Monitor your containers with the Elastic Stack

@monicasarbu

Elastic Stack

5

Beats are lightweight shippers that collect and ship all kinds of

• perational data to Elasticsearch

@monicasarbu

Multiple data types, one place

7

• D
• c

k e r m e t r i c s

• f

l

• w

s

• M

y S Q L l

• g

s

• d

i s k I O

• H

T T P t r a n s a c t i

• n

s

• M

y S Q L t r a n s a c t i

• n

s

• R

e d i s m e t r i c s

• A

p a c h e l

• g

s

• R

e d i s l

• g

s

• C

P U %

• D
• c

k e r m e t r i c s

• D
• c

k e r l

• g

s

• memory %
• filesystem
• Redis transactions
• flows

@monicasarbu

Central point for your distributed infrastructure

8

@monicasarbu

The Beats

9

30+ other community Beats shipping

Filebeat

10

• Tails log files, without parsing

them

• “At least once” guarantees,

handles backpressure

• Extra powers:
• Multiline
• JSON logs
• Filtering

11

Filebeat

@monicasarbu

Parse log lines with Ingest Node

12

I N G E S T

@monicasarbu

Parse log lines with Logstash

13

I N G E S T

Filebeat Back pressure handling

14

@monicasarbu

Why back-pressure is key?

15

@monicasarbu

Synchronous sending

16

batch of messages ack stream of log lines read read acked registry file

• Filebeat adapts its speed

automatically to as much as the next stage can process

• But: be aware when benchmarking

17

This means..

• Filebeat patiently waits
• Log lines are not lost
• It doesn’t allocate memory, it

doesn’t buffer things on disk

18

When the next stage is down..

Filebeat Collect container logs

19

@monicasarbu

Docker logging drivers

20

https://docs.docker.com/engine/admin/logging/overview/

@monicasarbu

Centralize Docker logs: option 1/522

• Use the Docker gelf driver and the Logstash-gelf-input
• Pros:
• No shipper to install, send directly to Logstash
• Cons:
• UDP based, no delivery guarantees, no congestion

control

21

@monicasarbu

Centralize Docker logs: option 2/522

• Use the Docker JSON driver, use Filebeat with the

JSON support

• Pros:
• Simple (default driver)
• Easy to add container metadata (name, labels, etc.)
• `docker logs` works
• Cons:
• JSON driver can slow down Docker

22

@monicasarbu

Centralize Docker logs: option 3/522

• Use the Docker syslog driver, and a local syslog

server, then Filebeat for shipping

• Pros:
• Good control over the path where the files are

written, rotation strategies, etc.

• Cons:
• you need to manage the syslog server
• metadata is serialized as string, needs to be de-

serialized again (opportunity for mistakes)

• multiline is difficult because data from containers

can be mixed

23

@monicasarbu

Centralize Docker logs: option 4/522

• Use the Docker journald driver then Filebeat for

shipping

• Pros:
• journald is often already available
• convenient support for metadata
• `docker logs` works
• Cons:
• Filebeat doesn’t yet support journald (a Journalbeat

exists, however)

24

@monicasarbu

Centralize Docker logs: option 5/522

• Mount a volume and have your app write logs into the

volume

• Pros:
• If your app can rotate it’s own logs, it’s very easy to

setup

• Scales well
• Cons:
• Difficult to pass metadata

25

@monicasarbu

Centralize Docker logs: conclusion

• json driver, syslog driver, and shared

volume are pretty good options today

• journald driver might be better options in

the future

26

Metricbeat

27

new in 5.0

@monicasarbu

One Metricbeat module for each service

28

+

Add your own

@monicasarbu

Metricbeat system module

29

CPU Mem diskIO filesystem processes load network cores

Metricbeat Collect container metrics

30

@monicasarbu

Querying the Docker API

• Dedicated Docker module
• Has access to container names and labels
• Easy to setup
• Offers:
• CPU and memory
• Docker container information
• network (in/out bytes, dropped)
• diskIO (reads/writes)
• status of containers (# of stopped, running, etc)

31

in progress

@monicasarbu

Reading cgroup data from /proc/

• Doesn’t require access to the Docker API

(can be a security issue)

• Works for any container runtime (Docker,

rkt, runC, LXD, etc.)

• Part of the system module
• Automatically enhances process data with

cgroup information

• Cannot get the container name and labels

32

@monicasarbu

Run as a container

33

App1 App2 App3 Host

34

Elasticsearch as time series DB

#velo @monicasarbu

Elasticsearch BKD trees

35

• Added for Geo-points
• faster to index
• faster to query
• more disk-efficient
• more memory efficient

@monicasarbu

10000 20000 30000 40000 50000 60000 70000 80000

float half float scaled float (factor = 4000) scaled float (factor = 100)

On Disk Usage in kb

Points disk usage (kb) docs_values disk usage (kb)

Float values

36

• half floats
• scaled floats (using a

scaling factor) - great for things like percentage points

#velo @monicasarbu

Why Elasticsearch for time series

• Horizontal scalability. Mature and

battle tested cluster support.

• Flexible aggregations (incl moving

averages & Holt Winters)

• One system for both logs and

metrics

• Timelion UI, Grafana
• Great ecosystem: e.g. alerting

tools

37

Packetbeat

38

@monicasarbu

Supported traffic decoders

39

+

Add your own http:// Thrift DNS ICMP AMQP

@monicasarbu

Unknown traffic, use flows

• Look into data for which we don’t

understand the application layer protocol

• TLS
• Protocols we don’t yet support
• Get data about IP / TCP / UDP layers
• number of packets & bytes
• retransmissions
• inter-arrival time

40

Packetbeat Monitor traffic exchanged between your containers

41

@monicasarbu

Monitor outside containers

42

App1 Host App2 App3 Packetbeat

traffic exchanged between your containers

43

Demo: Metricbeat, Filebeat, Packetbeat

Multiple data types, one view in Kibana

Thank you

• github.com/elastic/beats
• discuss.elastic.co
• @elastic #elasticbeats
• #beats on freenode

44