Unifying Events & Logs into the Cloud Eduardo Silva August 17, - - PowerPoint PPT Presentation

▶
unifying events logs into the cloud
SMART_READER_LITE
LIVE PREVIEW

Unifying Events & Logs into the Cloud Eduardo Silva August 17, - - PowerPoint PPT Presentation

Dec 01, 2023 918 likes •1.5k views

Unifying Events & Logs into the Cloud Eduardo Silva August 17, 2015 eduardo@treasuredata.com CloudOpen/LinuxCon, Seattle @ edsiper About Me Eduardo Silva Github & Twitter @edsiper Personal Blog

slide-1
SLIDE 1

Eduardo Silva eduardo@treasuredata.com @edsiper

Unifying Events & Logs into the Cloud

August 17, 2015 CloudOpen/LinuxCon, Seattle

slide-2
SLIDE 2

About Me

Eduardo Silva

  • Github & Twitter @edsiper
  • Personal Blog http://edsiper.linuxchile.cl

Treasure Data

  • Open Source Engineer
  • Fluentd / Fluent Bit http://github.com/fmuent

Projects

  • Monkey HTTP Server http://monkey-project.com
  • Duda I/O http://duda.io
slide-3
SLIDE 3

Logging

slide-4
SLIDE 4

Logging Matters

Pros

  • Application status
  • Debugging
  • General information about anomalies: errors
  • Troubleshooting / Support
  • Local or Remote (network)
slide-5
SLIDE 5

Logging Matters

From a business point of view

  • Input data

Analytics →

  • User interaction / behaviors
  • Improvements
slide-6
SLIDE 6

Assumptions

slide-7
SLIDE 7

Logging Matters

Assumptions

  • I have enough disk space
  • I/O operations will not block
  • Log messages are human readable
  • My logging mechanism scale
slide-8
SLIDE 8

Logging Matters

Assumptions Basically, yeah.. it should work.

slide-9
SLIDE 9

Concerns

slide-10
SLIDE 10

Logging Matters

Concerns

  • Logs increase = data increase
  • Message format get more complex
  • Did the Kernel fmush the bufgers ? (sync(2))
  • Multi-thread application ?, locking ?
  • Multiple Applications = Multiple Logs
slide-11
SLIDE 11

Logging Matters

Concerns

If Multiple Applications = Multiple logs Multiple Hosts x Multiple Applications = ???

slide-12
SLIDE 12

OK, so:

  • 1. Logging matters
  • 2. It's really benefjcial
  • 3. but...
slide-13
SLIDE 13

It needs to be done right.

slide-14
SLIDE 14

Logging

Common sources & inputs

  • Application Logs
  • Apache
  • NginX
  • Syslog (-ng)
  • Custom applications / Languages
  • C, Ruby, Python, PHP, Perl, NodeJS, Java, etc.
slide-15
SLIDE 15

In a galaxy not so far away...

slide-16
SLIDE 16
slide-17
SLIDE 17

How to parse/store multiple data sources ?

note: performance matters!

slide-18
SLIDE 18
slide-19
SLIDE 19

Fluentd is an open source data collector

It let's you unify the data collection for a better use and understanding of data.

slide-20
SLIDE 20

before

slide-21
SLIDE 21

after

slide-22
SLIDE 22

Fluentd

Highlights

  • High Performance
  • Built-in Reliability
  • Structured Logs
  • Pluggable Architecture
  • More than 300 plugins! (input/fjltering/output)
slide-23
SLIDE 23

Fluentd

Architecture

slide-24
SLIDE 24

Fluentd

Internals simplifjed

slide-25
SLIDE 25

Fluentd

Input plugins

slide-26
SLIDE 26

Fluentd

Output plugins

slide-27
SLIDE 27

Fluentd

Buffer plugins

slide-28
SLIDE 28

Fluentd

Buffer plugins

slide-29
SLIDE 29
slide-30
SLIDE 30

M x N M + N →

slide-31
SLIDE 31

Fluentd

Simple Forwarding

slide-32
SLIDE 32

Fluentd

Simple Forwarding: confjguration

# logs from a fjle # store logs to MongoDB <source> <match backend.*> type tail type mongo path /var/log/httpd.log database fmuent format apache2 collection test tag backend.apache </match> </source> # logs from client libraries <source> type forward port 24224 </source>

slide-33
SLIDE 33

Fluentd

Less Simple Forwarding

slide-34
SLIDE 34

Fluentd

Lambda Architecture

slide-35
SLIDE 35

Fluentd

# logs from a fjle # store logs to MongoDB <source> <match *.*> type tail type copy path /var/log/httpd.log <store> format apache2 type elasticsearch tag backend.apache logstash_format true </source> </store> # logs from client libraries <store> <source> type webhdfs type forward host 192.x.y.z port 24224 port 50070 </source> path /path/to/hdfs </store> </match>

slide-36
SLIDE 36

Who uses Fluentd in production ?

slide-37
SLIDE 37
slide-38
SLIDE 38
slide-39
SLIDE 39

We collect 800k events per second !

slide-40
SLIDE 40

Internet of Things

slide-41
SLIDE 41

Internet of Things

Facts

  • IoT will grow to many billions of devices over the

next decade.

  • Now it's about device to device connectivity.
  • Difgerent frameworks and protocols are emerging.
  • It needs Logging.
slide-42
SLIDE 42

Internet of Things

Alliances

Vendors formed alliances to join forces and develop generic software layers for their products:

slide-43
SLIDE 43

Internet of Things

Solutions provided

Alliance Framework

→ →

slide-44
SLIDE 44

IoT & Big Data

Analytics

IoT requires a generic solution to collect events and data from difgerent sources for further analysis. Data can come from a specifjc framework, radio device, sensor or other. How do we collect and unify data properly ?

slide-45
SLIDE 45

@fluentbit

slide-46
SLIDE 46

Fluent Bit is an open source data collector

It let's you collect data from IoT/Embedded devices and transport It to third party services.

slide-47
SLIDE 47

Fluent Bit

Targets

  • Services
  • Sensors / Signals / Radios
  • Operating System information
  • Automotive / Telematics
slide-48
SLIDE 48

Fluent Bit

Requirements

IoT and Embedded environment requires special handling, specifjcally on performance and resource utilization:

  • Lightweight
  • Written in C Language
  • Customizable, pluggable architecture
  • Full integration with Fluentd
slide-49
SLIDE 49

Fluent Bit

Integration

slide-50
SLIDE 50

Fluent Bit

Direct Output

slide-51
SLIDE 51

Containers

slide-52
SLIDE 52
slide-53
SLIDE 53

Docker

Logging driver

  • Docker v1.6 released the concept of logging drivers
  • Route container output
  • Fluentd ?
slide-54
SLIDE 54

Docker

slide-55
SLIDE 55

Docker v1.8

Fluentd Logging driver!

slide-56
SLIDE 56

We Love Data!

Thank you!

  • http://fmuentd.org
  • http://fmuentbit.io
  • https://docs.docker.com/reference/logging/fmuentd/
  • http://github.com/fmuent/fmuentd