About Me > Eduardo Silva Github & Twitter: @edsiper Personal - - PowerPoint PPT Presentation

▶
about me
SMART_READER_LITE
LIVE PREVIEW

About Me > Eduardo Silva Github & Twitter: @edsiper Personal - - PowerPoint PPT Presentation

Jun 04, 2023 436 likes •971 views

About Me > Eduardo Silva Github & Twitter: @edsiper Personal Blog : http://edsiper.linuxchile.cl > Treasure Data, Inc. Open Source Engineer Fluentd / FluentBit > Projects Monkey Server / monkey-project.com Duda

slide-1
SLIDE 1
slide-2
SLIDE 2

About Me

> Eduardo Silva

  • Github & Twitter: @edsiper
  • Personal Blog : http://edsiper.linuxchile.cl

> Treasure Data, Inc.

  • Open Source Engineer

Fluentd / FluentBit > Projects

  • Monkey Server / monkey-project.com
  • Duda I/O / duda.io
slide-3
SLIDE 3

Logging

slide-4
SLIDE 4

Logging Matters

  • Applications status
  • Debugging
  • General information about anomalies: errors
  • Troubleshooting / Support
  • Local or Remote (network)

Pros

slide-5
SLIDE 5

Logging Matters

  • Input Data → Analytics
  • User interactions / behaviors
  • Improvements

From a business point of view

slide-6
SLIDE 6

Assumptions

slide-7
SLIDE 7

Logging Matters

  • “I have enough disk space”
  • “I/O operations will not block”
  • “It's fast to write a simple message”
  • “Log formats are human readable”
  • “My logging mechanism scale”

Assumptions

slide-8
SLIDE 8

Logging Matters

Basically, yeah.. it should work.

Assumptions

slide-9
SLIDE 9

Concerns

slide-10
SLIDE 10

Logging Matters

  • Logs increase = data increase
  • Message format get more complex
  • Did the Kernel flush the buffers ? (sync(2))
  • Multi-thread application ?, Locking ?
  • Multiple Applications = Multiple Logs

Concerns

slide-11
SLIDE 11

Logging Matters

Concerns If... Multiple Applications = Multiple logs Multiple Host x Multiple Applications = ???

slide-12
SLIDE 12

OK...so:

  • 1. Logging Matters
  • 2. It's really beneficial
  • 3. ???

but...

slide-13
SLIDE 13

It needs to be done right.

slide-14
SLIDE 14

Logging

Common Sources / Inputs

  • Applications Logs
  • Apache
  • Nginx
  • Syslog (-ng)
  • Custom applications / Languages

C, Ruby, Python, PHP, Perl, NodeJS, Java...

slide-15
SLIDE 15

How to parse/store multiple data inputs ?

note: performance matters!

slide-16
SLIDE 16
slide-17
SLIDE 17

Fluentd is an open source data collector

It let's you unify the data collection for a better use and understanding of data.

slide-18
SLIDE 18

Fluentd

Highlights

  • High Performance
  • Built-in Reliability
  • Structured Information/Logs
  • Pluggable Architecture
  • More than 300 plugins (input/filtering/output)
slide-19
SLIDE 19

Fluentd

Before

slide-20
SLIDE 20

Fluentd

After

slide-21
SLIDE 21
slide-22
SLIDE 22

Configuration

slide-23
SLIDE 23
slide-24
SLIDE 24

Who uses Fluentd in Production ?

slide-25
SLIDE 25
slide-26
SLIDE 26

Logging and other use cases

slide-27
SLIDE 27

Logging and other use cases

We collect

500.000 events per second!

slide-28
SLIDE 28

Logging and other use cases

slide-29
SLIDE 29

Logging

More use cases

  • Internet of Things (IoT)
  • Embedded Applications
  • OS Containers (Docker)

We talked mostly about Server applications, let's talk now about

  • ther contexts:

The fun continues ;)

slide-30
SLIDE 30

Internet of Things (IoT)

slide-31
SLIDE 31

Internet of Things

Facts

  • IoT will grow to many billions of devices over the next decade.
  • Now it's about device to device connectivity.
  • Different frameworks and protocols are emerging.
  • It needs Logging.
slide-32
SLIDE 32

Internet of Things

Frameworks

In order to solve the device to device connectivity and proper messaging, multiple framework/stacks are emerging, all of them backed by different companies and vendors. The most remarkable at the moment are:

Brillo, by Google

Just announced, no logo..sorry!

slide-33
SLIDE 33

Internet of Things

Logging

  • How do we approach data collection for IoT ?
  • How can we improve our IoT based product if we have data

from thousand of devices with multiple sensors, meaning many inputs ?

slide-34
SLIDE 34

Embedded

slide-35
SLIDE 35

Embedded

Importance of the Embedded Market

  • Embedded devices are everywhere:
  • Medical Devices
  • Automotive
  • Home Automation
  • Many...
slide-36
SLIDE 36

Fluentd for Embedded ?

slide-37
SLIDE 37
slide-38
SLIDE 38

Fluent Bit

Objective

  • Services
  • Sensors / Signals / Radios
  • Operating System Info
  • Automotive / Telematics

Open Source data collection tool for Embedded Linux:

slide-39
SLIDE 39

Fluent Bit

Requirements

  • Lightweight
  • Written in C Language.
  • Customizable, pluggable architecture.
  • Integrate with Fluentd

An Embedded environment requires special handling, specifically on performance and resources utilization, for hence it needs to be:

slide-40
SLIDE 40

Fluent Bit

Requirements

  • Support custom Inputs
  • Support custom Outputs
  • Use Binary Serialization (MessagePack)
  • Configurable (though file system files)
slide-41
SLIDE 41

Features

slide-42
SLIDE 42

Fluent Bit

Features

  • Collection & Distribution.
  • Built-in system metrics.
  • C API for Developers (WIP).
  • Integration with third party services.
  • Open Source / Apache License v2.0
slide-43
SLIDE 43

Fluentd Integration

slide-44
SLIDE 44

Fluent Bit

Integration

slide-45
SLIDE 45

Direct Output

slide-46
SLIDE 46

Fluent Bit

Direct Output

slide-47
SLIDE 47

Examples

slide-48
SLIDE 48

Fluent Bit

Built-in Metrics: CPU usage

slide-49
SLIDE 49

Fluent Bit

Kernel Log Messages

slide-50
SLIDE 50

Fluent Bit

Configuration Example

slide-51
SLIDE 51

Roadmap

slide-52
SLIDE 52

Fluent Bit

Roadmap

  • Library mode.
  • Support a stock of sensors (inputs).
  • HTTP input.
  • Release first stable version, work in process!
  • Documentation.
slide-53
SLIDE 53

github.com/fluent/fluentd github.com/fluent/fluent-bit Eduardo Silva @edsiper / eduardo@treasuredata.com

Thanks!

http://fluentd.org http://fluentbit.io We are Hiring!