About Me > Eduardo Silva Github & Twitter: @edsiper Personal - PowerPoint PPT Presentation

About Me > Eduardo Silva ● Github & Twitter: @edsiper ● Personal Blog : http://edsiper.linuxchile.cl > Treasure Data, Inc. ● Open Source Engineer Fluentd / FluentBit > Projects ● Monkey Server / monkey-project.com ● Duda I/O / duda.io

Logging

Logging Matters Pros ● Applications status ● Debugging ● General information about anomalies: errors ● Troubleshooting / Support ● Local or Remote (network)

Logging Matters From a business point of view ● Input Data → Analytics ● User interactions / behaviors ● Improvements

Assumptions

Logging Matters Assumptions ● “I have enough disk space ” ● “I/O operations will not block ” ● “It's fast to write a simple message” ● “Log formats are human readable ” ● “My logging mechanism scale ”

Logging Matters Assumptions B asically, yeah.. it should work.

Concerns

Logging Matters Concerns ● Logs increase = data increase ● Message format get more complex ● Did the Kernel flush the buffers ? ( sync (2)) ● Multi-thread application ?, Locking ? ● M ultiple Applications = M ultiple Logs

Logging Matters Concerns If... Multiple Applications = Multiple logs Multiple Host x Multiple Applications = ???

OK...so: 1. Logging Matters 2. It's really beneficial 3. ??? but...

It needs to be done right.

Logging Common Sources / Inputs ● Applications Logs • Apache • Nginx • Syslog (-ng) ● Custom applications / Languages C , Ruby, Python, PHP, Perl, NodeJS, Java...

How to parse/store multiple data inputs ? note: performance matters!

Fluentd is an open source data collector It let's you unify the data collection for a better use and understanding of data.

Fluentd Highlights ● High Performance ● Built-in Reliability ● Structured Information/Logs ● Plug gable Architecture ● More than 300 plugins (input/filtering/output)

Fluentd Before

Fluentd After

Configuration

Who uses Fluentd in Production ?

Logging and other use cases

We collect Logging and other 500.000 events per second! use cases

Logging and other use cases

Logging More use cases We talked mostly about Server applications, let's talk now about other contexts: ● Internet of Things ( IoT ) ● Embedded Applications ● OS Containers ( Docker ) The fun continues ;)

Internet of Things (IoT)

Internet of Things Facts ● IoT will grow to many billions of devices over the next decade. ● Now it's about device to device connectivity. ● Different frameworks and protocols are emerging. ● It needs Logging .

Internet of Things Frameworks In order to solve the device to device connectivity and proper messaging, multiple framework / stacks are emerging, all of them backed by different companies and vendors. The most remarkable at the moment are: Brillo, by Google Just announced, no logo..sorry!

Internet of Things Logging ● How do we approach data collection for IoT ? ● How can we improve our IoT based product if we have data from thousand of devices with multiple sensors, meaning many inputs ?

Embedded

Embedded Importance of the Embedded Market ● Embedded devices are everywhere: ● Medical Devices ● Automotive ● Home Automation ● Many...

Fluentd for Embedded ?

Fluent Bit Objective Open Source data collection tool for Embedded Linux: ● Services ● Sensors / Signals / Radios ● Operating System Info ● Automotive / Telematics

Fluent Bit Requirements An Embedded environment requires special handling, specifically on performance and resources utilization, for hence it needs to be: ● Lightweight ● Written in C Language. ● Customizable, pluggable architecture. ● Integrate with Fluentd

Fluent Bit Requirements ● Support custom Inputs ● Support custom Outputs ● Use Binary Serialization (MessagePack) ● Config urable (though file system files)

Features

Fluent Bit Features ● C ollection & D istribution. ● Built-in system metrics . ● C API for Developers (WIP). ● Integration with third party services. ● Open Source / Apache License v2.0

Fluentd Integration

Fluent Bit Integration

Direct Output

Fluent Bit Direct Output

Examples

Fluent Bit Built-in Metrics: CPU usage

Fluent Bit Kernel Log Messages

Fluent Bit Configuration Example

Roadmap

Fluent Bit Roadmap ● Library mode. ● Support a stock of sensors (inputs). ● HTTP input. ● Release first stable version, work in process! ● Documentation.

Thanks! http:// fluentd.org http:// fluentbit.io github.com/ fluent/fluentd We are Hiring! github.com/ fluent/fluent-bit Eduardo Silva @edsiper / eduardo@treasuredata.com