Distributed Logging Architecture in Container Era LinuxCon Japan - PowerPoint PPT Presentation

Distributed Logging Architecture in Container Era LinuxCon Japan 2016 at Jun 13 2016 Satoshi "Moris" Tagomori (@tagomoris)

Satoshi "Moris" Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, ... Treasure Data, Inc.

Topics • Microservices and logging in various industries • Difficulties of logging with containers • Distributed logging architecture • Patterns of distributed logging architecture • Case Study: Docket and Fluentd • Why OSS are important for logging

Logging

Logging in Various Industries • Web access logs • Views/visitors on media • Views/clicks on Ads • Commercial transactions (EC, Game, ...) • Data from devices • Operation logs on Apps of phones • Various sensor data

Microservices and Logging Users Users Service (Application) Logs Logs • Microservices • many services produce data • Monolithic service about an users access • a service produces all data • it's needed to collect logs about an users access from many services to know what is happening

Logging and Containers

Containers: "a must" for microservices • Dividing a service into services • a service requires less computing resources 
 (VM -> containers) • Making services independent from each other • but it is very difficult :( • some dependency must be solved even in development environment 
 (containers on desktop)

Redesign Logging: Why? • No permanent storages • No fixed physical/network address • No fixed mapping between servers and roles

Containers: immutable & disposable • No permanent storages • Where to write logs? • file in container 
 → be gone w/ container instance 😟 • dir shared from host 
 → hosts are shared by many services ☹ • TODO: ship logs from container to anywhere ASAP

Containers: unfixed addresses • No fixed physical / network address • Where should we go to fetch logs? • Service discovery (e.g., consul) 
 → one more component 😟 • rsync? ssh+tail? or ..? Is it installed in container? 
 → one more tool to depend on ☹ • TODO: push logs to anywhere from containers

Containers: instances per roles • No fixed mapping between servers and roles • How can we parse / store these logs? • Central repository about log syntax 
 → very hard to maintain 😟 • Label logs by source address 
 → many containers/roles in a host ☹ • TODO: label & parse logs at source of logs

Distributed Logging Architecture

Core Architecture Collector nodes (Docker containers + agent) • Collector nodes • Aggregator nodes Aggregator nodes • Destination Destination 
 (Storage, Database, ...)

Collecting and Storing Data • Parse (collector) • Raw logs are not good for processing • Convert logs to structured data (key-value pairs) • Sort/Shuffle (aggregator) • Mixed logs are not good for scanning • Split whole data stream into streams • Store (destination) • Format logs(records) as destination expects

Scaling Logging • Network traffic • CPU load to parse / format • Parse logs on each collector (distributed) • Format logs on aggregator (to be distributed) • Capability • Make aggregators redundant • Controlling delay

Patterns

Aggregation Patterns source aggregation source aggregation NO YES destination aggregation NO destination aggregation YES

Source Side Aggregation Patterns w/ source aggregation w/o source aggregation collector aggregate container aggregator

Without Source Aggregation collector • Pros: • Simple configuration aggregator • Cons: • fixed aggregator (endpoint) address • many network connections • high load in aggregator

With Source Aggregation • Pros: • less connections aggregate • lower load in aggregator container • less configuration in containers 
 (by specifying localhost) • highly flexible configuration 
 (by deployment only for aggregate containers) • Cons: • a bit much resource (+1 container per host)

Destination Side Aggregation Patterns w/o destination aggregation w/ destination aggregation collector aggregator destination

Without Destination Aggregation • Pros: • Less nodes • Simpler configuration • Cons: • Storage side change affects collector side • Worse performance: many small write requests on storage

With Destination Aggregation • Pros: • Collector side configuration is 
 free from storage side changes aggregator • Better performance with fine tune 
 on destination side aggregator • Cons: • More nodes • A bit complex configuration

Scaling Patterns Scaling Up Endpoints Scaling Out Endpoints HTTP/TCP load balancer Round-robin clients Huge queue + workers Collector nodes Load balancer Aggregator nodes Backend nodes

Scaling Up Endpoints • Pros: • Simple configuration 
 Load balancer in collector nodes • Cons: • Scaling up limit Backend nodes

Scaling Out Endpoints • Pros: • Unlimited scaling 
 by adding aggregator nodes • Cons: • Complex configuration • Client features for round-robin

Without 
 With 
 Destination Aggregation Destination Aggregation Collecting logs over Internet Scaling Up Systems in early stages Endpoints or Using queues Impossible :( Scaling Out Collecting logs Collector nodes must know Endpoints in datacenter all endpoints ↓ Uncontrollable

Case Studies

Case Study: Docker+Fluentd • Destination aggregation + scaling up • Fluent logger + Fluentd • Source aggregation + scaling up • Docker json logger + Fluentd + Elasticsearch • Docker fluentd logger + Fluentd + Kafka • Source/Destination aggregation + scaling out • Docker fluentd logger + Fluentd

Why Fluentd? • Docker Fluentd logging driver • Docker container can send logs into Fluentd directly - less overhead • Pluggable architecture • Various destination systems • Small memory footprint • Source aggregation requires +1 container per host • Less additional resource usage ( < 100MB )

Destination aggregation + scaling up • Sending logs directly over TCP by Fluentd logger 
 in application code • Same with patterns of New Relic Application code

Source aggregation + scaling up • Kubernetes: Json logger + Fluentd + Elasticsearch • Applications write logs to STDOUT Application code • Docker writes logs as JSON in files • Fluentd 
 reads logs from file 
 Files (JSON) parse JSON objects 
 writes logs to Elasticsearch Elasticsearch http://kubernetes.io/docs/getting-started-guides/logging-elasticsearch/

Source aggregation + scaling up • Docker fluentd logging driver + Fluentd + Kafka • Applications write logs to STDOUT Application code • Docker sends logs 
 to localhost Fluentd • Fluentd 
 gets logs over TCP 
 pushes logs into Kafka Kafka

Source/Destination aggregation + scaling out • Docker fluentd logging driver + Fluentd • Applications write logs to STDOUT Application code • Docker sends logs 
 to localhost Fluentd • Fluentd 
 gets logs over TCP 
 sends logs into Aggregator Fluentd 
 w/ round-robin load balance

What's the Best? • Writing logs from containers: Some way to do it • Docker logging driver • Write logs on files + read/parse it • Send logs from apps directly • Keep it scalable! • Source aggregation: Fluentd on localhost • Scalable storage: (Kafka, external services, ...) • No destination aggregation + Scaling up • Non-scalable storage: (Filesystems, RDBMSs, ...) • Destination aggregation + Scaling out

Why OSS Are Important For Logging?

Why OSS? • Logging layer is interface • transparency • interoperability • Keep it scalable • number of nodes • number of types of source/destination

Use OSS, Make Logging Scalable