Centralized logging Using lasers and other cool stuff Sylvain - - PowerPoint PPT Presentation

Centralized logging

Using lasers and other cool stuff

Sylvain Fankhauser / Bastian Widmer

Agenda

• The beginnings
• What Liip is doing now
• Goals
• Possible Solutions
• Our Solution
• Implementations (SF2, Drupal 6&7, other

php apps)

• Demo

The early beginnings

August 2011 - Adrian Schwaller "Liipers like developing and Liipers are good in

• developing. However when it's going to

maintenance we often lack of quality."

The early beginnings

August 2011 - Adrian Schwaller "It is not rarely the case that the client has to inform us about an impediment. So why not developing an application that help us doing maintenance - a win-win solution."

Stock-check: what are we doing now?

• sending mails with logs (LOTS OF MAILS!)
• checking manually (humans fail regularly!)
• waiting for the client to complain about

issues on his website Not a good solution, right?

Second iteration

Sylvain and Bastian - September 2012 We get a Year Goal to centralize logging

Goals

• introduce a standard platform
• log unexpected results and thrown errors
• early on detection of problems (DDOS,

Spammers)

• be faster to notice the problem than our

customers

• Create a PHP tool that could be easily

plugged into any type of application (SF2, Drupal, Mahara, Moodle)

Solutions

there are many let's look at them!

Splunk (splunk.com)

+ nice graphical interface + google.com for logfiles + easy to maintain + easy to setup + can do everything you dream about

• costs a big amount of money $$$

Loggly (loggly.com)

+ Cloud Based + no maintenance work needed (SaaS)

• Costs: Depending on our needs - not cheap
• Cloud Based (Security concerns)

Graylog 2 (graylog2.org)

+ nice interface + not complicated to setup + active development

• very young project and architectural changes

(mongodb -> elastic search)

Kibana (kibana.org)

kibana went into our focus after we've done the

• ecostar. It's quite nice but we did not test it

during our ecostar. Bastian short tested it afterwards on his own: + simple setup + actively maintained + multimegafunctional by use of logstash frontend/backend : ruby datastorage : elasticsearch

Logstash, WHAT?

Logstash is the swiss army knife regarding logging needs, it's a translator between many different formats. e.g. translates .log files on the fly into gelf messages that could be consumed by graylog check : logstash.net

le Setup

graylog - what you could do

graylog- what we do currently

Graylog Server (currently)

• Frontend - UDP Backend : Graylog - Java
• Data Storage : MongoDB

Graylog Server Changes v0.10

• Frontend - UDP Backend :Graylog - Java
• Data Storage : Elastic Search
• optional : Logstash
• new : Support for Plugins

○ GECKOBOARD - ohai hackday! ○ sending XMPP Jabber Messages ○ writing Graphs with Graphite ○ GELF over HTTP (a log entry is just a curl request away)

Developers, developers, developers

finally some php ;)

Use Graylog in your project

• Configure your application to send logs to

Graylog

• Very easy to use: log stuff as usual, it's

automatically sent to Graylog

• Under the hood, you'll use a GELF library
• Docs about integration in your project are

available here: http://liip.to/graylog

Logging with Symfony >= 2.1

• Add the GELF library to composer:

"require": { ... "mlehner/gelf-php": "1.0.*" }

Logging with Symfony >= 2.1

• Configure monolog:

services: gelf_formatter: class: Monolog\Formatter\GelfMessageFormatter arguments: ['clientdomain.com'] monolog: handlers: gelf: type: gelf publisher: hostname: log.liip.ch port: 12201 formatter: gelf_formatter level: warn

Logging with Symfony >= 2.1

• Then log stuff as usual:

$logger = $this->get('logger'); $logger->warn('Something went wrong');

Logging with Symfony 2.0

• The version of Monolog shipped with SF2

doesn't support GELF

• Installation is a bit painful, but easy with the

guide @ http://liip.to/graylog

Logging with Drupal 7

• The easy way:

curl -L http://liip.to/graylogdrupal7 | sh

• Adds the required modules as submodules
• Enables them
• Adds them to your modules.enabled file

Logging with Drupal 7

• The less-easy way:

○ Install the gelf-php library ○ Install the libraries and gelf modules ○ Use the gelf module hosted on my github account

Logging with Drupal 6

• Again the easy way:

curl -L http://liip.to/graylogdrupal6 | sh

• Does the same as the Drupal 7 solution
• Uses the 6.x modules branches
• The less-easy way is the same as for Drupal 7

Logging with other PHP apps

• Use the gelf-php library from here: https:

//github.com/Graylog2/gelf-php

• Create the message:

<?php require('GELFMessage.php'); require('GELFMessagePublisher.php'); $message = new GELFMessage(); $message->setShortMessage('ohai.'); $message->setFullMessage('i are a cats'); $message->setHost('clientdomain.com'); $message->setLevel(GELFMessage::CRITICAL); $message->setFile(__FILE__); $message->setLine(__LINE__); $message->setAdditional('the_grass', 'is green'); $message->setAdditional('skies', 'so blue');

Logging with other PHP apps

• Publish the message:

<?php ... $publisher = new GELFMessagePublisher('log.liip.ch'); $publisher->publish($message);

DEMO!

Questions?

Thank You!

Logging for a better world : http://log.liip.ch username : liip password : on our pw.liip.ch server (search for graylog) http://liip.to/graylog