Event Logging System for Collaborative Communities Divyansh Sharma - - PowerPoint PPT Presentation

Event Logging System for Collaborative Communities

Divyansh Sharma Kartik Verma Rahul Raj Vishrut Jetly July 4, 2018

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 1 / 20

Outline

1

Motivation

2

Technologies Used

3

Work Flow

4

Capturing the Events

5

Creating the Log

6

Storing Logs in Elasticsearch

7

Event Log REST API

8

User Analytics

9

Problems Solved

10 Future Scope

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 2 / 20

Motivation

Motivation

What is Event Logging ?

Logging or storing the activities done by user of the system e.g. login, view a community, view article, edit article etc.

Adding an Event Logging System which can be easily integrated with the current system without affecting it. Adding an Event Logging System which can be easily extended to capture new events whenever they are added. Providing information For recommendation System. Providing User Analytics

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 3 / 20

Technologies Used

Technologies Used

Django Middleware ELK stack C3.js Django REST framework

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 4 / 20

Technologies Used

Technologies Used

Django Middleware

Every activity that gen- erates a http request to the server first passes through these middle- wares which attach and extract some informa- tion from the request e.g. session id, authen- tication tokens etc.

C3.js

C3.js is used to im- plement high quality, interactive charts for visualisation. Charts built using this library can be rendered in real time.

Django REST Framework

Django REST frame- work is a powerful and flexible toolkit for building Web APIs. Its a way to expose parts

• f the application to

third-parties

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 5 / 20

Technologies Used

Technologies Used

ELK Stack ELK is acronym for three open source projects Elasticsearch Logstash Kibana Logstash receives the data from a file or through an http request, applies filters, and then Elasticsearch stores the data, which can be later retrieved using Elasticsearch API. Kibana is used for visualisations and analytics.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 6 / 20

Work Flow

Work Flow Diagram

Figure: Work Flow Diagram

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 7 / 20

Capturing the Events

Capturing the Events

Figure: Django Middleware

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 8 / 20

Capturing the Events

Capturing the Events

The request after passing through all the middlewares finally moves through the eventlog middleware which captures all the incoming information from the request. This is a reason why the eventlog module is kept below all the middlewares. Primary Benefit of using middleware is that it keeps the eventlog system as a completely separate module or app which can be reused in django projects.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 9 / 20

Creating the Log

Creating the Log

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 10 / 20

Creating the Log

Important Points

The size of the bucket holding logs is not fixed. The function which processes the dict present in bucket runs in a seperate daemon thread. Logs can be stored either in file or in elasticsearch but here we are showing it only with elasticsearch because storing logs in a file are not useful. Logs are passed to ELK in JSON format.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 11 / 20

Storing Logs in Elasticsearch

Storing Logs in Elasticsearch

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 12 / 20

Storing Logs in Elasticsearch

Important Points

Elasticsearch, Logstash and Kibana runs on separate ports inside a docker. The name of the index which stores the logs is logs but can be changed in settings. Kibana is available only for the Super Admin users. Elasticsearch uses DSL query syntax built over Apache lucene Query Syntax parser for searching.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 13 / 20

Event Log REST API

Event Log REST API

Event log System provides a feature rich REST based API which is a wrapper over the Elasticsearch search API. This API is used for Recommendation System and for User Analytics

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 14 / 20

Event Log REST API

Features of REST API

The eventlog REST API is a feature rich API. The notable ones are: Pagination Sorting(both in ascending and descending) Custom Fields Filters based searching Time Range Based searching Aggregations or group by

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 15 / 20

User Analytics

User Analytics

User Analytics provides an overview of the activities of the users. Some of the Analytics added are: Trending articles in a community or group (based on logs from 10 days) Most viewed articles in community Users most viewed articles Users own articles in various states

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 16 / 20

User Analytics

Some Snapshots

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 17 / 20

Problems Solved

Problems Solved

Added the Event Logging System without affecting the existing system in any way using middleware. Added a System which can be easily modified to add new events for capture. Provided a feature rich API for use by recommendation system and for user analytics. Added User Analytics to the current System.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 18 / 20

Future Scope

Future Scope

Adding browser events i.e. events that are generated in front end, for instance

Window resize events Scroll events These events may provide some useful information about human behaviour

Adding Video related events Adding more User Analytics Add real time analytics whenever logs are updated Storing logs in cloud storage.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 19 / 20

Future Scope

Thank You

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 20 / 20