How Elasticsearch powers the Guardians newsroom shay banon @kimchy - - PowerPoint PPT Presentation

▶

Jan 25, 2023 427 likes •840 views

How Elasticsearch powers the Guardians newsroom shay banon @kimchy phil wills @philwills creator, co-founder and cto senior software architect elasticsearch guardian news and media created in 1936 ... to secure the financial and

SLIDE 1

How Elasticsearch powers the Guardian’s newsroom

phil wills ■ @philwills

senior software architect guardian news and media

shay banon ■ @kimchy

creator, co-founder and cto elasticsearch

SLIDE 2

SLIDE 3

SLIDE 4

SLIDE 5

“created in 1936 ... to secure the financial and editorial independence of the Guardian in perpetuity”

SLIDE 6

SLIDE 7

ur in-house real-time traffic tool

SLIDE 8

SLIDE 9

SLIDE 10

SLIDE 11

desktop workstation

production apaches

something htmly

?

SLIDE 12

ssh $SERVER "nice tail -f /apache2/logs/guardian-access_log"

SLIDE 13

desktop workstation

2 x production apaches publisher ssh “tail” zeromq

x

SEO dashboard

SLIDE 14

SLIDE 15

SLIDE 16

SLIDE 17

desktop workstation

x

SLIDE 18

Javascript in browser

SNS SQS

hidden pixel

Dashboard Tracker

SLIDE 19

SLIDE 20

Elasticsearch

“you know, for search”

SLIDE 21

SLIDE 22

Javascript in browser

Tracker SNS SQS

image pixel

SQS Dashboard Serf elasticsearch Dashboard

SLIDE 23

SLIDE 24

https://github.com/guardian/status-app

6 * c3.4xlarge

in an autoscaling group (with manual scaling) instance store (SSD)

SLIDE 25

SLIDE 26

SLIDE 27

{ "dt": "2014-06-13T20:01:48.026Z", "url": "http://www.theguardian.com/football/2014/jun/13/spain-v-holland-world-cup-2014- live-report", "queryString": "", "host": "www.theguardian.com", "path": "/football/2014/jun/13/spain-v-holland-world-cup-2014-live-report", "section": "football", "platform": "r2", "userAgent": { "type": "Browser", "family": "Safari 5.1.9", "os": "OS X 10.6.8", "device": "Personal computer" }, "documentReferrer": "http://www.theguardian.com/football", "browser": { "id": "gA6RUFLhWNQvWdt0rW4r78Fg", "isNew": false }, "referringHost": "theguardian.com", "referringPath": "/football", "isContent": true, "contentPublicationDate": "2014-03-03", "countryCode": "US", "countryName": "United States", "location": { "lonlat": [-73.4409, 41.2094] } }

⇠filter ⇠filter ⇠count per minute

SLIDE 28

{ "query" : { "filtered" : { "query" : { "match_all" : { } }, "filter" : { "term" : { "path": "/football/2014/jun/13/spain-v-holland-world-cup-2014-live-report" } } } }, …

SLIDE 29

… "facets": { "Reddit": { "date_histogram": { "field": "dt", "interval": "1m" }, "facet_filter": { "term": { "referringHost": "reddit.com" } } }, "Facebook": { "date_histogram": { "field": "dt", "interval": "1m" }, "facet_filter": { "term": { "referringHost": "facebook.com" } } }, "Google": { "date_histogram": { "field": "dt", "interval": "1m" }, "facet_filter": { "or": { "filters": [ { "prefix": { "referringHost": "www.google." } }, { "prefix": { "referringHost": "news.google." } } ] } } } } }

SLIDE 30

SLIDE 31

SLIDE 32

"aggregations" : { "dns" : { "date_histogram" : { "field" : "dt", "interval" : "1m" }, "aggregations" : { "dns" : { "percentiles" : { "field" : "dns", "percents" : [ 50.0 ], "estimator" : "tdigest", "compression" : 10.0 } } } } }

SLIDE 33

/graph/breakdown?section=commentisfree

SLIDE 34

?section=commentisfree

phan.StandardFilters
phan.StandardFiltersToElasticsearch
rg.elasticsearch.index.

query.FilterBuilder

SLIDE 35

{ "query" : { "filtered" : { "query" : { "match_all" : { } }, "filter" : { "term" : { "path": "/football/2014/jun/13/spain-v-holland-world-cup-2014-live-report" } } } }, …

SLIDE 36

"filter": { "and": { "filters": [ { "range": { "dt": { "from": "2014-03-03T00:00:00.000Z", "to": "2014-03-03T22:30:59.999Z", "include_lower": true, "include_upper": false } } }, { "not": { "filter": { "term": { "countryCode": "GNM" } } } }, { "not": { "filter": { "term": { "userAgent.type": "Robot" } } } }, { "filter": { "terms": { "section": [ "commentisfree" ] }} } ] } }

SLIDE 37

SLIDE 38

SLIDE 39

SLIDE 40

thank you

phil wills ■ @philwills

senior software architect guardian news and media

shay banon ■ @kimchy

creator, co-founder and cto elasticsearch