NOSQL WITH CACHING, SEARCH AND REAL-TIME ANALYTICS James Gorlick - - PowerPoint PPT Presentation

▶

Nov 03, 2023 29 likes •382 views

NOSQL WITH CACHING, SEARCH AND REAL-TIME ANALYTICS James Gorlick Basho bit.ly/bdp-on-the-under-ln @paegun About Me Application Developer with Lead and Project Management experience in a wide variety of business applications. Particularly

SLIDE 1

NOSQL WITH CACHING, SEARCH AND REAL-TIME ANALYTICS

James Gorlick Basho

bit.ly/bdp-on-the-under-ln @paegun

SLIDE 2

About Me

Application Developer with Lead and Project Management experience in a wide variety of business applications. Particularly interested in straight-through processing, removing data duplication, and increasing information availability throughout the organization in a secure manner. Previously with Dimensional Fund Advisors performed data science with much Fortran and many data sources while helping DFA "buy the market". Developed a number of in-house solutions for Myspace to serve purposes similar to what BDP does and will provide.

@paegun

SLIDE 3

Word Count

Just as developer activities extend beyond writing and testing software into what were typically operational concerns, data solutions are extending beyond storing and retrieving data from a single database management system into the deployment and management of heterogeneous services. A walk through the development and operational support of a reference application utilizing the Basho Data Platform, gains within the development and operations cycles will be demonstrated.

SLIDE 4

SLIDE 5

SLIDE 6

SLIDE 7

Excellent Data

NLC Origin TLC Station Name ernment Office Region (G County or Unitary Authority strict or Unitary Author S2 Spatial_Unit C NUTS2 Spatial Unit Station Facility Owne Station Group 5131 ABW Abbey Wood London Greater London Greenwich UKI2 Outer London Southeastern 3813 ABE Aber Wales - Cymru Caerphilly - Caerffili Caerphilly - Caerffili UKL1 West Wales and the Valleys Arriva Trains Wales 3801 ACY Abercynon Wales - Cymru Rhondda Cynon Taff - Rhondda Cynon Taf Rhondda Cynon Taff - Rhondda Cynon Taf UKL1 West Wales and the Valleys Arriva Trains Wales 3982 ABA Aberdare Wales - Cymru Rhondda Cynon Taff - Rhondda Cynon Taf Rhondda Cynon Taff - Rhondda Cynon Taf UKL1 West Wales and the Valleys Arriva Trains Wales 8976 ABD Aberdeen Scotland Aberdeen City Aberdeen City UKM1 North Eastern Scotland First ScotRail 9090 AUR Aberdour Scotland Fife Fife UKM2 Eastern Scotland First ScotRail 4435 AVY Aberdovey Wales - Cymru Gwynedd - Gwynedd Gwynedd - Gwynedd UKL1 West Wales and the Valleys Arriva Trains Wales 4440 ABH Abererch Wales - Cymru Gwynedd - Gwynedd Gwynedd - Gwynedd UKL1 West Wales and the Valleys Arriva Trains Wales 3602 AGV Abergavenny Wales - Cymru Monmouthshire - Sir Fynwy Monmouthshire - Sir Fynwy UKL2 East Wales Arriva Trains Wales 2448 AGL Abergele & Pensarn Wales - Cymru Conwy - Conwy Conwy - Conwy UKL1 West Wales and the Valleys Arriva Trains Wales 4303 AYW Aberystwyth Wales - Cymru Ceredigion - Sir Ceredigion Ceredigion - Sir Ceredigion UKL1 West Wales and the Valleys Arriva Trains Wales 2552 ACR Accrington North West Lancashire Hyndburn UKD4 Lancashire Northern Rail 8600 AAT Achanalt Scotland Highland Highland UKM4 Highlands and Islands First ScotRail 8601 ACN Achnasheen Scotland Highland Highland UKM4 Highlands and Islands First ScotRail 8602 ACH Achnashellach Scotland Highland Highland UKM4 Highlands and Islands First ScotRail 7662 ACK Acklington North East Northumberland Alnwick UKC2 Northumberland and Tyne and Wear Northern Rail 7286 ACL Acle East Norfolk Broadland UKH1 East Anglia Abellio Greater Anglia 4529 ACG Acocks Green West Midlands West Midlands Birmingham UKG3 West Midlands London Midland Trains 2287 ACB Acton Bridge North West Cheshire Vale Royal UKD2 Cheshire London Midland Trains Period and Financial year Reporting Period Days in period Period beginning Period ending Bus journeys (m) Underground journeys (m) DLR Journeys (m) Tram Journeys (m) Overground Journeys (m) Emirates Airline Journeys (m)

01_10/11 1

31 1-Apr-10 1-May-10 189.1 90.5 6.3 2.3

02_10/11 2

28 2-May-10 29-May-10 181.6 84.5 5.8 2.2

03_10/11 3

28 30-May-10 26-Jun-10 175.9 84.3 5.8 2.1

04_10/11 4

28 27-Jun-10 24-Jul-10 183.4 86.5 6.1 2.1

05_10/11 5

28 25-Jul-10 21-Aug-10 160.4 82.9 5.8 2.0

06_10/11 6

28 22-Aug-10 18-Sep-10 175.8 80.9 5.5 2.0

07_10/11 7

28 19-Sep-10 16-Oct-10 189.8 88.7 6.3 2.3

08_10/11 8

28 17-Oct-10 13-Nov-10 179.9 90.3 6.7 2.2 5.6

09_10/11 9

28 14-Nov-10 11-Dec-10 178.8 90.6 6.4 2.3 5.4

10_10/11 10

28 12-Dec-10 8-Jan-11 140.1 72.5 4.8 1.8 3.5

11_10/11 11

28 9-Jan-11 5-Feb-11 183.0 84.4 6.3 2.1 5.2

12_10/11 12

28 6-Feb-11 5-Mar-11 177.2 87.8 6.5 2.2 5.2

13_10/11 13

26 6-Mar-11 31-Mar-11 173.9 83.5 6.0 2.1 5.8

01_11/12 1

30 1-Apr-11 30-Apr-11 183.8 91.2 6.4 2.1 6.3

02_11/12 2

28 1-May-11 28-May-11 186.1 87.8 6.3 2.2 6.4

03_11/12 3

28 29-May-11 25-Jun-11 181.7 88.9 6.1 2.2 6.9

04_11/12 4

28 26-Jun-11 23-Jul-11 186.7 92.5 6.4 2.3 7.6

05_11/12 5

28 24-Jul-11 20-Aug-11 161.1 85.5 6.2 1.9 7.7

06_11/12 6

28 21-Aug-11 17-Sep-11 173.9 85.3 6.4 2.1 7.7

07_11/12 7

28 18-Sep-11 15-Oct-11 193.4 93.1 7.5 2.4 8.7

08_11/12 8

28 16-Oct-11 12-Nov-11 185.2 95.8 7.3 2.3 8.8

09_11/12 9

28 13-Nov-11 10-Dec-11 189.4 97.1 7.1 2.4 9.0

10_11/12 10

28 11-Dec-11 7-Jan-12 151.2 79.3 5.2 1.9 6.9

11_11/12 11

28 8-Jan-12 4-Feb-12 181.4 89.8 6.9 2.2 8.7

12_11/12 12

28 5-Feb-12 3-Mar-12 179.5 91.5 7.0 2.0 8.8

13_11/12 13

28 4-Mar-12 31-Mar-12 191.2 92.7 7.3 2.3 9.1

01_12/13 1

28 1-Apr-12 28-Apr-12 171.8 88.7 7.0 2.1 8.5

02_12/13 2

28 29-Apr-12 26-May-12 189.2 92.6 7.4 2.3 8.9

03_12/13 3

28 27-May-12 23-Jun-12 176.5 89.7 7.0 2.2 8.6

04_12/13 4

28 24-Jun-12 21-Jul-12 189.9 96.9 7.9 2.5 8.9 0.24

05_12/13 5

28 22-Jul-12 18-Aug-12 167.1 103.0 10.6 2.3 10.5 0.53

06_12/13 6

28 19-Aug-12 15-Sep-12 177.0 94.0 8.7 2.4 9.8 0.38

Period and Financial year Reporting Period Days in period Period ending Month Number of Lost Customer Hours Operated Kms (Peak and Off Peak) % of Scheduled Operated Excess Journey T (mins) 02_11/12 2 28 28-May-11 May-11 2,322,089 5,444,365 97.3% 03_11/12 3 28 25-Jun-11 Jun-11 1,929,445 5,463,580 97.6% 04_11/12 4 28 23-Jul-11 Jul-11 2,300,769 5,465,180 96.8% 05_11/12 5 28 20-Aug-11 Aug-11 2,059,198 5,595,332 97.1% 06_11/12 6 28 17-Sep-11 Sep-11 2,462,085 5,516,502 96.8% 07_11/12 7 28 15-Oct-11 Oct-11 2,155,102 5,512,264 97.5% 08_11/12 8 28 12-Nov-11 Nov-11 2,054,499 5,657,701 97.5% 09_11/12 9 28 10-Dec-11 Dec-11 1,989,066 5,694,703 97.4% 10_11/12 10 28 7-Jan-12 Jan-12 2,043,901 5,275,158 96.4% 11_11/12 11 28 4-Feb-12 Feb-12 2,073,884 5,730,948 97.1% 12_11/12 12 28 3-Mar-12 Mar-12 2,604,341 5,559,547 95.9% 13_11/12 13 28 31-Mar-12 Mar-12 1,997,431 5,682,740 97.4% 1 28 28-Apr-12

SLIDE 8

E_TOOMANYOPTIONS

SLIDE 9

Prescriptive, Open

SLIDE 10

SLIDE 11

Storage

SLIDE 12

SLIDE 13

Consistent Hash (Route)

SLIDE 14

Ring Around The Rosie

SLIDE 15

Multi Data Center

SLIDE 16

CACHE

SLIDE 17

Read-Through

SLIDE 18

Write-Around

SLIDE 19

SLIDE 20

Excuse Me Miss

SLIDE 21

Search

SLIDE 22

Ridealong

SLIDE 23

Multi Service

SLIDE 24

Analysis

SLIDE 25

Ship The Compute?

SLIDE 26

Spread The Load

SLIDE 27

No Pets

SLIDE 28

SLIDE 29

Graph Reality

SLIDE 30

Window Seat

SLIDE 31

SLIDE 32

QUEUE

SLIDE 33

HOST

SLIDE 34

For More Information

London Datastore - http://data.london.gov.uk/
Basho Data Platform - http://docs.basho.com/dataplatform/latest/
Spark - http://spark.apache.org/
Redis - http://redis.io
Kafka - http://kafka.apache.org/
Solr - http://lucene.apache.org/solr/
Riak - http://docs.basho.com/riak/latest/

SLIDE 35