Scalaris: Scalable Web Applications with a Transactional Key-Value - - PowerPoint PPT Presentation

▶
scalaris scalable web applications with a transactional
SMART_READER_LITE
LIVE PREVIEW

Scalaris: Scalable Web Applications with a Transactional Key-Value - - PowerPoint PPT Presentation

Aug 19, 2023 116 likes •441 views

Scalaris Wiki on Scalaris Optimisations Questions Scalaris: Scalable Web Applications with a Transactional Key-Value Store Nico Kruber Michael Berlin Zuse Institut Berlin Parallel and Distributed Systems 5th June @ Berlin Buzzwords 2012

slide-1
SLIDE 1

Scalaris Wiki on Scalaris Optimisations Questions

Scalaris: Scalable Web Applications with a Transactional Key-Value Store

Nico Kruber Michael Berlin

Zuse Institut Berlin Parallel and Distributed Systems

5th June @ Berlin Buzzwords 2012

The research leading to these results has received funding from the European Union’s Seventh Frame- work Programme (FP7/2007-2013 ) under grant agree- ments n◦ 258862 (4CaaSt) and n◦ 257438 (Contrail). Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 1/17

slide-2
SLIDE 2

Scalaris Wiki on Scalaris Optimisations Questions

Outline

Scalaris Introduction Scalability Wiki on Scalaris MediaWiki Introduction Data Model Optimisations Strategies Results

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 2/17

slide-3
SLIDE 3

Scalaris Wiki on Scalaris Optimisations Questions

Scalaris - A Transactional Key-Value Store

Relational DB

  • multiple relations/tables
  • SQL
  • Joins
  • Transactions

most NoSQL stores

  • key-value/document mapping
  • scalable
  • fault-tolerant (replication)
  • eventual consistency

Scalaris

  • key-value pairs
  • scalable
  • transaction support
  • select, update/insert
  • fault-tolerant (replication)
  • strong consistency

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 3/17

slide-4
SLIDE 4

Scalaris Wiki on Scalaris Optimisations Questions

Scalaris Implementation

  • Scalaris: ~23000 lines of Erlang code
  • 2200 for transactions
  • APIs:
  • Java library (11500 lines)
  • Python library (1400 lines)
  • Ruby library (700 lines)
  • HTTP-JSON interface
  • http://scalaris.googlecode.com

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 4/17

slide-5
SLIDE 5

Scalaris Wiki on Scalaris Optimisations Questions

Scalaris - Scalability

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 5/17

slide-6
SLIDE 6

Scalaris Wiki on Scalaris Optimisations Questions

Wikitext A-Z

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 6/17

slide-7
SLIDE 7

Scalaris Wiki on Scalaris Optimisations Questions

Wikitext A-Z

page title including namespace

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 6/17

slide-8
SLIDE 8

Scalaris Wiki on Scalaris Optimisations Questions

Wikitext A-Z

page title including namespace link to another page

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 6/17

slide-9
SLIDE 9

Scalaris Wiki on Scalaris Optimisations Questions

Wikitext A-Z

page title including namespace link to another page template inclusion

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 6/17

slide-10
SLIDE 10

Scalaris Wiki on Scalaris Optimisations Questions

Wikitext A-Z

page title including namespace link to another page template inclusion category the page belongs to

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 6/17

slide-11
SLIDE 11

Scalaris Wiki on Scalaris Optimisations Questions

Wiki Operations

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 7/17

slide-12
SLIDE 12

Scalaris Wiki on Scalaris Optimisations Questions

Wiki Operations

show page: read revisions

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 7/17

slide-13
SLIDE 13

Scalaris Wiki on Scalaris Optimisations Questions

Wiki Operations

show page: read revisions revision history overview

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 7/17

slide-14
SLIDE 14

Scalaris Wiki on Scalaris Optimisations Questions

Wiki Operations

show page: read revisions revision history overview search for articles

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 7/17

slide-15
SLIDE 15

Scalaris Wiki on Scalaris Optimisations Questions

Wiki Operations

show page: read revisions revision history overview search for articles redirect to random article

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 7/17

slide-16
SLIDE 16

Scalaris Wiki on Scalaris Optimisations Questions

Wiki Operations

show page: read revisions revision history overview search for articles redirect to random article backlinks (“what links here”)

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 7/17

slide-17
SLIDE 17

Scalaris Wiki on Scalaris Optimisations Questions

Wiki Operations

show page: read revisions revision history overview search for articles redirect to random article backlinks (“what links here”) save new revision

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 7/17

slide-18
SLIDE 18

Scalaris Wiki on Scalaris Optimisations Questions

Editing Pages

Do atomically:

  • 1. store new revision
  • 2. update page object
  • 3. add new backlinks
  • 4. remove old backlinks
  • 5. add to new categories
  • 6. remove from old categories
  • 7. if new page:

a) add to page list b) increase page counter

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 8/17

slide-19
SLIDE 19

Scalaris Wiki on Scalaris Optimisations Questions

Editing Pages

Do atomically:

  • 1. store new revision
  • 2. update page object
  • 3. add new backlinks
  • 4. remove old backlinks
  • 5. add to new categories
  • 6. remove from old categories
  • 7. if new page:

a) add to page list b) increase page counter

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 8/17

slide-20
SLIDE 20

Scalaris Wiki on Scalaris Optimisations Questions

Editing Pages

Do atomically:

  • 1. store new revision
  • 2. update page object
  • 3. add new backlinks
  • 4. remove old backlinks
  • 5. add to new categories
  • 6. remove from old categories
  • 7. if new page:

a) add to page list b) increase page counter

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 8/17

slide-21
SLIDE 21

Scalaris Wiki on Scalaris Optimisations Questions

Editing Pages

Do atomically:

  • 1. store new revision
  • 2. update page object
  • 3. add new backlinks
  • 4. remove old backlinks
  • 5. add to new categories
  • 6. remove from old categories
  • 7. if new page:

a) add to page list b) increase page counter

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 8/17

slide-22
SLIDE 22

Scalaris Wiki on Scalaris Optimisations Questions

Editing Pages

Do atomically:

  • 1. store new revision
  • 2. update page object
  • 3. add new backlinks
  • 4. remove old backlinks
  • 5. add to new categories
  • 6. remove from old categories
  • 7. if new page:

a) add to page list b) increase page counter

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 8/17

slide-23
SLIDE 23

Scalaris Wiki on Scalaris Optimisations Questions

Editing Pages

Do atomically:

  • 1. store new revision
  • 2. update page object
  • 3. add new backlinks
  • 4. remove old backlinks
  • 5. add to new categories
  • 6. remove from old categories
  • 7. if new page:

a) add to page list b) increase page counter

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 8/17

slide-24
SLIDE 24

Scalaris Wiki on Scalaris Optimisations Questions

MediaWiki Schema

adapted from https://commons.wikimedia.org/wiki/File:MediaWiki_database_schema_1-19_%28r102798%29.svg

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 9/17

slide-25
SLIDE 25

Scalaris Wiki on Scalaris Optimisations Questions

Wiki on Scalaris - Data Layout

key content pages:<namespace> list of pages in <namespace> pages:<namespace>:count number of pages in <namespace> <title>:page page information (JSON object) <title>:rev:<id> single page revision (JSON object) <title>:revs list of revision info for history <title>:blpages list of page titles linking to <title> <cat_title>:cpages list of page titles inside a category <cat_title>:cpages:count number of pages inside a category

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 10/17

slide-26
SLIDE 26

Scalaris Wiki on Scalaris Optimisations Questions

Data Model - Issues?

  • non-scalable data model!
  • Problem: page lists, counters

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 11/17

slide-27
SLIDE 27

Scalaris Wiki on Scalaris Optimisations Questions

Data Model - Solution: Partitioning!

a) spread randomly over partitions

  • suitable for e.g. counters

b) spread list contents with hashing

  • allows efficient lookup and delete
  • suitable for e.g. lists:

mycat:cpages page1, page2, page3, page4, page5, page6, page7, page8 mycat:cpages:0 mycat:cpages:1 mycat:cpages:2 page4, page1, page5 page8, page6 page7, page2, page3 hashing

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 12/17

slide-28
SLIDE 28

Scalaris Wiki on Scalaris Optimisations Questions

Wiki on Scalaris - Architecture

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 13/17

slide-29
SLIDE 29

Scalaris Wiki on Scalaris Optimisations Questions

Wiki on Scalaris - Optimisation Results (I)

  • effects on page edits:

50 100 150 200 250 Non-Partitioned Keys Partitioned Keys average execution time in ms Scalaris Tomcat Other

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 14/17

slide-30
SLIDE 30

Scalaris Wiki on Scalaris Optimisations Questions

Wiki on Scalaris - Optimisation Results (II)

  • effects on page creates:

500 1000 1500 2000 2500 3000 Non-Partitioned Keys Partitioned Keys average execution time in ms Scalaris Tomcat Other

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 15/17

slide-31
SLIDE 31

Scalaris Wiki on Scalaris Optimisations Questions

Wiki on Scalaris - Scalability

600 800 1000 1200 1400 1600 1800 5 10 15 20 25 throughput (requests / s) nodes

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 16/17

slide-32
SLIDE 32

Scalaris Wiki on Scalaris Optimisations Questions

Questions

http://scalaris.googlecode.com

Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 17/17