Scalaris: Scalable Web Applications with a Transactional Key-Value - PowerPoint PPT Presentation

Scalaris Wiki on Scalaris Optimisations Questions Scalaris: Scalable Web Applications with a Transactional Key-Value Store Nico Kruber Michael Berlin Zuse Institut Berlin Parallel and Distributed Systems 5th June @ Berlin Buzzwords 2012 The research leading to these results has received funding from the European Union’s Seventh Frame- work Programme (FP7/2007-2013 ) under grant agree- ments n ◦ 258862 (4CaaSt) and n ◦ 257438 (Contrail). Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 1/17

Scalaris Wiki on Scalaris Optimisations Questions Outline Scalaris Introduction Scalability Wiki on Scalaris MediaWiki Introduction Data Model Optimisations Strategies Results Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 2/17

Scalaris Wiki on Scalaris Optimisations Questions Scalaris - A Transactional Key-Value Store Relational DB most NoSQL stores • multiple relations/tables • key-value/document mapping • SQL • scalable • Joins • fault-tolerant (replication) • Transactions • eventual consistency Scalaris • key-value pairs • select, update/insert • scalable • fault-tolerant (replication) • transaction support • strong consistency Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 3/17

Scalaris Wiki on Scalaris Optimisations Questions Scalaris Implementation • Scalaris: ~23000 lines of Erlang code • 2200 for transactions • APIs: • Java library (11500 lines) • Python library (1400 lines) • Ruby library (700 lines) • HTTP-JSON interface • http://scalaris.googlecode.com Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 4/17

Scalaris Wiki on Scalaris Optimisations Questions Scalaris - Scalability Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 5/17

Scalaris Wiki on Scalaris Optimisations Questions Wikitext A-Z Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 6/17

Scalaris Wiki on Scalaris Optimisations Questions Wikitext A-Z page title including namespace Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 6/17

Scalaris Wiki on Scalaris Optimisations Questions Wikitext A-Z page title including namespace link to another page Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 6/17

Scalaris Wiki on Scalaris Optimisations Questions Wikitext A-Z page title including namespace link to another page template inclusion Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 6/17

Scalaris Wiki on Scalaris Optimisations Questions Wikitext A-Z page title including namespace link to another page template inclusion category the page belongs to Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 6/17

Scalaris Wiki on Scalaris Optimisations Questions Wiki Operations Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 7/17

Scalaris Wiki on Scalaris Optimisations Questions Wiki Operations show page: read revisions Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 7/17

Scalaris Wiki on Scalaris Optimisations Questions Wiki Operations revision history overview show page: read revisions Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 7/17

Scalaris Wiki on Scalaris Optimisations Questions Wiki Operations search for articles revision history overview show page: read revisions Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 7/17

Scalaris Wiki on Scalaris Optimisations Questions Wiki Operations search for articles revision history overview show page: read revisions redirect to random article Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 7/17

Scalaris Wiki on Scalaris Optimisations Questions Wiki Operations search for articles revision history overview show page: read revisions redirect to random article backlinks (“what links here”) Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 7/17

Scalaris Wiki on Scalaris Optimisations Questions Wiki Operations search for articles revision history overview show page: read revisions redirect to random article backlinks (“what links here”) save new revision Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 7/17

Scalaris Wiki on Scalaris Optimisations Questions Editing Pages Do atomically: 1. store new revision 2. update page object 3. add new backlinks 4. remove old backlinks 5. add to new categories 6. remove from old categories 7. if new page: a) add to page list b) increase page counter Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 8/17

Scalaris Wiki on Scalaris Optimisations Questions Editing Pages Do atomically: 1. store new revision 2. update page object 3. add new backlinks 4. remove old backlinks 5. add to new categories 6. remove from old categories 7. if new page: a) add to page list b) increase page counter Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 8/17

Scalaris Wiki on Scalaris Optimisations Questions Editing Pages Do atomically: 1. store new revision 2. update page object 3. add new backlinks 4. remove old backlinks 5. add to new categories 6. remove from old categories 7. if new page: a) add to page list b) increase page counter Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 8/17

Scalaris Wiki on Scalaris Optimisations Questions Editing Pages Do atomically: 1. store new revision 2. update page object 3. add new backlinks 4. remove old backlinks 5. add to new categories 6. remove from old categories 7. if new page: a) add to page list b) increase page counter Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 8/17

Scalaris Wiki on Scalaris Optimisations Questions Editing Pages Do atomically: 1. store new revision 2. update page object 3. add new backlinks 4. remove old backlinks 5. add to new categories 6. remove from old categories 7. if new page: a) add to page list b) increase page counter Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 8/17

Scalaris Wiki on Scalaris Optimisations Questions Editing Pages Do atomically: 1. store new revision 2. update page object 3. add new backlinks 4. remove old backlinks 5. add to new categories 6. remove from old categories 7. if new page: a) add to page list b) increase page counter Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 8/17

Scalaris Wiki on Scalaris Optimisations Questions MediaWiki Schema adapted from https://commons.wikimedia.org/wiki/File:MediaWiki_database_schema_1-19_%28r102798%29.svg Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 9/17

Scalaris Wiki on Scalaris Optimisations Questions Wiki on Scalaris - Data Layout key content pages:<namespace> list of pages in <namespace> pages:<namespace>:count number of pages in <namespace> <title>:page page information (JSON object) <title>:rev:<id> single page revision (JSON object) <title>:revs list of revision info for history <title>:blpages list of page titles linking to <title> <cat_title>:cpages list of page titles inside a category <cat_title>:cpages:count number of pages inside a category Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 10/17

Scalaris Wiki on Scalaris Optimisations Questions Data Model - Issues? • non-scalable data model! • Problem: page lists, counters Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 11/17

Scalaris Wiki on Scalaris Optimisations Questions Data Model - Solution: Partitioning! a) spread randomly over partitions • suitable for e.g. counters b) spread list contents with hashing • allows efficient lookup and delete • suitable for e.g. lists: mycat:cpages page1, page2, page3, page4, page5, page6, page7, page8 hashing page4, page1, page5 page8, page6 page7, page2, page3 mycat:cpages:0 mycat:cpages:1 mycat:cpages:2 Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 12/17

Scalaris Wiki on Scalaris Optimisations Questions Wiki on Scalaris - Architecture Nico Kruber, Michael Berlin Scalaris: Scalable Web Applications with a Transactional Key-Value Store 13/17