Riak to the Rescue Migrating Big Data Big Data. Buzzwords. Dont - - PowerPoint PPT Presentation

▶
riak to the rescue migrating big data big data buzzwords
SMART_READER_LITE
LIVE PREVIEW

Riak to the Rescue Migrating Big Data Big Data. Buzzwords. Dont - - PowerPoint PPT Presentation

Jan 18, 2024 578 likes •1.71k views

Riak to the Rescue Migrating Big Data Big Data. Buzzwords. Dont believe the Hype. Who am I? Support Development SysAdmin Managing Operations 8 Ops Engineers Operations 4 Offices 650 physical Operations 200 virtual 3 data centres

slide-1
SLIDE 1

Riak to the Rescue Migrating Big Data

slide-2
SLIDE 2

Big Data.

slide-3
SLIDE 3

Buzzwords.

slide-4
SLIDE 4

Don’t believe the Hype.

slide-5
SLIDE 5

Who am I?

Support Development SysAdmin Managing Operations

slide-6
SLIDE 6

Operations

8 Ops Engineers 4 Offices

slide-7
SLIDE 7

Operations

650 physical 200 virtual 3 data centres

slide-8
SLIDE 8

Contact

  • Based in Berlin
  • twitter: @geidies
  • seb@meltwater.com
  • http://underthehood.meltwater.com/
slide-9
SLIDE 9

Migrating Big Data

  • Meltwater
  • Social Media Data Volumes
  • Try and Fail
  • Analyse and Succeed
  • Things to Learn
slide-10
SLIDE 10

Meltwater

slide-11
SLIDE 11

Meltwater

News Monitoring

slide-12
SLIDE 12

Paper-Clip

Read News Cut and Glue Telefax

slide-13
SLIDE 13

Meltwater News

Crawl the Web Match new Articles Morning Report Analytics UI

slide-14
SLIDE 14

Products

PR m|news m|press Marketing m|buzz / engage icerocket

slide-15
SLIDE 15

SaaS

Subscription model 24,000 clients

slide-16
SLIDE 16
slide-17
SLIDE 17

riak

  • Open Source
  • Dynamo Paper
  • Erlang
slide-18
SLIDE 18

2.0

OMG, OMG!!

slide-19
SLIDE 19

thanks, basho.

slide-20
SLIDE 20

Meltwater Buzz

slide-21
SLIDE 21

m|news m|buzz

20 D/s - 8400 S/s 600 D/s - ??

slide-22
SLIDE 22 Interesting Shtuff By Joan Doe - 2014/05/06 Something amazing happened yesterday. It was more interesting than what happened the day before, but maybe it won’t change the events that are about to come tomorrow. What does Lorem ipsum dolor really mean? we know it is not real latin. But it looks pretty good, since the characters are evenly distributed. I once tried translating it, and it really doesn’t make any sense. Talking here is amazing. Wow, Denmark - it’s actually really cool being in Aarhus. You should have a chat with me after the talk if you have further questions. Please don’t hesitate to say hi. If you’re in Berlin, come stop by the meltwater office for a chat about big data, a cup of coffee, a game of table tennis of foosball. You can find us at Rotherstraße 22 in Friedrichshain.
slide-23
SLIDE 23
slide-24
SLIDE 24

Social Media

  • 140 Characters
  • Pages Long
slide-25
SLIDE 25

Social Media

  • Metadata
  • Location
  • Followers
  • Threads
slide-26
SLIDE 26

Social Media

  • Extracted Metadata
  • sentiment
  • named entities
  • intent
  • Editorial vs. Opinion vs. Both
slide-27
SLIDE 27

m|buzz version 1

  • Buzzgain
  • php, MySQL, SolR
slide-28
SLIDE 28
slide-29
SLIDE 29

Attention!

slide-30
SLIDE 30

Your Use Case

Research Evaluate Test

slide-31
SLIDE 31

m|buzz version 2

Scalability, Features, Buzzwords!

slide-32
SLIDE 32
slide-33
SLIDE 33
slide-34
SLIDE 34 – Jamie Zawinski

“Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.”

slide-35
SLIDE 35
slide-36
SLIDE 36

Requirements

  • Fail-Safety
  • High Availability
  • A Lot of Unstructured Data
  • Near-Real-Time Indexing
  • Time-Based Ordering instead of Relevancy
slide-37
SLIDE 37

m|buzz version 2

  • Hadoop Ecosystem
  • Apache Projects
slide-38
SLIDE 38

m|buzz version 2

fetcher fetcher fetcher HBase HDFS Katta API M-R hourly daily
slide-39
SLIDE 39

It’s a trap!

  • buzzwords
  • commodity hardware
  • scale
slide-40
SLIDE 40
slide-41
SLIDE 41
  • Build upon lucene
  • Master -> Worker -> Client
  • communication through zookeeper
  • multiple index copies
  • copied from HDFS -> local disk
slide-42
SLIDE 42
  • OK in theory.
  • Out Of Memory
  • Garbage Collection Hell
  • version 0.62 - odd bugs.
slide-43
SLIDE 43
slide-44
SLIDE 44
slide-45
SLIDE 45

0.20.5

slide-46
SLIDE 46
slide-47
SLIDE 47
slide-48
SLIDE 48 keyspace region key n -> key o split region key a -> key c
slide-49
SLIDE 49
  • ROOT-
.META.
slide-50
SLIDE 50

Fail-Safety

slide-51
SLIDE 51

Fail-Safety

Does NOT mean High Availability Data on a Single Node

slide-52
SLIDE 52

Minutes.

55,000 posts / minute

slide-53
SLIDE 53

Funny Regions

Overlapping Gaps Negative Length

slide-54
SLIDE 54

Funny Regions

REGION => {NAME => 'buzz_data, 1333073443000_62gfsHBsE5vNSz168ByvP5tDPu0A,1333173530871', STARTKEY => '1333073443000_62gfsHBsE5vNSz168ByvP5tDPu0A', ENDKEY => '1326306499000_evKK670FSV9MAas2CMZAr41wLm0A', ENCODED => 128988498, TABLE => {{NAME => 'buzz_data', FAMILIES => [{NAME => 'fm_contents',VERSIONS => '1', COMPRESSION => 'LZO', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'fm_input_info', VERSIONS=> '1', COMPRESSION => 'LZO', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'fm_metadata', VERSIONS => '1', COMPRESSION => 'LZO', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'fm_output_info', VERSIONS => '1', COMPRESSION => 'LZO', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
slide-55
SLIDE 55

HBase

  • .META. corruption
  • Data Unavailability
  • Slow Start of Regions
  • Full Cluster Restarts Slow
  • Hotspots
slide-56
SLIDE 56

Good News!

NameNode never crashed. Great.

slide-57
SLIDE 57

Changes…

…do you speak it?

slide-58
SLIDE 58
slide-59
SLIDE 59 HBase HDFS Katta M-R hourly daily rabbitMQ

m|buzz version 2.5

fetcher fetcher fetcher API fetcher fetcher fetcher enrichment HBase HDFS Katta M-R hourly daily couchbase MapR ¡Distribu-on
slide-60
SLIDE 60
slide-61
SLIDE 61
slide-62
SLIDE 62
  • Message Queue System
  • Erlang
  • Redundant Setup, fail-safe and high-available
  • Write to Exchange -> Distribute to Multiple Queues
slide-63
SLIDE 63 HBase HDFS Katta M-R hourly daily rabbitMQ

m|buzz version 2.5

fetcher fetcher fetcher API fetcher fetcher fetcher enrichment HBase HDFS Katta M-R hourly daily couchbase MapR ¡Distribu-on
slide-64
SLIDE 64
slide-65
SLIDE 65
slide-66
SLIDE 66

First Read Wins

Parallel Reads: couchbase vanilla HBase MapR HBase

slide-67
SLIDE 67

couchbase scales!

…to four weeks of data. 2.2B entries TTL

slide-68
SLIDE 68

Are we there yet?

slide-69
SLIDE 69
slide-70
SLIDE 70

Options

Pro Con custom WAL works safely doesn’t scale (easily) MySQL cluster A lot of experience hitting limit of scaling commercial Object storage commercial support up-front investment riak
slide-71
SLIDE 71

Requirements

✓ High Availability ✓ Data Safety ✓ Scalability ? Range Scans or TTL to limit data
slide-72
SLIDE 72

riak

Key-Value model Objects in Buckets

slide-73
SLIDE 73
slide-74
SLIDE 74
slide-75
SLIDE 75
slide-76
SLIDE 76 – riak documentation “While there are mechanisms such as Vector Clocks to help deal with these issues, if your application requires the kind of strong consistency found in ACID systems, Riak may not be a good fit.”
slide-77
SLIDE 77 rabbitMQ

m|buzz version 2.6

fetcher fetcher fetcher API fetcher fetcher fetcher enrichment HBase HDFS Katta M-R couchbase riak elasticsearch
slide-78
SLIDE 78

Commodity Hardware

  • HP DL360 G1
  • 4c CPU
  • 32GB RAM
  • 1x 2TB 7.2k spinner
  • …37 of those.
slide-79
SLIDE 79

Configuration

  • levelDB
  • erlang VM
  • Map-Reduce
slide-80
SLIDE 80

Future-Proof

Setting the ring-size to… 2048.

slide-81
SLIDE 81 – riak mailing list

“2048 is definitely the upper bound of what we recommend, but with the right amount of machines, this can work.”

slide-82
SLIDE 82 – riak mailing list re-niced

“Are you guys insane? We didn’t even know that was possible!!”

slide-83
SLIDE 83

Numbers

  • 37 nodes
  • 55,000 writes per minute
  • 350,000 reads per minute
  • 1.8TB data per node
slide-84
SLIDE 84

Hey, wait.

A good three weeks?

slide-85
SLIDE 85

Let’s do it.

parallel reads gather numbers stability speed

slide-86
SLIDE 86

riak is slow.

but consistent, and massively parallel.

slide-87
SLIDE 87

riak is slow.

riak is not as fast as a memory-only key-value store.

slide-88
SLIDE 88

stability over speed.

slide-89
SLIDE 89

stability

  • availability during
  • node failures
  • upgrades
  • configuration updates
slide-90
SLIDE 90

Search

slide-91
SLIDE 91 rabbitMQ

m|buzz version 3

fetcher fetcher fetcher API fetcher fetcher fetcher enrichment couchbase riak elasticsearch
slide-92
SLIDE 92

Naming Things

slide-93
SLIDE 93 rabbitMQ

m|buzz version 3

fetcher fetcher fetcher API fetcher fetcher fetcher enrichment couchbase riak elasticsearch

ES/R

slide-94
SLIDE 94

Putting it live

slide-95
SLIDE 95

Still live

  • 58,000,000,000 key-value pairs written
  • 365,000,000,000 reads
  • 3.5ms mean (8ms 95th, 35ms 99th, 2s 100)
slide-96
SLIDE 96

Monitoring

  • Input “valves”
  • throughput of any intermediate processing step
  • output valves
  • distribution of data across cluster
  • handovers of data within the cluster
slide-97
SLIDE 97

Dashboards

And APIs.

slide-98
SLIDE 98
slide-99
SLIDE 99
slide-100
SLIDE 100
slide-101
SLIDE 101
slide-102
SLIDE 102
slide-103
SLIDE 103

necessary but not sufficient

dashboard API fool-safe performance configuration good documentation

slide-104
SLIDE 104
slide-105
SLIDE 105

Summary

slide-106
SLIDE 106

Buzzwords

Be amazed. Doubt. Evaluate.

slide-107
SLIDE 107

Hardware

There is no such thing as “too much RAM”

slide-108
SLIDE 108

Scale

You’ll need it.

slide-109
SLIDE 109

Configuration Management

who’s the master of puppet?

slide-110
SLIDE 110

Monitoring

looks exciting even when things work.

slide-111
SLIDE 111

Time.

Operational Stability beats Features when it comes to Big A Lot of Data.

slide-112
SLIDE 112

Thank you.

@geidies - seb@meltwater.com http://underthehood.meltwater.com/ slides w/ notes on github.com/geidies/slides