Apache Spark in einem Raspberry Cluster Dozenten Burkhard - - PowerPoint PPT Presentation

▶
apache spark in einem raspberry cluster dozenten
SMART_READER_LITE
LIVE PREVIEW

Apache Spark in einem Raspberry Cluster Dozenten Burkhard - - PowerPoint PPT Presentation

May 20, 2023 99 likes •528 views

Der Funke springt ber: Apache Spark in einem Raspberry Cluster Dozenten Burkhard Hoppenstedt atr Software GmbH & Universitt Ulm burkhard.hoppenstedt@uni-ulm.de Nicolas Kuhaupt Universitt Ulm nicolas.kuhaupt@uni-ulm.de Gliederung

slide-1
SLIDE 1

Der Funke springt über:

Apache Spark in einem Raspberry Cluster

slide-2
SLIDE 2

Dozenten

Burkhard Hoppenstedt

atr Software GmbH & Universität Ulm burkhard.hoppenstedt@uni-ulm.de

Nicolas Kuhaupt

Universität Ulm nicolas.kuhaupt@uni-ulm.de

slide-3
SLIDE 3

Gliederung

Spark

Architektur Geschichte Code Snippets

Cluster

Aufbau Konfigurierung

Ergebnisse

Algorithmen-Skalierung Clusterleistung

slide-4
SLIDE 4

Spark Geschichte

MapReduce Forschung (Google) MapReduce Paper Hadoop sortiert 1.8TB in 47.9 h Konferenz: Hadoop Summit 2010 Spark Paper Apache Hadoop 1.0 Apache Spark top-level Projekt Spark sortiert 100TB in 23 min, Hadoop in 72 min 2002 2004 2006 2008 2010 2012 2014 2014

slide-5
SLIDE 5

Powered By

eBay Inc. Spark Amazon IBM Yahoo!

slide-6
SLIDE 6

Dateiformate

Format Strukturiert Textdateien Nein JSON Halb CSV Ja SequenceFile Ja Protocol buffers Ja Object files Ja

slide-7
SLIDE 7

Dateiformate

Format Strukturiert Textdateien Nein JSON Halb CSV Ja SequenceFile Ja Protocol buffers Ja Object files Ja Sequence File Header

3 Byte (SEQ) + 1 Byte Version Key Class Name Value Class Name isCompressed IsBlockCompressed Class Name Meta Data SyncMarker

slide-8
SLIDE 8

Dateiformate

Format Strukturiert Textdateien Nein JSON Halb CSV Ja SequenceFile Ja Protocol buffers Ja Object files Ja

message Person { required string name = 1; required int32 id = 2;

  • ptional string email = 3;

} Person john = Person.newBuilder() .setId(1234) .setName("John Doe") .setEmail("jdoe@example.com") .build();

  • utput = new FileOutputStream(args[0]);

john.writeTo(output); Person john; fstream input(argv[1], ios::in | ios::binary); john.ParseFromIstream(&input); id = john.id(); name = john.name(); email = john.email();

slide-9
SLIDE 9

Dateiformate

Format Strukturiert Textdateien Nein JSON Halb CSV Ja SequenceFile Ja Protocol buffers Ja Object files Ja

Java Serialisierung

  • Nicht standardisiert mit

Hadoop-Output

  • Langsam
  • Schnelle Verarbeitung

beliebiger Objekte

slide-10
SLIDE 10

Word count

//Python Example rdd = textFile(“…path…“) words = rdd.flatMap(lambda x: x.split(“ “)) result = words.map(lambda x: (x,1)).reduceByKey(lambda x, y: x + y)

slide-11
SLIDE 11

Amazon JSON

{'asin': '0078764343', 'description': 'Brand new sealed!', 'price': 37.98, 'imUrl': 'http://ecx.images- amazon.com/images/I/513h6dPbwLL._SY300_.jpg', 'related': {'also_bought': ['B000TI836G', 'B003Q53VZC', 'B00EFFW0HC', 'B003VWGBC0', 'B003O6G5TW', 'B0037LTTRO', 'B002I098JE', 'B008OQTS0U', 'B005EVEODY', 'B008B3AVNE', 'B000PE0HBS', 'B00354NAYG', 'B0050SYPV2', 'B00503E8S2', 'B0050SY77E', 'B0022TNO7S', 'B0056WJA30', 'B0023CBY4E', 'B002SRSQ72', 'B005EZ5GQY', 'B004XACA60', 'B00273Z9WM', 'B004HX1QFY', 'B002I0K50U'], 'bought_together': ['B002I098JE'], 'buy_after_viewing': ['B0050SY5BM', 'B000TI836G', 'B0037LTTRO', 'B002I098JE']}, 'salesRank': {'Video Games': 28655}, 'categories': [['Video Games', 'Xbox 360', 'Games']]}

slide-12
SLIDE 12

FP Growth

Customer Also bought John Doe Banana, Unicorn, Sword Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Banana, Unicorn, Steam Boat Stephen Hawking Banana, Sword Nyan Cat Unicorn, Sword

  • Cpt. Jack Sparrow

Banana, Sword Steve Jobs Banana, Unicorn, Sword, Razor

  • Mr. Nobody

Banana, Unicorn, Sword Banana: 6 | Unicorn: 7 | Sword : 7 | Steam Boat: 2 | Razor: 1

slide-13
SLIDE 13

FP Growth - resort

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack Sparrow

Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana Banana: 6 | Unicorn: 7 | Sword : 7 | Steam Boat: 2 | Razor: 1

slide-14
SLIDE 14

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

{ }

slide-15
SLIDE 15

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Unicorn: 1 Banana: 1 { } Sword: 1

slide-16
SLIDE 16

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Unicorn: 2 Steam Boat:1 Banana: 1 { } Sword: 1

slide-17
SLIDE 17

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Unicorn: 3 Steam Boat:1 Banana: 1 { } Sword: 2

slide-18
SLIDE 18

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Unicorn: 4 Steam Boat:1 Banana: 1 { } Sword: 2 Banana: 1 Steam Boat:1

slide-19
SLIDE 19

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Unicorn: 4 Steam Boat:1 Banana: 1 { } Sword: 2 Banana: 1 Banana: 1 Sword: 1 Steam Boat:1

slide-20
SLIDE 20

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Unicorn: 5 Steam Boat:1 Banana: 1 { } Sword: 3 Banana: 1 Banana: 1 Sword: 1 Steam Boat:1

slide-21
SLIDE 21

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Unicorn: 5 Steam Boat:1 Banana: 1 { } Sword: 3 Banana: 1 Banana: 2 Sword: 2 Steam Boat:1

slide-22
SLIDE 22

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Unicorn: 6 Steam Boat:1 Razor: 1 { } Sword: 4 Banana: 1 Banana: 2 Sword: 2 Banana: 2 Steam Boat:1

slide-23
SLIDE 23

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Unicorn: 7 Steam Boat:1 Razor: 1 { } Sword: 5 Banana: 1 Banana: 2 Sword: 2 Banana: 3 Steam Boat:1

slide-24
SLIDE 24

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Unicorn: 7 Steam Boat:1 Razor: 1 { } Sword: 5 Banana: 1 Banana: 2 Sword: 2 Banana: 3 Steam Boat:1

slide-25
SLIDE 25

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Frequent Patterns Banana

Unicorn: 7 Steam Boat:1 Razor: 1 { } Sword: 5 Banana: 1 Banana: 2 Sword: 2 Banana: 3 Steam Boat:1

slide-26
SLIDE 26

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Frequent Patterns Banana {Unicorn, Sword, Banana: 3}

Unicorn: 7 Steam Boat:1 Razor: 1 { } Sword: 5 Banana: 1 Banana: 2 Sword: 2 Banana: 3 Steam Boat:1

slide-27
SLIDE 27

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Frequent Patterns Banana {Unicorn, Sword, Banana: 3} , {Unicorn, Banana:1}

Unicorn: 7 Steam Boat:1 Razor: 1 { } Sword: 5 Banana: 1 Banana: 2 Sword: 2 Banana: 3 Steam Boat:1

slide-28
SLIDE 28

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Frequent Patterns Banana {Unicorn, Sword, Banana: 3} , {Unicorn, Banana:1} , {Sword, Banana: 2}

Unicorn: 7 Steam Boat:1 Razor: 1 { } Sword: 5 Banana: 1 Banana: 2 Sword: 2 Banana: 3 Steam Boat:1

slide-29
SLIDE 29

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Unicorn: 7 Steam Boat:1 Razor: 1 { } Sword: 5 Banana: 1 Banana: 2 Sword: 2 Banana: 3 Steam Boat:1

Frequent Patterns Banana {Unicorn, Sword, Banana: 3} , {Unicorn, Banana:1} , {Sword, Banana: 2}

slide-30
SLIDE 30

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Unicorn: 7 Steam Boat:1 Razor: 1 { } Sword: 5 Banana: 1 Banana: 2 Sword: 2 Banana: 3 Steam Boat:1

Frequent Patterns Banana {Unicorn, Sword, Banana: 3} , {Unicorn, Banana:1} , {Sword, Banana: 2}, {Unicorn, Banana: 4}

slide-31
SLIDE 31

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Unicorn: 7 Steam Boat:1 Razor: 1 { } Sword: 5 Banana: 1 Banana: 2 Sword: 2 Banana: 3 Steam Boat:1

Frequent Patterns Banana {Unicorn, Sword, Banana: 3} , {Unicorn, Banana:1} , {Sword, Banana: 2}, {Unicorn, Banana: 4} Sword {Unicorn, Sword: 5} Steam boat {Unicorn, Steam Boat: 1}, {Unicorn, Banana, Steam Boat: 1}, {Unicorn, Steam Boat : 2} Razor {Unicorn, Sword, Banana, Razor:1}

slide-32
SLIDE 32

Frequent Patterns Banana {Unicorn, Sword, Banana: 3} {Sword, Banana: 2}, {Unicorn, Banana: 4} Sword {Unicorn, Sword: 5} Steam boat {Unicorn, Steam Boat : 2} Razor

Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword

  • Cpt. Jack

Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor

  • Mr. Nobody

Unicorn, Sword, Banana

Frequent Patterns Banana {Unicorn, Sword, Banana: 3} , {Unicorn, Banana:1} , {Sword, Banana: 2}, {Unicorn, Banana: 4} Sword {Unicorn, Sword: 5} Steam boat {Unicorn, Steam Boat: 1}, {Unicorn, Banana, Steam Boat: 1}, {Unicorn, Steam Boat : 2} Razor {Unicorn, Sword, Banana, Razor:1}

min support:2

Unicorn: 7 Steam Boat:1 Razor: 1 { } Sword: 5 Banana: 1 Banana: 2 Sword: 2 Banana: 3 Steam Boat:1

slide-33
SLIDE 33

20000 40000 60000 80000 100000 120000 140000 160000 180000 200000 10% 30% 50% 70% 90%

Scaling of FP Growth

slide-34
SLIDE 34

10000 20000 30000 40000 50000 60000 70000 80000 90000 10% 40% 70% 100%

Scaling of KMeans

slide-35
SLIDE 35

Customer Money Spend (€) Time (h) John Doe 2 4 Mickey Mouse 4 5 Donald Trump 10 10 John Cena 1 10 Stephen Hawking 2 Nyan Cat 2 3

  • Cpt. Jack Sparrow

2 6 Steve Jobs 5 8

  • Mr. Nobody

7 2

slide-36
SLIDE 36
slide-37
SLIDE 37
slide-38
SLIDE 38
slide-39
SLIDE 39

„John Cena and Donald Trump shop similar to Steve Jobs“

slide-40
SLIDE 40

Fazit

slide-41
SLIDE 41

Quellen

  • Sequence Files Header: http://hadooptutorial.info/hadoop-sequence-files-example/ [30.01.2017]
  • Protocol Buffers: https://developers.google.com/protocol-buffers/ [30.01.2017]
  • Raspberry Pi 3: https://www.raspberrypi.org/products/raspberry-pi-3-model-b/ [31.01.2017]