apache spark in einem raspberry cluster dozenten
play

Apache Spark in einem Raspberry Cluster Dozenten Burkhard - PowerPoint PPT Presentation

May 20, 2023 •99 likes •525 views

Der Funke springt ber: Apache Spark in einem Raspberry Cluster Dozenten Burkhard Hoppenstedt atr Software GmbH & Universitt Ulm burkhard.hoppenstedt@uni-ulm.de Nicolas Kuhaupt Universitt Ulm nicolas.kuhaupt@uni-ulm.de Gliederung

  1. Der Funke springt über: Apache Spark in einem Raspberry Cluster

  2. Dozenten Burkhard Hoppenstedt atr Software GmbH & Universität Ulm burkhard.hoppenstedt@uni-ulm.de Nicolas Kuhaupt Universität Ulm nicolas.kuhaupt@uni-ulm.de

  3. Gliederung Architektur Aufbau Code Snippets Spark Cluster Geschichte Konfigurierung Ergebnisse Algorithmen-Skalierung Clusterleistung

  4. Spark Geschichte 2002 MapReduce Forschung (Google) 2004 MapReduce Paper 2006 Hadoop sortiert 1.8TB in 47.9 h 2008 Konferenz: Hadoop Summit 2010 2010 Spark Paper 2012 Apache Hadoop 1.0 2014 Apache Spark top-level Projekt 2014 Spark sortiert 100TB in 23 min, Hadoop in 72 min

  5. Powered By Yahoo! eBay Inc. Spark Amazon IBM

  6. Dateiformate Format Strukturiert Textdateien Nein JSON Halb CSV Ja SequenceFile Ja Protocol buffers Ja Object files Ja

  7. Dateiformate Format Strukturiert Sequence File Header Textdateien Nein 3 Byte (SEQ) + 1 Byte Version JSON Halb Key Class Name CSV Ja Value Class Name isCompressed SequenceFile Ja IsBlockCompressed Protocol buffers Ja Class Name Object files Ja Meta Data SyncMarker

  8. Dateiformate Format Strukturiert message Person { required string name = 1; required int32 id = 2; Textdateien Nein optional string email = 3; } JSON Halb Person john = Person.newBuilder() .setId(1234) CSV Ja .setName("John Doe") .setEmail("jdoe@example.com") .build(); SequenceFile Ja output = new FileOutputStream(args[0]); john.writeTo(output); Protocol buffers Ja Person john; fstream input(argv[1], Object files Ja ios::in | ios::binary); john.ParseFromIstream(&input); id = john.id(); name = john.name(); email = john.email();

  9. Dateiformate Format Strukturiert Java Serialisierung Textdateien Nein • Nicht standardisiert mit Hadoop-Output JSON Halb • Langsam • Schnelle Verarbeitung CSV Ja beliebiger Objekte SequenceFile Ja Protocol buffers Ja Object files Ja

  10. Word count //Python Example rdd = textFile (“… path …“) words = rdd.flatMap(lambda x: x.split (“ “)) result = words.map(lambda x: (x,1)).reduceByKey(lambda x, y: x + y)

  11. Amazon JSON {'asin': '0078764343', 'description': 'Brand new sealed!', 'price': 37.98, 'imUrl': 'http://ecx.images- amazon.com/images/I/513h6dPbwLL._SY300_.jpg', 'related': {'also_bought': ['B000TI836G', 'B003Q53VZC', 'B00EFFW0HC', 'B003VWGBC0', 'B003O6G5TW', 'B0037LTTRO', 'B002I098JE', 'B008OQTS0U', 'B005EVEODY', 'B008B3AVNE', 'B000PE0HBS', 'B00354NAYG', 'B0050SYPV2', 'B00503E8S2', 'B0050SY77E', 'B0022TNO7S', 'B0056WJA30', 'B0023CBY4E', 'B002SRSQ72', 'B005EZ5GQY', 'B004XACA60', 'B00273Z9WM', 'B004HX1QFY', 'B002I0K50U'], 'bought_together': ['B002I098JE'], 'buy_after_viewing': ['B0050SY5BM', 'B000TI836G', 'B0037LTTRO', 'B002I098JE']}, 'salesRank': {'Video Games': 28655}, 'categories': [['Video Games', 'Xbox 360', 'Games']]}

  12. FP Growth Customer Also bought John Doe Banana, Unicorn, Sword Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Banana, Unicorn, Steam Boat Stephen Hawking Banana, Sword Nyan Cat Unicorn, Sword Cpt. Jack Sparrow Banana, Sword Steve Jobs Banana, Unicorn, Sword, Razor Mr. Nobody Banana, Unicorn, Sword Banana: 6 | Unicorn: 7 | Sword : 7 | Steam Boat: 2 | Razor: 1

  13. FP Growth - resort Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword Cpt. Jack Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor Mr. Nobody Unicorn, Sword, Banana Banana: 6 | Unicorn: 7 | Sword : 7 | Steam Boat: 2 | Razor: 1

  14. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Nyan Cat Unicorn, Sword Cpt. Jack Sword, Banana Sparrow Steve Jobs Unicorn, Sword, Banana, Razor Mr. Nobody Unicorn, Sword, Banana

  15. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 1 Cpt. Jack Sword, Banana Sparrow Sword: Steve Jobs Unicorn, Sword, Banana, Razor 1 Mr. Nobody Unicorn, Sword, Banana Banana: 1

  16. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 2 Cpt. Jack Sword, Banana Sparrow Sword: Steve Jobs Unicorn, Sword, Banana, Razor 1 Mr. Nobody Unicorn, Sword, Banana Steam Boat:1 Banana: 1

  17. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 3 Cpt. Jack Sword, Banana Sparrow Sword: Steve Jobs Unicorn, Sword, Banana, Razor 2 Mr. Nobody Unicorn, Sword, Banana Steam Boat:1 Banana: 1

  18. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 4 Cpt. Jack Sword, Banana Banana: Sparrow 1 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 2 Mr. Nobody Unicorn, Sword, Banana Steam Steam Boat:1 Boat:1 Banana: 1

  19. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 4 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 1 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 2 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 1 Banana: 1

  20. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 5 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 1 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 3 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 1 Banana: 1

  21. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 5 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 2 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 3 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 2 Banana: 1

  22. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 6 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 2 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 4 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 2 Banana: 2 Razor: 1

  23. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 7 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 2 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 5 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 2 Banana: 3 Razor: 1

  24. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 7 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 2 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 5 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 2 Banana: 3 Razor: 1

  25. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 7 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 2 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 5 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 2 Banana: 3 Razor: 1 Frequent Patterns Banana

  26. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 7 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 2 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 5 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 2 Banana: 3 Razor: 1 Frequent Patterns Banana {Unicorn, Sword, Banana: 3}

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Introduction to Apache Spark Slides from: Patrick Wendell - Databricks 1 What is Sp Spark?

Introduction to Apache Spark Slides from: Patrick Wendell - Databricks 1 What is Sp Spark?

Introduction to Apache Spark Slides from: Patrick Wendell - Databricks 1 What is Sp Spark? rk? Fast ast and Ex Expres essiv sive Cluster Computing Engine Compatible with Apache Hadoop Eff fficie cient nt Usabl able Rich APIs in

792 views • 31 slides
Introduction to Apache Spark Slides from: Patrick Wendell - Databricks What hat is is Sp

Introduction to Apache Spark Slides from: Patrick Wendell - Databricks What hat is is Sp

Introduction to Apache Spark Slides from: Patrick Wendell - Databricks What hat is is Sp Spark? rk? Fast st and Expr pressiv essive Cluster Computing Engine Compatible with Apache Hadoop Efficient ficient Usa sabl ble Rich APIs

1.18k views • 25 slides
Intr Intro o to Spark to Spark and Spark and Spark SQL SQL AMP Camp 2014 Michael Armbrust -

Intr Intro o to Spark to Spark and Spark and Spark SQL SQL AMP Camp 2014 Michael Armbrust -

Intr Intro o to Spark to Spark and Spark and Spark SQL SQL AMP Camp 2014 Michael Armbrust - @michaelarmbrust What is Apache Spark? Fast and general cluster computing system, interoperable with Hadoop, included in all major distros

667 views • 43 slides
PySpark()(Data(Processing(in(Python( on(top(of(Apache(Spark Peter%Hoffmann

PySpark()(Data(Processing(in(Python( on(top(of(Apache(Spark Peter%Hoffmann

PySpark()(Data(Processing(in(Python( on(top(of(Apache(Spark Peter%Hoffmann Twi$er:(@peterhoffmann github.com/blue.yonder Spark&Overview Spark&is&a& distributed)general)purpose)cluster) engine

658 views • 24 slides
Streaming items through a cluster with Spark Streaming Tathagata TD Das @tathadas CME 323:

Streaming items through a cluster with Spark Streaming Tathagata TD Das @tathadas CME 323:

Streaming items through a cluster with Spark Streaming Tathagata TD Das @tathadas CME 323: Distributed Algorithms and Optimization Stanford, May 6, 2015 Who am I? Who am I? > Project Management Committee (PMC) member of Apache Spark >

679 views • 41 slides
Spark Code Camp Discover Spark Streaming & Spark SQL Project Overview Focus on Spark

Spark Code Camp Discover Spark Streaming & Spark SQL Project Overview Focus on Spark

Spark Code Camp Discover Spark Streaming & Spark SQL Project Overview Focus on Spark Streaming and Spark SQL Explored Streaming API of Apache Spark on Ukko Cluster Window based Stream Content Direct Stream content

221 views • 9 slides
Introducing HPC with a Raspberry Pi Cluster A practical use of and good excuse to build Raspberry

Introducing HPC with a Raspberry Pi Cluster A practical use of and good excuse to build Raspberry

Introducing HPC with a Raspberry Pi Cluster A practical use of and good excuse to build Raspberry Pi Clusters Colin Sauz <cos@aber.ac.uk> Research Software Engineer Super Computing Wales Project Aberystwyth University Overview

193 views • 18 slides
Making Big Data Processing Simple with Spark Matei Zaharia December 17, 2015 What is Apache

Making Big Data Processing Simple with Spark Matei Zaharia December 17, 2015 What is Apache

Making Big Data Processing Simple with Spark Matei Zaharia December 17, 2015 What is Apache Spark? Fast and general cluster computing engine that generalizes the MapReduce model Makes it easy and fast to process large datasets High-level

667 views • 32 slides
Unified Big Data nified Big Data Pr Processing ocessing with with Apache Spark pache Spark

Unified Big Data nified Big Data Pr Processing ocessing with with Apache Spark pache Spark

Unified Big Data nified Big Data Pr Processing ocessing with with Apache Spark pache Spark Matei Zaharia @matei_zaharia What is Apache Spark? Fast & general engine for big data processing Generalizes MapReduce model to support more

1.5k views • 52 slides
Downscaling: The Achilles heel of Autoscaling Apache Spark Clusters Prakhar Jain Sourabh Goyal

Downscaling: The Achilles heel of Autoscaling Apache Spark Clusters Prakhar Jain Sourabh Goyal

Downscaling: The Achilles heel of Autoscaling Apache Spark Clusters Prakhar Jain Sourabh Goyal Agenda Why Autoscaling on cloud? How nodes in spark cluster are used? Easy upscale, Difficult downscale Optimizations

727 views • 34 slides
Everything I know about Kubernetes I learned from a cluster of Raspberry Pis Jeff Geerling

Everything I know about Kubernetes I learned from a cluster of Raspberry Pis Jeff Geerling

Everything I know about Kubernetes I learned from a cluster of Raspberry Pis Jeff Geerling (geerlingguy) Builder Track - www.pidramble.com http://www.shockinglydelicious.com/dorothys-fresh-raspberry-pie-recipe/ The Big Question What can I do with

821 views • 25 slides
Large-Scale Data Engineering Spark and MLLIB event.cwi.nl/lsde OVERVIEW OF SPARK

Large-Scale Data Engineering Spark and MLLIB event.cwi.nl/lsde OVERVIEW OF SPARK

Large-Scale Data Engineering Spark and MLLIB event.cwi.nl/lsde OVERVIEW OF SPARK event.cwi.nl/lsde What is Spark? Fast and expressive cluster computing system interoperable with Apache Hadoop Up to 100 faster Improves efficiency

548 views • 36 slides
Building a Scalable Recommender System with Apache Spark, Apache Kafka and Elasticsearch About

Building a Scalable Recommender System with Apache Spark, Apache Kafka and Elasticsearch About

Nov / 14 / 16 Nick Pentreath Building a Scalable Recommender System with Apache Spark, Apache Kafka and Elasticsearch About @MLnick Principal Engineer, IBM Apache Spark PMC Focused on machine learning Author of Machine Learning

666 views • 53 slides
Apache Spark CS240A T Yang Some of them are based on P. Wendells Spark slides Parallel

Apache Spark CS240A T Yang Some of them are based on P. Wendells Spark slides Parallel

Apache Spark CS240A T Yang Some of them are based on P. Wendells Spark slides Parallel Processing using Spark+Hadoop Hadoop: Distributed file system that connects machines. Mapreduce: parallel programming style built on a Hadoop

840 views • 36 slides
Big Data Processing with Apache Spark Jay Urbain, PhD Credits: Resilient Distributed Datasets

Big Data Processing with Apache Spark Jay Urbain, PhD Credits: Resilient Distributed Datasets

Big Data Processing with Apache Spark Jay Urbain, PhD Credits: Resilient Distributed Datasets Resilient Distributed Datasets A Fault-T A Fault-Tolerant Abstraction for In-Memory Cluster Computing olerant Abstraction for In-Memory Cluster

1.32k views • 67 slides
ROOT4J / SPARK-ROOT: ROOT I/O for JVM and Applications for Apache Spark V. Khristenko 1 J.

ROOT4J / SPARK-ROOT: ROOT I/O for JVM and Applications for Apache Spark V. Khristenko 1 J.

Introduction Functionality Examples Summary ROOT4J / SPARK-ROOT: ROOT I/O for JVM and Applications for Apache Spark V. Khristenko 1 J. Pivarski 2 1 Department of Physics The University of Iowa 2 Princeton University - DIANA ROOT I/O Workshop,

544 views • 19 slides
Loops Python - Nick Reynolds September 31, 2017 Loops This Class Loop Types

Loops Python - Nick Reynolds September 31, 2017 Loops This Class Loop Types

Loops Python - Nick Reynolds September 31, 2017 Loops This Class Loop Types Assignment 1 Loops Whats next on the list? Whats a loop? I have a shopping list: Whats next on the list? Apple Banana

274 views • 14 slides
CSCI 2132 Software Development Lecture 26: Writing Large Programs Instructor: Vlado Keselj

CSCI 2132 Software Development Lecture 26: Writing Large Programs Instructor: Vlado Keselj

CSCI 2132 Software Development Lecture 26: Writing Large Programs Instructor: Vlado Keselj Faculty of Computer Science Dalhousie University 5-Nov-2018 (26) CSCI 2132 1 Previous Lecture A common mistake with VLA declarations More

197 views • 8 slides
Event Calculus and Answer Set Programming Joohyung Lee Automated Reasoning Group School of

Event Calculus and Answer Set Programming Joohyung Lee Automated Reasoning Group School of

Event Calculus and Answer Set Programming Joohyung Lee Automated Reasoning Group School of Computing, Informatics and Decision Systems Engineering Arizona State University KOCSEA 2009 (Based on IJCAI 2007, 2009 Papers) Knowledge in Action

568 views • 32 slides
ECE 20875 Python for Data Science Milind Kulkarni and Chris Brinton Data Structures coding in

ECE 20875 Python for Data Science Milind Kulkarni and Chris Brinton Data Structures coding in

ECE 20875 Python for Data Science Milind Kulkarni and Chris Brinton Data Structures coding in python Standard Integrated Development Environments (IDEs) IDLE: Pythons own, basic IDE PyCharm: Code completion, unit tests,

372 views • 10 slides
Mathematical Logics Description Logic: Introduction Fausto Giunchiglia and Mattia Fumagallli

Mathematical Logics Description Logic: Introduction Fausto Giunchiglia and Mattia Fumagallli

Mathematical Logics Description Logic: Introduction Fausto Giunchiglia and Mattia Fumagallli University of T rento *Originally by Luciano Serafini and Chiara Ghidini Modified by Fausto Giunchiglia and Mattia Fumagalli 1 Mental Model

625 views • 36 slides
15-251 Great Theoretical Ideas in Computer Science Lecture 5: Cantors Legacy September 15th,

15-251 Great Theoretical Ideas in Computer Science Lecture 5: Cantors Legacy September 15th,

15-251 Great Theoretical Ideas in Computer Science Lecture 5: Cantors Legacy September 15th, 2015 Poll Select the ones that apply to you: - I know what an uncountable set means. - I know Cantors diagonalization argument. - I used to

852 views • 57 slides
For Friday Read chapter 12, sections 3-5 None Program 2 Any questions? Plan-Space

For Friday Read chapter 12, sections 3-5 None Program 2 Any questions? Plan-Space

For Friday Read chapter 12, sections 3-5 None Program 2 Any questions? Plan-Space Planners Plan-space planners search through the space of partial plans, which are sets of actions that may not be totally ordered.

654 views • 54 slides
Contents Introduction Market demand study on fresh products and Research: Market demand study

Contents Introduction Market demand study on fresh products and Research: Market demand study

2/1/2013 Contents Introduction Market demand study on fresh products and Research: Market demand study on Research Overview derived products of banana, jackfruit, and cashew fresh products and derived products nut in Phnom Penh market

756 views • 4 slides

More recommend