Scala & Spark PTT18/19 Prof. Dr. Ralf Lmmel Msc. Johannes - - PowerPoint PPT Presentation

▶
scala spark ptt18 19
SMART_READER_LITE
LIVE PREVIEW

Scala & Spark PTT18/19 Prof. Dr. Ralf Lmmel Msc. Johannes - - PowerPoint PPT Presentation

Jan 02, 2023 353 likes •971 views

Scala & Spark PTT18/19 Prof. Dr. Ralf Lmmel Msc. Johannes Hrtel Msc. Marcel Heinz (C) 2018, SoftLang Team, University of Koblenz-Landau What is Scala? - Scala is a general purpose programming language. - Scala provides support for

slide-1
SLIDE 1

(C) 2018, SoftLang Team, University of Koblenz-Landau

Scala & Spark PTT18/19

  • Prof. Dr. Ralf Lämmel
  • Msc. Johannes Härtel
  • Msc. Marcel Heinz
slide-2
SLIDE 2

(C) 2018, SoftLang Team, University of Koblenz-Landau

What is Scala?

  • Scala is a general purpose programming language.
  • Scala provides support for functional programming
  • Scala has a strong static type system.
  • Scala source code is compiled to Java bytecode that runs on the JVM.
  • Scala provides language interoperability with Java.

This is hello world:

[wik]

slide-3
SLIDE 3

(C) 2018, SoftLang Team, University of Koblenz-Landau

Trending Scala Projects

slide-4
SLIDE 4

(C) 2018, SoftLang Team, University of Koblenz-Landau

Trending Scala Projects

Message-driven Applications

slide-5
SLIDE 5

(C) 2018, SoftLang Team, University of Koblenz-Landau

Trending Scala Projects

Message-driven Applications A Distributed Streaming Platform

slide-6
SLIDE 6

(C) 2018, SoftLang Team, University of Koblenz-Landau

Trending Scala Projects

Message-driven Applications A Distributed Streaming Platform High Velocity Web Framework

slide-7
SLIDE 7

(C) 2018, SoftLang Team, University of Koblenz-Landau

Trending Scala Projects

Message-driven Applications A Distributed Streaming Platform High Velocity Web Framework Lightning-fast Unified Analytics Engine

slide-8
SLIDE 8

(C) 2018, SoftLang Team, University of Koblenz-Landau

Trending Scala Projects

Message-driven Applications A Distributed Streaming Platform High Velocity Web Framework Lightning-fast Unified Analytics Engine Extensible RPC System

slide-9
SLIDE 9

(C) 2018, SoftLang Team, University of Koblenz-Landau

slide-10
SLIDE 10

(C) 2018, SoftLang Team, University of Koblenz-Landau

Context

IDEs, SBT and JVM.

slide-11
SLIDE 11

(C) 2018, SoftLang Team, University of Koblenz-Landau

Context: IDEs

Intellij or Eclipse provide an interactive development environment for Scala.

slide-12
SLIDE 12

(C) 2018, SoftLang Team, University of Koblenz-Landau

Context: SBT

Scala comes with the Scala Build Tool (SBT) written in Scala using a DSL that also supports dependency management.

slide-13
SLIDE 13

(C) 2018, SoftLang Team, University of Koblenz-Landau

Context: JVM

[jvm]

Scala compiles to Java bytecode that runs on the JVM. Calling Scala from Java looks funny (see this decompiled scala class). Getter Setter Constructor

slide-14
SLIDE 14

(C) 2018, SoftLang Team, University of Koblenz-Landau

Basics

[scdoc] https://docs.scala-lang.org/

slide-15
SLIDE 15

(C) 2018, SoftLang Team, University of Koblenz-Landau

Basics: Expressions and Values

[scdoc]

Expressions are computable statement. The keyword ‘val’ defines values that name results of expressions. They do not need to be recomputed and they can not be reassigned.

slide-16
SLIDE 16

(C) 2018, SoftLang Team, University of Koblenz-Landau

Basics: Variables

[scdoc]

The keyword ‘var’ defines Variables that can be declared like values. Variables can be reassigned to a different expression. 2 3

slide-17
SLIDE 17

(C) 2018, SoftLang Team, University of Koblenz-Landau

Basics: Blocks

[scdoc]

Expressions can be surrounded by a Block with ‘{‘ and ‘}’. The result of the last expression in this block is the result of the overall block. 3

slide-18
SLIDE 18

(C) 2018, SoftLang Team, University of Koblenz-Landau

Basics: Functions

[scdoc]

Functions are expressions that take parameters. To the left of keyword ‘=>’, a list declares available parameters and to the right an expression involving those parameters. 2

slide-19
SLIDE 19

(C) 2018, SoftLang Team, University of Koblenz-Landau

Basics: Methods

[scdoc]

Methods look and behave very similar to functions. The keyword ‘def’ is followed by a name, multiple parameter lists, an optional return type, and a body.

slide-20
SLIDE 20

(C) 2018, SoftLang Team, University of Koblenz-Landau

Basics: Classes

[scdoc]

The keyword ‘class’ defines classes taking a list of constructor parameters. Methods with the singleton ‘Unit’ return type carry no information and are called because of its side-effects.

slide-21
SLIDE 21

(C) 2018, SoftLang Team, University of Koblenz-Landau

Basics: Case Classes

[scdoc]

The prefix ‘case’ distinguishes case classes from classes. Case classes are immutable and can be compared by value. True

slide-22
SLIDE 22

(C) 2018, SoftLang Team, University of Koblenz-Landau

Basics: Objects

[scdoc]

Objects are singleton instances of their own definition.

slide-23
SLIDE 23

(C) 2018, SoftLang Team, University of Koblenz-Landau

Basics: Name Arguments

[scdoc]

Comparable to Python you can pass the method arguments by name.

slide-24
SLIDE 24

(C) 2018, SoftLang Team, University of Koblenz-Landau

Basics: For Comprehension

[scdoc]

An enumerator contains either a generator which introduces new variables, or it is a

  • filter. The yield expression is executed for every generated binding of the variables.

Travis Dennis

slide-25
SLIDE 25

(C) 2018, SoftLang Team, University of Koblenz-Landau

Basics: Main Method

[scdoc]

The Java Virtual Machine requires a main method to be named ‘main’ as an entry point of the program. It takes an array of strings as arguments.

slide-26
SLIDE 26

(C) 2018, SoftLang Team, University of Koblenz-Landau

Best Practices

[twbp] http://twitter.github.io/effectivescala/

slide-27
SLIDE 27

(C) 2018, SoftLang Team, University of Koblenz-Landau

‘While highly effective, Scala is also a large language, and our experiences have taught us to practice great care in its application. What are its pitfalls? Which features do we embrace, which do we eschew? When do we employ “purely functional style”, and when do we avoid it?’ [twbp]

slide-28
SLIDE 28

(C) 2018, SoftLang Team, University of Koblenz-Landau

Best Practices: Optional

[twbp]

Using the ‘Optional’ container provides a safe alternative to the use of ‘null’.

slide-29
SLIDE 29

(C) 2018, SoftLang Team, University of Koblenz-Landau

Best Practices: Destructuring

[twbp]

Destructure tuples or case classes during the binding instead of accessing its properties using the methods ‘_1’ or ‘_2’.

slide-30
SLIDE 30

(C) 2018, SoftLang Team, University of Koblenz-Landau

Best Practices: Destructuring & Matching

[twbp]

Combine pattern matching with such destructuring.

slide-31
SLIDE 31

(C) 2018, SoftLang Team, University of Koblenz-Landau

Best Practices: Matching

[twbp]

Use pattern matching whenever applicable but collapse it.

slide-32
SLIDE 32

(C) 2018, SoftLang Team, University of Koblenz-Landau

Best Practices: Mutable Collections

[twbp]

Prefer using immutable collections. If referencing to mutable Collections, use the ‘mutable’ namespace explicitly.

slide-33
SLIDE 33

(C) 2018, SoftLang Team, University of Koblenz-Landau

Best Practices: Collection Construction

[twbp]

Use the default constructors for collection type.This style separates the semantics

  • f the collection from its implementation and allows compiler optimization.
slide-34
SLIDE 34

(C) 2018, SoftLang Team, University of Koblenz-Landau

Best Practices: Java Collections

[twbp]

Use the converters to interoperate with the Java collection types.

slide-35
SLIDE 35

(C) 2018, SoftLang Team, University of Koblenz-Landau

Best Practices: Implicit Conversion

[twbp]

Implicits should be used sparingly, for instance in case of a library extension (“pimp my library” pattern).

slide-36
SLIDE 36

(C) 2018, SoftLang Team, University of Koblenz-Landau

Best Practices: Return

[twbp]

Use ‘return’ to enhance readability but not as you would in an imperative programming language.

slide-37
SLIDE 37

(C) 2018, SoftLang Team, University of Koblenz-Landau

Best Practices: Style

[twbp]

Keep track of all the intermediate results that are only implied.

slide-38
SLIDE 38

(C) 2018, SoftLang Team, University of Koblenz-Landau

Best Practices: FlatMap

[twbp]

High order functions like ‘map’ or ‘flatMap’ are also available in nontraditional collections such as Future and Option. Using ‘for’ translates into the former.

slide-39
SLIDE 39

(C) 2018, SoftLang Team, University of Koblenz-Landau

Best Practices: ADTs

[twbp]

Using case classes to encode ADTs together with pattern matching. This results in code that is “obviously correct”.

slide-40
SLIDE 40

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark

Distributing Scala’s high level functions.

slide-41
SLIDE 41

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: A simple Task.

[spark]

Counting the words of some Lorem Ipsum.

slide-42
SLIDE 42

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: Distributing and Fetching Data

[spark]

A spark session is created (this time a local one with 16 cores). The data is processed using the provided API in the RDD class (resilient distributed dataset). RDD RDD Fetch back the data Distribute the data

slide-43
SLIDE 43

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: Infrastructure

Spark serializes the functions and sends them to the workers. Further it provides 4 mechanisms to exchange data, i.e., parallelize, broadcast, collect and accumulate.

Data [spark2] Functions

slide-44
SLIDE 44

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: Partitions

The Lorem Ipsum is split into several partitions that can be processed in isolation; hence, on different nodes.

slide-45
SLIDE 45

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: Partitions

Reading the lines of a local file into a resilient distributed dataset (RDD) with three partitions.

slide-46
SLIDE 46

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: Partitions

Splitting the lines with ‘flatMap’ into

  • words. This can be done within the

same partition as there is no dependency between the different sentences.

slide-47
SLIDE 47

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: Partitions

Tuples of word and number are formed that can later be summed up. Now there are plenty of tuples with the same key and the value ‘1’.

slide-48
SLIDE 48

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: Partitions

‘reduceByKey’ aggregates the values of all tuples with the same key. Unfortunately this requires records to switch between partitions causing network traffic.

SHUFFLE

slide-49
SLIDE 49

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: Partitions

Sorting of the records also requires movement data between partitions. This time the shuffling is not based on the hash of the key.

SHUFFLE

slide-50
SLIDE 50

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: Benefits

The data and processing can be distributed over several nodes of a cluster. The partitions can be (i) recomputed when needed, (ii) persistent on the hard drive if the computation is expensive or (iii) kept in memory if it is accessed frequently.

slide-51
SLIDE 51

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: A complex Task (LOC)

Question: ‘Who contributed the most LOC in the evolution of a Git repository?

slide-52
SLIDE 52

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: A complex Task (LOC)

Some values needed during the analysis. First and last commit time. Values needed and send with the instructions. A way to keep track of how many objects have been processed. A target project and some access wrapper for the repository. All commits

slide-53
SLIDE 53

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: A complex Task (LOC)

Send the commits to the worker nodes forming 16 partitions (by calling ‘parallelize’). Fatten this records in that they represent all ‘objects’ in a tree of a particular commit (like splitting the Lorem Ipsum sentence into words). 22.443.734 records

slide-54
SLIDE 54

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: A complex Task (LOC)

We don’t want to analysis the same object twice so we group by object and path. We increase the number of partitions to 256 during the required shuffle step. 39.754 records

slide-55
SLIDE 55

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: A complex Task (LOC)

Now GIT ‘blame’ can be applied on each object. We cannot serialize the repository wrapper ‘git’ in the instructuction so we create a new one on each worker. Increase

  • bject

conter 154.565 records

slide-56
SLIDE 56

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: A complex Task (LOC)

We sum up the values by author ignoring the path.

slide-57
SLIDE 57

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: A complex Task (LOC)

All stages, i.e., a set of parallel tasks with one task per partition can be supervised in the Web UI. Stage Tasks

slide-58
SLIDE 58

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: A Comparison

There is also a Spark API available for Python and Java, however, Spark is implemented in Scala.

Python Java

slide-59
SLIDE 59

(C) 2018, SoftLang Team, University of Koblenz-Landau

Spark: A Comparison

The Java 8 native solution is to used parallel Streams which is also based on chunking. However, the partitions are not distributed over a cluster.

Java

slide-60
SLIDE 60

(C) 2018, SoftLang Team, University of Koblenz-Landau

References

  • [wik]

https://en.wikipedia.org/wiki/Scala_(programming_language)

  • [scdoc]

https://docs.scala-lang.org/

  • [twbp]

http://twitter.github.io/effectivescala/

  • [twsc]

http://twitter.github.io/scala_school/

  • [ep1]

Zenger, Matthias, and Martin Odersky. Independently extensible solutions to the expression problem.

  • No. LAMP-REPORT-2004-004. 2004.
  • [ep2]

http://lampwww.epfl.ch/~odersky/papers/ExpressionProblem

  • [ep3]

https://gist.github.com/calincru/cea751f050883581730093e93eaf2723

  • [tp]

https://www.tutorialspoint.com/scala

  • [spark]

https://spark.apache.org/examples.html

  • [jvm]

https://alvinalexander.com/scala/how-to-disassemble-decompile-scala-source-code-javap-scalac-jad

  • [spark2]

https://spark.apache.org/docs/latest/

slide-61
SLIDE 61

(C) 2018, SoftLang Team, University of Koblenz-Landau