Algorithmns and Datastructures O-Notation, L Hopital - - PowerPoint PPT Presentation

Algorithmns and Datastructures

O-Notation, L ’Hopital

Albert-Ludwigs-Universität Freiburg

• Prof. Dr. Rolf Backofen

Bioinformatics Group / Department of Computer Science Algorithmns and Datastructures, November 2018

Structure

O-Notation Motivation / Definition Examples Ω-Notation Θ-Notation Runtime Summary Limit / Convergence L ’Hôpital / l’Hospital Practical use

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

2 / 56

O-Notation

Motivation

We are interested in: Example: sorting

Runtime of Minsort “is growing as” n2 Runtime of Heapsort “is growing as” n logn

Growth of a function in runtime T(n)

the role of constants (e.g. 1ns) is minor it is enough if relation holds for some n ≥ ...

Describe the growth of the function more formally

by the means of Landau-Symbols [Wik]):

O(n) (Big O of n), Ω(n) (Omega of n), Θ(n) (Theta of n)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

4 / 56

O-Notation

Definition

Big O-Notation: Consider the function: f : N → R, n → f(n)

N: Natural numbers → input size R: Real numbers → runtime

Example:

f(n) = 3n f(n) = 2n logn f(n) = 1

10n2

f(n) = n2 +3n logn−4n

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

5 / 56

O-Notation

Definition

Big O-Notation: Given two functions f and g: f,g: N → R Intuitive: f is Big-O of g (f is O(g))

... if f relative to g does not grow faster than g the growth rate matters, not the absolute values

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

6 / 56

O-Notation

Definition

Big O-Notation: Informal: f = O(g)

“=” corresponds to "is" not "is equal to" ... if for some value n0 for all n ≥ n0 f(n) ≤ C ·g(n) for a constant C (f = O(g): From a value n0 for all n ≥ n0 → f(n) ≤ C ·g(n))

Formal: f ∈ O(g)

Formal: f ∈ O(g)

O(g) = { f : N → R | ∃n0 ∈ N, ∃C > 0, ∀n > n0 : f(n) ≤ C·g(n)}

“set of all functions” “for which” “it exists” “for all” “such that”

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

7 / 56

O-Notation

Examples

Illustration of the Big O-Notation:

Figure: Runtime of two algorithms f1,f2

g(n) f2 ∈ O(g) f1 ∈ O(g)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

9 / 56

O-Notation

Examples

Example: f(n) = 5n+7, g(n) = n ⇒ 5n+7 ∈ O(g) ⇒ f ∈ O(g) Intuitive: f(n) = 5n+7 → linear growth

Attention

f(n) ≤ g(n) is not guaranteed, better is f(n) ≤ C ·g(n) ∀n > n0.

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

10 / 56

O-Notation

Proof

We have to proof: ∃n0, ∃C, ∀n ≥ n0 : 5n+7 ≤ C ·n .

5n+7 ≤ 5n+n (for n ≥ 7) = 6n ⇒ n0 = 7, C = 6

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

11 / 56

O-Notation

Proof

Alternate proof:

5n+7 ≤ 5n+7n (for n ≥ 1) = 12n ⇒ n0 = 1, C = 12

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

12 / 56

O-Notation

Examples

Big O-Notation: We are only interested in the term with the highest-order, the fastest growing summand, the others will be ignored f(n) is limited from above by C ·g(n) Examples: 2n2 +7n−20 ∈O(n2) 2n2 +7n logn−20 ∈ 7n logn−20 ∈ 5 ∈ 2n2 +7n logn+n3 ∈

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

13 / 56

O-Notation

Examples

Harder Example: Polynomes are simple More problematic: combination of complex functions 2 √ x +3lnx ∈ O(??)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

14 / 56

Ω-Notation

Definition

Omega-Notation: Intuitive:

f ∈ Ω(g), f is growing at least as fast as g So the same as Big-O but with at-least and not at-most

Formal: f ∈ Ω(g)

Ω(g) = {f : N → R | ∃n0 ∈ N, ∃C > 0, ∀n > n0 : f(n) ≥ C ·g(n)}

“in O(n) we had ≤”

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

16 / 56

Ω-Notation

Proof

Example:

Proof of f(n) = 5n+7 ∈ Ω(n):

5n+7

f(n)

≥ 1·n

• g(n)

(for n ≥ 1) ⇒ n0 = 1, C = 1

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

17 / 56

Ω-Notation

Examples

Illustration of the Omega-Notation:

Figure: Runtime of two algorithms f1,f2

f2 ∈ Ω(g) f1 ∈ Ω(g) g(n)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

18 / 56

Ω-Notation

Examples

Big Omega-Notation: We are only interested in the term with the highest-order, the fastest growing summand, the others will be ignored f(n) is limited from underneath by C ·g(n) Examples: 2n2 +7n−20 ∈Ω(n2) 2n2 +7n logn−20 ∈ 7n logn−20 ∈ 5 ∈ 2n2 +7n logn+n3 ∈

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

19 / 56

Θ-Notation

Definition

Theta-Notation: Intuitive: f is Theta of g ...

... if f is growing as much as g f ∈ Θ(g), f is growing at the same speed as g

Formal: f ∈ Θ(g)

Θ(g) = O(g)∩Ω(g)

• Intersection

Example: f(n) = 5n+7, f(n) ∈ O(n), f(n) ∈ Ω(n) ⇒ f(n) ∈ Θ(n) Proof for O(g) and Ω(g) look at slides 11 and 17

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

21 / 56

Θ-Notation

Graphs

n n0

number of steps (runtime (s))

runtime of algorithm upper bound lower bound

f and g have the same “growth”

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

22 / 56

Runtime

Landau-Symbol Summary

Big O-Notation O(n): f is growing at most as fast as g C ·g(n) is the upper bound Big Omega-Notation Ω(n): f is growing at least as fast as g C ·g(n) is the lower bound Big Theta-Notation Θ(n): f is growing at the same speed as g

C1 ·g(n) is the lower bound C2 ·g(n) is the upper bound

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

24 / 56

Runtime

Common Runtimes

Table: Common runtime types

Runtime Growth f ∈ Θ(1) constant time f ∈ Θ(logn) = Θ(logk n) logarithmic time f ∈ Θ(n) linear time f ∈ Θ(n logn) n-log-n time (nearly linear) f ∈ Θ(n2) squared time f ∈ Θ(n3) cubic time f ∈ Θ(nk) polynomial time f ∈ Θ(kn), f ∈ Θ(2n) exponential time

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

25 / 56

So far discussed:

Membership in O(...) proofed by hand: Explicit calculation of n0 and C However: Both hint at limits in calculus

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

27 / 56

O-Notation

Limit / Convergence

Definition of “Limit” The limit L exists for an infinite sequence f1,f2,f3,... if for all ε > 0 one n0 ∈ N exists, such that for all n ≥ n0 the following holds true: |fn −L| ≤ ε A function f : N → R can be written as a sequence ⇒ lim

n→∞fn = L

The limit is converging:

∀ε > 0 ∃n0 ∈ N ∀n ≥ n0 : |fn −L| ≤ ε

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

28 / 56

O-Notation

Limit / Convergence

Example for the proof of a limit Function f(n) = 2+ 1

n with limes limn→∞ f(n) = 2

“Engineering” solution: use n = ∞ 1 ∞ = 0 ⇒ lim

n→∞f(n) = lim n→∞2+ 1

n = 2

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

29 / 56

O-Notation

Limit / Convergence

Now a more formal proof for lim

n→∞2+ 1

n = 2 We need to show: for all given ε there is an n0 such that for all n ≥ n0

• 2+ 1

n −2

• =
• 1

n

• ≤ ε

E.g.: for ε = 0.01 we get 1

n ≤ ε for n ≥ 100

In general n0 = 1 ε

• Then we get:
• 1

n

• = 1

n ≤ 1 n0 = 1

• 1

ε

≤ 1

1 ε

= ε

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

30 / 56

O-Notation

Limit / Convergence

Let f,g: N → R with an existing limit lim

n→∞

f(n) g(n) = L Hence the following holds: f ∈ O(g) ⇔ lim

n→∞

f(n) g(n) < ∞ (1) f ∈ Ω(g) ⇔ lim

n→∞

f(n) g(n) > 0 (2) f ∈ Θ(g) ⇔ 0 < lim

n→∞

f(n) g(n) < ∞ (3)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

31 / 56

O-Notation

Limit / Convergence

f ∈ O(g) ⇔ lim

n→∞

f(n) g(n) < ∞

Forward proof (⇒):

f ∈ O(g)

• def. of O(n)

⇒ ∃n0, C ∀n ≥ n0 : f(n) ≤ C ·g(n) ⇒∃n0, C ∀n ≥ n0 : f(n) g(n) ≤ C ⇒ lim

n→∞

f(n) g(n) ≤ C

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

32 / 56

O-Notation

Limit / Convergence

Backward proof (⇐):

lim

n→∞

f(n) g(n) < ∞ ⇒ lim

n→∞

f(n) g(n) = C For some C ∈ R (Limit)

• def. limes

⇒ ∃n0, ∀n ≥ n0 : f(n) g(n) ≤ C +ε (e.g. ε = 1) ⇒ ∃n0, ∀n ≥ n0 : f(n) ≤ (C +1)

O−notation constant

·g(n) ⇒ f ∈ O(g)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

33 / 56

Limits with L’Hôpital

Intuitive: lim

n→∞2+ 1

n = 2+ 1 ∞ = 2 With L’Hôpital:

Let f, g : N → R If lim

n→∞f(n) = lim n→∞g(n) = ∞/0

⇒ lim

n→∞

f(n) g(n) = lim

n→∞

f ′(n) g′(n) Holy inspiration you need a doctoral degree for that

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

35 / 56

Limits with L’Hôpital

The limit can not be determined in the way of an Engineer: lim

n→∞

ln(n) n = limn→∞ ln(n) lim

n→∞ n plugging in

− → ∞ ∞ Determine the limit using L’Hôpital: lim

n→∞

f(n) g(n) = lim

n→∞

f ′(n) g′(n)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

36 / 56

Limits with L’Hôpital

Using L’Hôpital:

Numerator: f(n): n → ln(n) Denominator: g(n): n → n ⇒ f ′(n) = 1 n (derivation from Numerator) ⇒ g′(n) = 1 (derivation from Denominator) lim

n→∞

f ′(n) g′(n) = lim

n→∞

1 n = 0 ⇒ lim

n→∞

f(n) g(n) = lim

n→∞

ln(n) n = 0

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

37 / 56

Limits with L’Hôpital

What can we take for granted without proofing? Only things that are trivial It is always better to proof it Examples: lim

n→∞

1 n = 0 is trivial lim

n→∞

1 n2 = 0 is trivial lim

n→∞

log(n) n = 0 use L ’Hopital

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

38 / 56

O-Notation

Practical use

Practical use: It is much easier to determine the runtime of an algorithm by using the O-Notation

1 Computing rules 2 Practical use

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

40 / 56

O-Notation

Characteristics

Transitivity: f ∈ Θ(g) ∧ g ∈ Θ(h) → f ∈ Θ(h) f ∈ O(g) ∧ g ∈ O(h) → f ∈ O(h) f ∈ Ω(g) ∧ g ∈ Ω(h) → f ∈ Ω(h) Symmetry: f ∈ Θ(g) ↔ g ∈ Θ(f) f ∈ O(g) ↔ g ∈ Ω(f) Reflexivity: f ∈ Θ(f) f ∈ Ω(f) f ∈ O(f)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

41 / 56

O-Notation

Calculation Rules

Trivial: f ∈ O(f) C ·O(f) = O(f) O(f +C) = O(f) Addition: O(f)+O(g) = O(max{f, g}) Multiplication: O(f)·O(g) = O(f ·g)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

42 / 56

O-Notation

Runtime Complexity

The input size for all examples is n Basic operations

i1 = 0

O(1) Sequences of basic operations

i1 = 0

O(1)

i2 = 0

O(1)

···

···

i327 = 0

O(1)        327·O(1) = O(1)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

43 / 56

O-Notation

Runtime Complexity

Loops

for i in range(0, n):

O(n)

a[i] = 0

O(1)

• O(1)·O(n) = O(n)

for i in range(0, n):

O(n)

a1[i] = 0

O(1)

···

···

a137[i] = 0

O(1)    137·O(1) = O(1)        O(1)·O(n) = O(n)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

44 / 56

O-Notation

Runtime Complexity

Loops

for i in range(0, n):

O(n)

for j in range(0, n−1):

O(n−1) O(n)·O(n) = O(n2)

a1[i][j] = 0

O(1)

···

···

a137[i][j] = 0

O(1)    137·O(1) = O(1)            O(1)·O(n2) = O(n2)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

45 / 56

O-Notation

Runtime Complexity

Conditions

if x < 100:

O(1)

y = x

O(1)

• O(1)

else: for i in range(0, n):

O(n)

if a[i] > y:

O(1)

y = a[i]

O(1)    O(n)·O(1) = O(n)                O(max{1,n}) = O(n)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

46 / 56

O-Notation

Arithmetic mean

Input: List x with n numbers Output: a[i] is the arithmetic mean of x[0] to x[i]

def arithMean ( x ) : a = [ 0 ] ∗ len ( x ) f o r i in range (0 , len ( x ) ) : s = 0 f o r j in range (0 , i +1) : s = s + x [ j ] a [ i ] = s / ( i +1) return a

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

47 / 56

O-Notation Runtime complexity

for i in range(0, len(x)):

O(n)

s = 0

O(1)

• O(n)

for j in range(0, i+1):

O(i +1)

s = s + x[j]

O(1)

• O(i)

a[i] = s / ( i+1)

O(1)            O(n)·O(i) = O(n2) How often will the instructions in the loop be executed, when the problem has size n? 1+2+...+n = n·(n+1) 2 ∈ O(n2)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

48 / 56

O-Notation

Discussion

Way of speaking: With the O-Notation we look at the behavior of a function when n → ∞ We only analyze the runtime when n ≥ n0 We talk about asymptotic analysis, when we discuss cost, runtime, etc. as O(...), Ω(...) or Θ(...)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

49 / 56

O-Notation

Discussion

Attention:

If you are using asymptotic analysis, you can not make any predictions about the runtime of smaller input sizes (n < n0) For small input sizes (mostly n < 10), the runtime is predictably small n0 does not necessarily have to be small

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

50 / 56

O-Notation

Discussion

Examples: Let A and B be algorithms

A has the runtime f(n) = 80n B has the runtime g(n) = 2n log2 n

So f = O(g) but not Θ(g)

⇒ A is asymptotic faster than B ⇒ There is an n0 for that n ≥ n0 : f(n) ≤ g(n)

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

51 / 56

O-Notation

Discussion

When is A faster then B?

We search the minimal n0: f(n0) = g(n0) 80n0 = 2n0 log2 n0 40 = log2 n0 n0 = 240 = (210)4 = (1024)4 ≈ (103)4 = 1012 ≈ 1trillion A ist faster than B if n0 has more than 1 trillion elements

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

52 / 56

Runtime Examples

Continued

Logarithm of different bases differ only by a constant loga n = logb n logb a = 1 logb a ·logb n Hence: loga n ∈ Θ(logb n) For exponent this does not hold 3n ∈ Θ(2n) Proof: Use equation (1) from Slide 31 3n ∈ O(2n) ⇔ lim

n→∞

3n 2n < ∞ However: lim

n→∞

3n 2n = lim

n→∞

3 2 n = ∞

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

53 / 56

Additional Figure

Figure for slide 28

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

54 / 56

Further Literature

General [MS08] Kurt Mehlhorn and Peter Sanders. Algorithms and data structures, 2008. https://people.mpi-inf.mpg.de/~mehlhorn/ ftp/Mehlhorn-Sanders-Toolbox.pdf.

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

55 / 56

Further Literature

Big O notation [Wik] Big O notation https://en.wikipedia.org/wiki/Big_O_notation

November 2018

• Prof. Dr. Rolf Backofen – beamer-ufcd

56 / 56