Big- Big -O O Analyzing Algorithms Asymptotically Analyzing - - PDF document

SLIDE 1

1

Big Big-

• O

O

Analyzing Algorithms Asymptotically Analyzing Algorithms Asymptotically CS 226 CS 226 5 February 2002 5 February 2002 Noah Smith ( Noah Smith (nasmith@cs

nasmith@cs)

)

Comparing Algorithms Comparing Algorithms

Should we use Program 1 or Program 2? Should we use Program 1 or Program 2? Is Program 1 “fast”? “Fast enough”? Is Program 1 “fast”? “Fast enough”?

P1 P2

You and Igor: You and Igor: the empirical approach the empirical approach

Implement each candidate Run it Time it

That could be lots

• f work – also

error-prone. Which inputs? What machine/OS?

Toward an analytic approach … Toward an analytic approach …

Today is just math: Today is just math: How to solve “which algorithm” How to solve “which algorithm” problems problems without

without machines, test

machines, test data, or Igor! data, or Igor!

The Big Picture The Big Picture

Input (n = 3) Output Input (n = 4) Output Input (n = 8) Output

How long does it take for the algorithm to finish? How long does it take for the algorithm to finish?

Algorithm

Primitives Primitives

• Primitive operations

Primitive operations

• x = 4

x = 4 assignment assignment

• ... x + 5 ...

... x + 5 ... arithmetic arithmetic

• if (x < y) ...

if (x < y) ... comparison comparison

• x[4]

x[4] index an array index an array

• *x

*x dereference (C) dereference (C)

• x.foo

x.foo( ) ( ) calling a method calling a method

• Others

Others

• new/

new/malloc malloc memory usage memory usage

SLIDE 2

2 How many How many foos foos? ?

for (j = 1; j <= N; ++j) { for (j = 1; j <= N; ++j) { foo foo( ); ( ); } }

Σ Σ

N N j = 1 j = 1

1 = N 1 = N

How many How many foos foos? ?

for (j = 1; j <= N; ++j) { for (j = 1; j <= N; ++j) { for (k = 1; k <= M; ++k) { for (k = 1; k <= M; ++k) { foo foo( ); ( ); } } } }

Σ Σ

N N j = 1 j = 1

1 = NM 1 = NM

Σ Σ

M M k = 1 k = 1

How many How many foos foos? ?

for (j = 1; j <= N; ++j) { for (j = 1; j <= N; ++j) { for (k = 1; k <= j; ++k) { for (k = 1; k <= j; ++k) { foo foo( ); ( ); } } } }

Σ Σ

N N j = 1 j = 1

1 = 1 =

Σ Σ

j j k = 1 k = 1

Σ Σ

N N j = 1 j = 1

j = j = N (N + 1) N (N + 1) 2 2

How many How many foos foos? ?

for (j = 0; j < N; ++j) { for (j = 0; j < N; ++j) { for (k = 0; k < j; ++k) { for (k = 0; k < j; ++k) { foo foo( ); ( ); } } } } for (j = 0; j < N; ++j) { for (j = 0; j < N; ++j) { for (k = 0; k < M; ++k) { for (k = 0; k < M; ++k) { foo foo( ); ( ); } } } } N(N + 1)/2 N(N + 1)/2 NM NM

How many How many foos foos? ?

void void foo(int foo(int N) { N) { if(N if(N <= 2) <= 2) return; return; foo(N foo(N / 2); / 2); } }

T(0) = T(1) = T(2) = 1 T(0) = T(1) = T(2) = 1 T(n T(n) ) = 1 + T(n/2) if n > 2 = 1 + T(n/2) if n > 2 T(n T(n) ) = 1 + (1 + T(n/4)) = 1 + (1 + T(n/4)) = 2 + T(n/4) = 2 + T(n/4) = 2 + (1 + T(n/8)) = 2 + (1 + T(n/8)) = 3 + T(n/8) = 3 + T(n/8) = 3 + (1 + T(n/16)) = 3 + (1 + T(n/16)) = 4 + T(n/16) = 4 + T(n/16) … … ≈ log ≈ log2

2 n

n

The trick The trick

an ank

k +

+ bn

bnk

k -

• 1

1 + … +

+ … + yn

yn +

+ z

z an ank

k

n nk

k

SLIDE 3

3 Big O Big O

Definition: Definition: Let

Let f

f and

and g

g be functions mapping

be functions mapping N

N

to to R

• R. We say that

. We say that f

f(

(n

n) is

) is O

O(

(g

g(

(n

n)) if there exist

)) if there exist c c є є R

R, c > 0

, c > 0 and and

n n0

0 є

є N

N,

, n

n0

0 ≥ 1

≥ 1 such that such that

f f(

(n

n) ≤

) ≤ c cg

g(

(n

n) for all

) for all n

n є

є N

N,

, n

n ≥

≥ n

n0

Example 1 Example 1

f(n) g(n) n0

Example 2 Example 2

h(n) g(n) n0

Example 3 Example 3

h(n) k (n)

Example 3 Example 3

k (n) h(n)

Example 3 Example 3

h(n) k (n)

SLIDE 4

4 Example 4 Example 4

3n2

n2

Some complexity classes … Some complexity classes …

O O(log

(log n

n)

) Logarithmic Logarithmic

O O(

(n

n)

) Linear Linear

O O(

(a

an

n)

) Exponential Exponential

O O(

(n

np

p)

) Polynomial Polynomial

O O(

(n

n3

3)

) Cubic Cubic

O O(

(n

n2

2)

) Quadratic Quadratic

O O(1)

(1) Constant Constant

Don’t be confused … Don’t be confused …

• We typically say,

We typically say,

f f(

(n

n) is

) is O

O(

(g

g(

(n

n)) or

)) or f

f(

(n

n) =

) = O

O(

(g

g(

(n

n))

))

• But

But O

O(

(g

g(

(n

n)) is really a

)) is really a set set of functions.

• f functions.
• It might be more clear to say,

It might be more clear to say,

f f(

(n

n)

) є є O

O(

(g

g(

(n

n))

))

• But I don’t make the rules.

But I don’t make the rules.

• Crystal clear: “

Crystal clear: “f

f(

(n

n) is

) is order

• rder (

(g

g(

(n

n)”

)”

Intuitively … Intuitively …

• To say

To say f

f(

(n

n) is

) is O

O(

(g

g(

(n

n)) is to say that

)) is to say that

f f(

(n

n) is “less than or equal to”

) is “less than or equal to” g

g(

(n

n)

)

• We also have (G&T pp. 118

We also have (G&T pp. 118-

• 120):

120): Little omega Little omega Little o Little o Big Omega Big Omega Big Theta Big Theta “strictly greater than” “strictly greater than” ω ω( (g

g(

(n

n))

)) “strictly less than” “strictly less than”

• (

(g

g(

(n

n))

)) “greater than or equal to” “greater than or equal to” Ω Ω( (g

g(

(n

n))

)) “equal to” “equal to” Θ Θ( (g

g(

(n

n))

))

Big Big-

• Omega and Big

Omega and Big-

• Theta

Theta

Ω Ω is just like is just like O

O except that

except that f

f(

(n

n) ≥

) ≥ c cg

g(

(n

n);

);

f f(

(n

n) is

) is O

O(

(g

g(

(n

n))

)) ⇔ ⇔ g

g(

(n

n) is

) is Ω Ω( (f

f(

(n

n))

)) Θ Θ is both is both O

O and

and Ω Ω (and the constants need not (and the constants need not match); match);

f f(

(n

n) is

) is O

O(

(g

g(

(n

n))

)) ∧ ∧ f

f(

(n

n) is

) is Ω Ω( (g

g(

(n

n))

)) ⇔ ⇔ f

f(

(n

n) is

) is Θ Θ( (g(n

g(n))

))

little o little o

Definition: Definition: Let

Let f

f and

and g

g be functions mapping

be functions mapping N

N

to to R

• R. We say that

. We say that f

f(

(n

n) is

) is o

• (

(g

g(

(n

n)) if

)) if for any for any c c є є R

R, c > 0

, c > 0 there exists there exists

n n0

0 є

є N

N,

, n

n0

0 > 0

> 0 such that such that

f f(

(n

n) ≤

) ≤ c cg

g(

(n

n) for all

) for all n

n є

є N

N,

, n

n ≥

≥ n

n0

(little omega, (little omega, ω ω, is the same but with ≥) , is the same but with ≥)

SLIDE 5

5 Multiple variables Multiple variables

for(j for(j = 1; j <= N; ++j) = 1; j <= N; ++j) for(k for(k = 1; k <= N; ++k) = 1; k <= N; ++k) for(l for(l = 1; l <= M; ++l) = 1; l <= M; ++l) foo foo(); (); for(j for(j = 1; j <= N; ++j) = 1; j <= N; ++j) for(k for(k = 1; k <= M; ++k) = 1; k <= M; ++k) for(l for(l = 1; l <= M; ++l) = 1; l <= M; ++l) foo foo(); ();

O O(N

(N2

2M + NM

M + NM2

2)

)

Multiple primitives Multiple primitives

for(j for(j = 1; j <= N; ++j){ = 1; j <= N; ++j){ sum += sum += A[j A[j]; ]; for(k for(k = 1; k <= M; ++k) { = 1; k <= M; ++k) { sum2 += sum2 += B[j][k B[j][k]; ]; C[j][k C[j][k] = ] = B[j][k B[j][k] * ] * A[j A[j] + 1; ] + 1; for(l for(l = 1; l <= k; ++l) = 1; l <= k; ++l) B[j][k B[j][k] ] -

• =

= B[j][l B[j][l]; ]; } } } }

Tradeoffs: an example Tradeoffs: an example

0, 0 0, 1 0, 2 0, 3 … 1, 0 1, 1 1, 2 1, 3 … N, 0 N, 1 N, 2 … N, N 0 0 0 1 0 2 ∅ N N … (lots of options in between)

Another example Another example

• I have a set of integers between 0

I have a set of integers between 0 and 1,000,000. and 1,000,000.

• I need to store them, and I want

I need to store them, and I want

O O(1) lookup, insertion, and deletion.

(1) lookup, insertion, and deletion.

• Constant time and constant space,

Constant time and constant space, right? right?

1 2 3 … 1,000,000

Big Big-

• O and Deceit

O and Deceit

• Beware huge coefficients

Beware huge coefficients

• Beware key lower order terms

Beware key lower order terms

• Beware when

Beware when n

n is “small”

is “small”

Does it matter? Does it matter?

Let Let n

n = 1,000, and 1 ms / operation.

= 1,000, and 1 ms / operation.

96 96 32 years 32 years

n n4

4

3,943,234 3,943,234 10 seconds 10 seconds

n n log

log2

2 n

n

1.07 1.07 × × 10 10301

301 years

years 3.17 3.17 × × 10 1019

19 years

years 12 days 12 days 17 minutes 17 minutes 1 second 1 second

n n = 1000, 1 ms/op

= 1000, 1 ms/op 26 26 6 6 442 442 9,295 9,295 86,400,000 86,400,000 max max n

n in one day

in one day

n n

2 2n

n

n n10

10

n n3

3

n n2

2

SLIDE 6

6 Worst, best, and average Worst, best, and average

Gideon is a fast runner Gideon is a fast runner

• … up hills.

… up hills.

• … down hills.

… down hills.

• … on flat ground.

… on flat ground. Gideon is the Gideon is the fastest fastest swimmer swimmer

• … on the JHU team.

… on the JHU team.

• … in molasses.

… in molasses.

• … in our research lab.

… in our research lab.

• … in 5

… in 5-

• yard race.

yard race.

• … on Tuesdays.

… on Tuesdays.

• … in an average race.

… in an average race.

What’s average? What’s average?

• Strictly speaking, average (mean) is relative to

Strictly speaking, average (mean) is relative to some probability distribution. some probability distribution. mean( mean(X

X ) =

) = Pr( Pr(x

x)

) × × x

x

• Unless you have some notion of a probability

Unless you have some notion of a probability distribution over test cases, it’s hard to talk distribution over test cases, it’s hard to talk about average requirements. about average requirements.

Σ Σ

x x

Now you know … Now you know …

• How to analyze the run

How to analyze the run-

• time (or space

time (or space requirements) of a piece of pseudo requirements) of a piece of pseudo-

• code.

code.

• Some new uses for Greek letters.

Some new uses for Greek letters.

• Why the order of an algorithm matters.

Why the order of an algorithm matters.

• How to avoid some pitfalls.

How to avoid some pitfalls.