Big- Big -O O Analyzing Algorithms Asymptotically Analyzing - PDF document

Comparing Algorithms Comparing Algorithms Big- Big -O O Analyzing Algorithms Asymptotically Analyzing Algorithms Asymptotically P1 P2 CS 226 CS 226 5 February 2002 5 February 2002 Noah Smith ( nasmith@cs Noah Smith ( nasmith@cs ) ) Should we use Program 1 or Program 2? Should we use Program 1 or Program 2? Is Program 1 “fast”? “Fast enough”? Is Program 1 “fast”? “Fast enough”? You and Igor: You and Igor: Toward an analytic approach … Toward an analytic approach … the empirical approach the empirical approach Implement each Run it Time it Today is just math: Today is just math: candidate How to solve “which algorithm” How to solve “which algorithm” problems without problems without machines, test machines, test data, or Igor! data, or Igor! That could be lots Which What of work – also inputs? machine/OS? error-prone. The Big Picture Primitives The Big Picture Primitives � Primitive operations Primitive operations � � x = 4 x = 4 assignment assignment Input ( n = 3) � � ... x + 5 ... ... x + 5 ... arithmetic arithmetic � Input ( n = 4) � if (x < y) ... if (x < y) ... comparison comparison � � x[4] x[4] index an array index an array Input ( n = 8) Algorithm � � *x *x dereference (C) dereference (C) � � x.foo x.foo( ) ( ) calling a method calling a method Output Output Output � � Others Others � � new/ new/malloc malloc memory usage memory usage � How long does it take for the algorithm to finish? How long does it take for the algorithm to finish? 1

How many foos How many foos? ? How many foos How many foos? ? for (j = 1; j <= N; ++j) { for (j = 1; j <= N; ++j) { for (j = 1; j <= N; ++j) { for (j = 1; j <= N; ++j) { foo( ); foo ( ); for (k = 1; k <= M; ++k) { for (k = 1; k <= M; ++k) { } } foo foo( ); ( ); } } } } N N Σ Σ 1 = N 1 = N N N M M j = 1 j = 1 Σ Σ Σ Σ 1 = NM 1 = NM j = 1 k = 1 j = 1 k = 1 How many foos foos? ? How many foos foos? ? How many How many for (j = 0; j < N; ++j) { for (j = 1; j <= N; ++j) { for (j = 0; j < N; ++j) { for (j = 1; j <= N; ++j) { for (k = 0; k < j; ++k) { for (k = 0; k < j; ++k) { for (k = 1; k <= j; ++k) { for (k = 1; k <= j; ++k) { N(N + 1)/2 N(N + 1)/2 foo( ); ( ); foo foo( ); foo ( ); } } } } } } for (j = 0; j < N; ++j) { } } for (j = 0; j < N; ++j) { for (k = 0; k < M; ++k) { for (k = 0; k < M; ++k) { foo( ); foo ( ); NM NM N (N + 1) N (N + 1) N N j j N N Σ Σ Σ Σ Σ Σ 1 = 1 = j = j = } } 2 2 } } j = 1 j = 1 k = 1 k = 1 j = 1 j = 1 How many foos foos? ? The trick How many The trick k + 1 + … + void foo(int foo(int N) { N) { T(0) = T(1) = T(2) = 1 T(0) = T(1) = T(2) = 1 + bn k - -1 + … + yn yn + + z void an k bn k an z T(n) ) = 1 + T(n/2) if n > 2 T(n = 1 + T(n/2) if n > 2 if(N <= 2) <= 2) if(N return; return; T(n) T(n ) = 1 + (1 + T(n/4)) = 1 + (1 + T(n/4)) an k k an = 2 + T(n/4) = 2 + T(n/4) foo(N / 2); / 2); foo(N = 2 + (1 + T(n/8)) = 2 + (1 + T(n/8)) = 3 + T(n/8) = 3 + T(n/8) k } n k } n = 3 + (1 + T(n/16)) = 3 + (1 + T(n/16)) = 4 + T(n/16) = 4 + T(n/16) … … ≈ log 2 ≈ log 2 n n 2

Big O Big O Example 1 Example 1 Definition: Let Let f f and and g g be functions mapping be functions mapping N N Definition: to R . We say that f ( n ) is O ( g ( n )) if there exist to R . We say that f ( n ) is O ( g ( n )) if there exist c є є R , c > 0 c R , c > 0 g ( n ) and and 0 є є N N , , n 0 ≥ 1 ≥ 1 n 0 n n 0 f ( n ) such that such that f ( ( n n ) ≤ ) ≤ c c g g ( ( n n ) for all ) for all n n є є N N , , n n ≥ ≥ n n 0 f 0 n 0 Example 2 Example 3 Example 2 Example 3 h ( n ) h ( n ) g ( n ) k ( n ) n 0 Example 3 Example 3 Example 3 Example 3 k ( n ) k ( n ) h ( n ) h ( n ) 3

Example 4 Example 4 Some complexity classes … Some complexity classes … Constant (1) Constant O (1) O Logarithmic (log n ) Logarithmic O (log n ) O 3 n 2 Linear Linear O ( ( n n ) ) O Quadratic ( n 2 ) ) n 2 Quadratic O ( n 2 O Cubic Cubic O ( ( n n 3 3 ) ) O Polynomial Polynomial O ( ( n p ) ) n p O Exponential ( a ) Exponential O ( n ) a n O Don’t be confused … Intuitively … Don’t be confused … Intuitively … � We typically say, We typically say, � To say To say f ( n ) is O ( g ( n )) is to say that f ( n ) is O ( g ( n )) is to say that � � f ( ( n n ) is “less than or equal to” ) is “less than or equal to” g g ( ( n n ) ) ( n ) is O ( g ( n )) or f ( n ) = O ( g ( n )) f ( n ) is O ( g ( n )) or f ( n ) = O ( g ( n )) f f � We also have (G&T pp. 118 We also have (G&T pp. 118- -120): 120): � But But O O ( ( g g ( ( n n )) is really a )) is really a set set of functions. of functions. � � � It might be more clear to say, It might be more clear to say, Θ ( ( g ( n )) Big Theta “equal to” Θ g ( n )) Big Theta “equal to” � f ( ( n n ) ) є є O O ( ( g g ( ( n n )) )) f Ω ( Ω ( g g ( ( n n )) )) Big Omega Big Omega “greater than or equal to” “greater than or equal to” � But I don’t make the rules. But I don’t make the rules. � o ( ( g g ( ( n n )) )) Little o Little o “strictly less than” “strictly less than” o � Crystal clear: “ Crystal clear: “ f ( n ) is order order ( ( g ( n )” f ( n ) is g ( n )” � ω ( ( g ( n )) Little omega “strictly greater than” ω g ( n )) Little omega “strictly greater than” Big- -Omega and Big Omega and Big- -Theta Theta little o Big little o Ω is just like is just like O O except that except that f ( n ) ≥ c c g ( n ); Definition: Let Let f f and and g g be functions mapping be functions mapping N Ω f ( n ) ≥ g ( n ); Definition: N to to R R . We say that . We say that f f ( ( n n ) is ) is o o ( ( g g ( ( n n )) if )) if ⇔ g )) ⇔ f ( ( n n ) is ) is O O ( ( g g ( ( n n )) g ( ( n n ) is ) is Ω Ω ( ( f f ( ( n n )) )) f for any for any c c є є R R , c > 0 , c > 0 there exists there exists Θ Θ is both is both O O and and Ω Ω (and the constants need not (and the constants need not match); є N , n > 0 match); 0 є N , 0 > 0 n 0 n n 0 )) ∧ ∧ f )) ⇔ ⇔ f f ( ( n n ) is ) is O O ( ( g g ( ( n n )) f ( ( n n ) is ) is Ω Ω ( ( g g ( ( n n )) f ( ( n n ) is ) is Θ Θ ( ( g(n g(n )) )) such that such that f f ( ( n n ) ≤ ) ≤ c c g g ( ( n n ) for all ) for all n n є є N N , , n n ≥ ≥ n n 0 f 0 (little omega, ω (little omega, ω , is the same but with ≥ ) , is the same but with ≥ ) 4

Multiple variables Multiple variables Multiple primitives Multiple primitives for(j = 1; j <= N; ++j) = 1; j <= N; ++j) for(j for(j = 1; j <= N; ++j){ = 1; j <= N; ++j){ for(j sum += sum += A[j A[j]; ]; for(k = 1; k <= N; ++k) for(k = 1; k <= N; ++k) for(k = 1; k <= M; ++k) { = 1; k <= M; ++k) { for(k for(l = 1; l <= M; ++l) = 1; l <= M; ++l) for(l sum2 += B[j][k sum2 += B[j][k]; ]; foo(); foo (); O (N (N 2 2 M + NM M + NM 2 2 ) ) C[j][k C[j][k] = ] = B[j][k B[j][k] * ] * A[j A[j] + 1; ] + 1; O for(j = 1; j <= N; ++j) for(j = 1; j <= N; ++j) for(l = 1; l <= k; ++l) = 1; l <= k; ++l) for(l for(k = 1; k <= M; ++k) = 1; k <= M; ++k) for(k B[j][k B[j][k] ] - -= = B[j][l B[j][l]; ]; for(l = 1; l <= M; ++l) for(l = 1; l <= M; ++l) } } foo(); (); foo } } Tradeoffs: an example Another example Tradeoffs: an example Another example 0, 0 0 � I have a set of integers between 0 I have a set of integers between 0 0 0 0, 1 1 � 0, 2 2 and 1,000,000. and 1,000,000. 0, 3 3 0 1 … … � I need to store them, and I want I need to store them, and I want 1, 0 � (lots of options in between) 0 2 1, 1 O (1) lookup, insertion, and deletion. (1) lookup, insertion, and deletion. O 1, 2 1, 3 … … � Constant time and constant space, Constant time and constant space, � N N right? right? N, 0 ∅ N, 1 N, 2 … N, N 1,000,000 Does it matter? Does it matter? Big- -O and Deceit O and Deceit Big Let n Let n = 1,000, and 1 ms / operation. = 1,000, and 1 ms / operation. � Beware huge coefficients Beware huge coefficients = 1000, 1 ms/op max n in one day n = 1000, 1 ms/op max n in one day n � � Beware key lower order terms Beware key lower order terms 1 second 1 second 86,400,000 86,400,000 n n � n log log 2 10 seconds 3,943,234 10 seconds 3,943,234 � Beware when Beware when n n is “small” is “small” n n 2 n � n 2 2 17 minutes 17 minutes 9,295 9,295 n 3 n 3 12 days 12 days 442 442 n n 4 4 32 years 96 32 years 96 n 19 years 3.17 × × 10 n 10 10 3.17 10 19 years 6 6 n 301 years × 10 2 n 1.07 × 26 2 1.07 10 301 years 26 n 5