Programming Design Complexity and Graphs Ling-Chieh Kung - PowerPoint PPT Presentation

The “big O” notation Complexity Terminology of graphs Graph algorithms Programming Design Complexity and Graphs Ling-Chieh Kung Department of Information Management National Taiwan University Programming Design – Complexity and Graphs 1 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Outline • Complexity • The “big O” notation • Terminology of graphs • Graph algorithms Programming Design – Complexity and Graphs 2 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Complexity • Given a task, we design algorithms. – These algorithms may all be correct. – One algorithm may be better than another one. – To compare algorithms, we compare their complexity . • Time complexity and space complexity: – Time: We hope an algorithm takes a short time to complete the task. – Space: We hope an algorithm uses a small space to complete the task. • Let’s see some examples. Programming Design – Complexity and Graphs 3 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Space complexity • Given a matrix 𝐵 of 𝑛 × 𝑜 integers, find the row whose row sum is the largest. • Two algorithms: – For each row, find the sum. Store the 𝑛 row sums, scan through them, and find the target row. – For each row, find the sum and compare it with the currently largest row sum. Update the currently largest row sum if it is larger. Programming Design – Complexity and Graphs 4 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Space complexity: algorithm 1 • Let’s implement algorithm 1: const int MAX_COL_CNT = 3; // find the row with the max row sum const int MAX_ROW_CNT = 4; int maxRowSumValue = rowSum[0]; int maxRowNumber = 1; int maxRowSum(int A[][MAX_COL_CNT], for(int i = 0; i < m; i++) int m, int n) { { if(rowSum[i] > maxRowSumValue) // calculate row sums { int rowSum[MAX_ROW_CNT] = {0}; maxRowSumValue = rowSum[i]; for(int i = 0; i < m; i++) maxRowNumber = i + 1; { } int aRowSum = 0; } for(int j = 0; j < n; j++) return maxRowNumber; aRowSum += A[i][j]; } rowSum[i] = aRowSum; } Programming Design – Complexity and Graphs 5 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Space complexity: algorithm 2 • int maxRowSum(int A[][MAX_COL_CNT], Let’s implement algorithm 2: int m, int n) { int maxRowSumValue = 0; int maxRowNumber = 0; for(int i = 0; i < m; i++) { int aRowSum = 0; for(int j = 0; j < n; j++) aRowSum += A[i][j]; if(aRowSum > maxRowSumValue) { maxRowSumValue = aRowSum; maxRowNumber = i + 1; } } return maxRowNumber; } Programming Design – Complexity and Graphs 6 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Space complexity: comparison • The two algorithms use different amounts of space: – Algorithm 1: Declaring an array and three integers. – Algorithm 2: Declaring three integers. • Algorithm 2 has the lower space complexity . Programming Design – Complexity and Graphs 7 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Time complexity • In general, people care more about time complexity. – When we say “complexity,” we mean time complexity. • Intuitively, the complexity of an algorithm can be measured by executing the algorithm and counting the running time . – Maybe you want to do this several times and calculate the average. • However, we need to remove the impact of machine capability. • We may count the number of basic operations instead. – Basic operations: declaration, assignment, arithmetic, comparisons, etc. Programming Design – Complexity and Graphs 8 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Time complexity: example • Consider the previous example. • Let’s count the number of basic operations algorithm 1. • For the first part of algorithm 1, we have 5𝑛𝑜 + 10𝑛 + 2 basic operations. int rowSum[MAX_ROW_CNT] = {0}; // (1) Decl. Assi. Arith. Comp. for(int i = 0; i < m; i++) // (2) 𝑛 𝑛 0 0 (1) { int aRowSum = 0; // (3) 1 𝑛 + 1 𝑛 𝑛 (2) for(int j = 0; j < n; j++) // (4) 𝑛 𝑛 0 0 (3) aRowSum += A[i][j]; // (5) rowSum[i] = aRowSum; // (6) 𝑛 𝑛(𝑜 + 1) 𝑛𝑜 𝑛𝑜 (4) } 0 𝑛𝑜 𝑛𝑜 0 (5) // the remaining are skipped 0 𝑛 0 0 (6) Programming Design – Complexity and Graphs 9 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Time complexity: principle • Wait… this is so tedious! And there is no need to be that precise. • Consider algorithm 1: – 5𝑛𝑜 + 10𝑛 + 2 is roughly 5𝑛𝑜 if 𝑜 is large enough. – The bottleneck is the first part (the second part has only one level of loop). – The total number of operations is roughly 5𝑛𝑜 . • Moreover, that constant 5 does not mean a lot: – It does not change when we get more integers ( 𝑛 or 𝑜 increases). • As we care the complexity of an algorithm the most when the instance size is large , we will ignore those constants and minor (non-bottleneck) parts. – We only focus on how the number of operations grow at the bottleneck . Programming Design – Complexity and Graphs 10 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Time complexity: example • int maxRowSum(int A[][MAX_COL_CNT], Let’s analyze algorithm 2. int m, int n) • The bottleneck is the two nested loops . { • int maxRowSumValue = 0; The complexity is roughly 𝑛𝑜 : int maxRowNumber = 0; – This is how the execution time would for(int i = 0; i < m; i++) grow as the input size increases. { int aRowSum = 0; • To formalize the above idea, let’s for(int j = 0; j < n; j++) introduce the “big O” notation. aRowSum += A[i][j]; if(aRowSum > maxRowSumValue) { maxRowSumValue = aRowSum; maxRowNumber = i + 1; } } return maxRowNumber; } Programming Design – Complexity and Graphs 11 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Outline • Complexity • The “big O” notation • Terminology of graphs • Graph algorithms Programming Design – Complexity and Graphs 12 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms The “big O” notation • Mathematically, let 𝑔 𝑜 ≥ 0 and 𝑕 𝑜 ≥ 0 be two functions defined for 𝑜 ∈ ℕ . We say 𝒈 𝒐 ∈ 𝑷(𝒉 𝒐 ) if and only if there exists a positive number 𝑑 and a number 𝑂 such that 𝒈 𝒐 ≤ 𝒅𝒉(𝒐) for all 𝑜 ≥ 𝑂 . • Intuitively, that means when 𝒐 is large enough, 𝒉(𝒐) will dominate 𝒈(𝒐) . • If 𝑔 𝑜 is the number of operations that an algorithms takes to complete a task, we say the algorithm’s time complexity is 𝑕(𝑜) . – We write 𝑔 𝑜 ∈ 𝑃(𝑕 𝑜 ) , but some people write 𝑔 𝑜 = 𝑃(𝑕 𝑜 ) . Programming Design – Complexity and Graphs 13 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Examples • Let 𝑔 𝑜 = 100𝑜 2 , we have 𝑕 𝑜 = 𝑜 3 , i.e., 𝑔 𝑜 ∈ 𝑃(𝑜 3 ) . – We may choose 𝑑 = 100 and 𝑂 = 1 : 100𝑜 2 ≤ 𝟐𝟏𝟏𝑜 3 for all 𝑜 ≥ 𝟐 . – We may choose 𝑑 = 1 and 𝑂 = 100 : 100𝑜 2 ≤ 𝟐𝑜 3 for all 𝑜 ≥ 𝟐𝟏𝟏 . • Let 𝑔 𝑜 = 100 𝑜 + 5𝑜 , we have 𝑕 𝑜 = 𝑜 : – We may choose 𝑑 = 6 and 𝑂 = 10 : 100 𝑜 + 5𝑜 ≤ 𝟕𝑜 for all 𝑜 ≥ 𝟐𝟏 . Let 𝑔 𝑜 = 𝑜 log 𝑜 + 𝑜 2 , we have 𝑕 𝑜 = 𝑜 2 . • • Let 𝑔 𝑜 = 10000 , we have 𝑕 𝑜 = 1 . Let 𝑔 𝑜 = 0.0001𝑜 2 , we cannot have 𝑕 𝑜 = 𝑜 : • – For any value of 𝑑 , we have 0.0001𝑜 2 > 𝑑𝑜 if 𝑜 > 10000𝑑 . • Let 𝑔 𝑜 = 2 𝑜 , we cannot have 𝑕 𝑜 = 𝑜 100 . Programming Design – Complexity and Graphs 14 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Growth of functions • In general, we may say that functions have different growth speeds . • If a function grows faster than another one, we say the former “dominates” the latter or t he former is “an upper bound” of the latter. 𝑜 5 10 50 100 1000 log 𝑜 2.32 3.32 5.64 6.64 9.97 𝑜 2.24 3.16 7.07 10.00 31.62 𝑜 5 10 50 100 1000 𝑜 log 𝑜 11.61 33.22 282.19 664.39 9965.78 𝑜 2 25 100 2500 10000 1000000 2 𝑜 1.13 × 10 15 1.27 × 10 30 1.07 × 10 301 32 1024 3.04 × 10 64 9.33 × 10 157 𝑜! 120 3628800 Too big!! Programming Design – Complexity and Graphs 15 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Growth of functions Programming Design – Complexity and Graphs 16 / 54 Ling-Chieh Kung (NTU IM)

The “big O” notation Complexity Terminology of graphs Graph algorithms Growth of functions Programming Design – Complexity and Graphs 17 / 54 Ling-Chieh Kung (NTU IM)