COMP 3403 Algorithm Analysis Part 6 Chapters 11 and 12 Jim - - PowerPoint PPT Presentation

COMP 3403 — Algorithm Analysis Part 6 — Chapters 11 and 12

Jim Diamond CAR 409 Jodrey School of Computer Science Acadia University

Chapter 11

Limitations of Algorithm Power: Computational Complexity

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 188

Review and Looking Forward

• So far, we have looked at

– techniques for analyzing the time complexity and space complexity

• f a given algorithm

– a number of algorithm design techniques – – the analysis of most of these algorithms

• We must also consider the following question:

are (any of) these algorithms the best possible?

• In other words, what is the minimum possible amount of time (or space)

required to solve a given problem? – that is, what is the lower bound of the complexity of any algorithm which solves the given problem?

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 189

Lower Bounds

• Lower bound: an estimate on a minimum amount of work needed to

solve a given problem

• Examples:

– the number of comparisons needed to find the largest element in a set of n numbers – the number of comparisons needed to sort an array of size n – – the number of multiplications needed to multiply two n × n matrices

• A lower bound can be

– – an efficiency class (Ω())

• A lower bound is said to be tight if there exists an algorithm with the

same efficiency as the lower bound

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 190

Lower Bound Examples

Problem Lower Bound Tight? sorting

Ω(n log n)

yes searching in a sorted array

Ω(log n)

yes element uniqueness

Ω(n log n)

yes

n-digit integer multiplication Ω(n)

no multiplication of n × n matrices

Ω(n2)

no

• For example, we saw that we could multiply two n × n matrices in

O(n2.72) time (roughly)

– but there is no obvious reason why we need more than n2 time to do this – goal: either find a O(n2) algorithm for matrix multiplication,

• r prove that ω(n2) time is required

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 191

Techniques for Establishing Lower Bounds

• Trivial lower bounds
• Information-theoretic arguments (decision trees)
• Adversary arguments
• Problem reduction

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 192

Trivial Lower Bounds

• Idea: just counting the amount of given input or required output may

give us a useful lower bound

• Examples

– to multiply two n × n matrices, we must read 2 · n2 input numbers and write n2 output numbers: thus we need at least n2 time (i.e., Ω(n2) time) – to find the maximum number in an array of n numbers, each number must be examined, taking Ω(n) time – – polynomial evaluation: p(x) = anxn + an−1xn−1 + · · · + a1x + a0 to evaluate this, we must look at every coefficient, and thus we have to do Ω(n) work (tight, because we have O(n) algorithms for this)

• Counting the amount of input is not always useful; e.g.,

– – the amount of input/output may be tiny compared to any known algorithm (such as in the travelling salesman problem)

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 193

Information Theoretic Arguments

• Information theory deals with (among other things) the amount of

information contained in a given collection of data

• Q: how much information is in this image? That is, if you were talking

to someone on the phone and had to describe this image to them, how much would you have to say so that they could draw the same picture?

• A 390-byte PostScript program creates this

–

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 194

Information Theoretic Arguments

• An information theoretic approach to lower bounds uses the amount of

information which any algorithm solving the problem must generate – e.g., the game of guessing a number m between 0 and n − 1 there are n possible answers, which means we need ⌈log2 n⌉ bits to represent the answer – if each question we ask (e.g., “is m bigger than 31”) gains one bit of information, we must ask ⌈log2 n⌉ questions, at a minimum, on average – note: if you ask a question like “is m 42?” you might be lucky and get the answer with one question, but if the guess is wrong you have gained very little information, increasing the number of

• verall questions which you must ask
• Note that you can ask your questions in such a way that you fill in the

bits of the binary representation of m – – Q: if the answer is no, what is the next question?

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 195

Information Theoretic Arguments: Decision Trees

• Many algorithms for sorting and searching data involve a sequence of

comparison operations, applied to the input data

• We can represent these algorithms pictorially; example:

Decision tree for finding a minimum of three numbers

• Note that in this case there are more leaves on the tree than there are

possible answers (note ⌈log2 3⌉ = 2, we must ask 2 questions, the tree must have height at least 2)

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 196

Information Theoretic Arguments: Decision Trees: 2

• Here is a decision tree for selection sort

–

• Note: selection sort does redundant comparisons!

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 197

Information Theoretic Arguments: Decision Trees: 3

• A decision tree’s leaves may not all be at the same depth

– Decision tree for the three-element insertion sort

• Note that no redundant comparisons are done!
• In the worst case, ⌈log2 6⌉ = 3 questions are needed

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 198

Information Theoretic Arguments: Decision Trees: 4

• Consider the problem of sorting n distinct numbers:

there are n! possible permutations of n numbers – sorting these numbers is equivalent to finding the permutation which sorts the numbers

APL sort operator!

• The decision tree for such a sort must have at least n! leaves

– this means the height of the tree (== the number of comparisons done from the root to the farthest leaf) must be ≥ ⌈log2 n!⌉ – thus the worst-case number of comparisons Cworst(n) ≥ ⌈log2 n!⌉

• Applying Stirling’s formula, we get

⌈log2 n!⌉ ≈ log2 √ 2πn

n

e

n

= n log2 n−n log2 e+log2 n 2 +log2 2π 2 ≈ n log2 n

Thus about n log2 n comparisons are required in the worst case by any comparison-based sorting algorithm

• A more difficult analysis shows Cavg(n) ≥ log2 n!

Surprising?

• Since we have O(n log2 n) (average and worst case) sorting algorithms,

these bounds are (asymptotically) tight

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 199

Lower Bounds by Problem Reduction

• Idea: if some problem P is at least as hard as another problem Q, then

a lower bound for Q is also a lower bound for P – common mistake: reversing this and saying “. . . then a lower bound for P is also a lower bound for Q”

• To use the problem reduction idea, find a problem Q with a known

lower bound that can be reduced† to your problem P in question – – the reduction must show how we can turn any instance of Q into an instance of P , so that a solution to the P instance gives us a solution to the Q instance

† the reduction procedure must follow some technical rules for validity

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 200

Lower Bounds by Problem Reduction: Example

• Example: suppose

P is the problem of finding a MST for n points in the Cartesian plane Q is the element uniqueness problem

–

• This reduction from Q to P proceeds as follows:

– take an instance of Q: {x1, x2, . . . , xn} – create a corresponding instance of P : {(x1, 0), (x2, 0), . . . , (xn, 0)} – –

• bserve:

the Q instance has two equal elts ⇐

⇒ P ’s MST has a 0-length edge

(GEQ: why?) – thus we can conclude that

P (MST) must be at least as hard as Q (uniqueness)

– and thus (the time complexity of) MST is in Ω(n log n)

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 201

Adversary Arguments

• Also known as “devil’s advocate” arguments
• Idea: you employ your algorithm to try to solve some problem without

seeing the data

⇒

the operations are done by an “adversary”, who does the calculations, announces the results of comparisons, and so on – the adversary must play “fairly” in the sense that there must be some set of data which would result in the stated results

• One way to think about this is an adversary which generates bad data in

“real time”

• Example: in quicksort, it is necessary to pick a “good” pivot to avoid

O(n2) behaviour

– – the adversary would generate data for which the median of those three values is the second smallest of all values in the sub-array – repeating this, your quicksort would run in O(n2) time

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 202

Computational Complexity: The Class P

• We are interested in dividing problems into three categories:

– – those which can be solved, but not “efficiently”, and – those which can’t be solved at all

• For this categorization to be useful, we must define what we mean by

“efficiently”

• We say an algorithm solves a problem in polynomial time if, in the

worst case, the algorithm’s run time is O(p(n)), where p() is a polynomial and n is some “reasonable” measure of the problem’s size, when the input is given in some “reasonable” way –

• Definition: the class P is the set of all problems solvable in polynomial

time – these problems are said to be tractable –

• ther problems are said to be intractable

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 203

Computation Complexity: Rationale for P

• Early in the term we examined plots for various functions such as 2n, n!,

n2 and so on

• We saw that functions like 2n and n! grow very quickly

– algorithms with such running times will terminate in a “reasonable” amount of time only for “very small” n, even with very fast computers

• You might argue that algorithms with time complexity like n100 are also

impractical –

• It turns out that there are “few” non-pathological problems with

polynomial-time algorithms where there is no O(n3) algorithm to solve the problem – you can find such problems, but you have to look a bit

• Consequently, it is practical to refer to problems which have

polynomial-time solutions as tractable

Jim Diamond, Jodrey School of Computer Science, Acadia University

Chapter 11 204

Computational Complexity: Problem Types

• We often discuss these two types of problems:
• Decision problems: these problems have a Yes/No answer

– e.g., does this graph have a clique of size 5? – e.g., is there a solution to a linear programming problem where the

• bjective function is > 10?

– e.g., is there a winning strategy for player 1 in a given game? – e.g.,

• Optimization problems: these problems usually have a numeric answer

– e.g., what is the size of the largest clique in this graph? – e.g., what is the largest value of the objective function for a given linear programming problem? – e.g., what is the smallest number of colours I need to colour this graph?

• Note: decision problems often have optimization versions

(and vice versa)

Jim Diamond, Jodrey School of Computer Science, Acadia University