What makes a good algorithm? Does what its supposed to Analysis of - - PDF document

Analysis of Algorithms

Slides courtesy of Prof. Joe Geigel

What makes a good algorithm?

• Does what it’s supposed to
• Easy to understand, code, and debug
• Efficient

– Makes efficient use of computer’s resources

• Memory
• Speed

Suppose we’re given 2 algorithms:

• Both do what they’re supposed to,
• Both are pretty easy to understand,
• Must decide on how each algorithm

handles

– Space (memory) – Time (computation)

Complexity

• (Resource) Complexity normally refers to the rate

at which the storage or time needs grow as a function of the input to the algorithm.

– T(n) = time complexity (amount of time an algorithm will take based on input of size n) – S(n) = space complexity (amount of space an algorithm will take based on input of size n)

• For the rest of this lecture, we’ll only consider

T(n). But the discussion can also be applied to S(n).

Units

• What does time in T(n) mean?

– We could have T(n) return the actual amount of time (in µsec., for example) when run.

• May vary based on platform (OS / machine)
• May vary based on implementation language
• May vary based on compiler used
• May vary depending on what else the system is

doing.

Units

• What does T(n) return?

– Instead we say that T(n) returns the number of “basic operations” performed.

• Basic operation

– Will vary based on algorithm considered – Independent of machine/language/compiler – Examples » Number of elements compared in a search » Number of database records accessed » Number of messages sent through the network

Finding T(n)

• Can be difficult to determine an exact

number

– The size of the data may not be enough for an exact determination. – We’re really only interested in how T(n) grows as n, the size of the input, gets larger and larger.

Asymptotic Analysis

• Based on the idea that as the input to an

algorithm gets large

– The complexity will become proportional to a known function. – Notations:

• O(f(n)) (Big-O) – upper bounds on time
• Ω(f(n)) (Big-Omega) – lower bounds on time
• Θ(f(n)) (Big-Theta) – tight bounds on time

Asymptotic Analysis

• Big-O (order of)

– T(n) = O(f(n)) if and only if there are constants c0 and n0 such that

• T(n) ≤ c0f(n) for all n ≥ n0

– What this really means

• Eventually, when the size of the input gets large

enough, the upper bounds on the runtime will be no worse than proportional to the function f(n).

Asymptotic Analysis

• Big-Theta

– T(n) = Θ (f(n)) if and only if there exists constants c1, c2, and n0 such that

• c1f(n) ≤ T(n) ≤ c2f(n) for all n ≥ n0

– What this really means

• Eventually, when the size of the input gets large

enough, both the upper and lower bounds of the algorithm will be proportional to f(n).

Asymptotic Analysis

• Why these constants don’t concern us

– Accounts for the fact that different machines are faster than other

• Simple instructions execute in constant time.
• This constant can vary from machine to machine

– Really only interested on how the runtime grows with input size

Asymptotic Analysis

• Commonly used functions (Best to worst)

– O(1)

• constant growth.
• T(n) does not grow at all as the size of your input grows
• Example: indexing an array (in memory)

– O(log n)

• logarithmic growth.
• T(n) increases as input size does, but very slowly
• Proportional to the log – Doesn’t double until n reaches n2
• Example: Binary searches (like “Twenty Questions”)

Asymptotic Analysis

• Commonly used functions (Best to worst)

– O(n)

• Linear growth
• Runtime increases at same rate as input size
• When input doubles, runtime will also double
• Example: looping over a 1 dimensional array

– O(n log(n))

• n log n growth
• When input size doubles, runtime increases slightly more than

twice.

• Algorithms that split problems into smaller problems

Asymptotic Analysis

• Commonly used functions (Best to worst)

– O(n2)

• Quadratic growth
• When input size doubles, runtime quadruples
• Example: scanning an array for duplicates;

looping over a 2D array?

– O(n3)

• Cubic growth
• When input size doubles, runtime increases eightfold

Asymptotic Analysis

• Commonly used functions (Best to worst)

– O(nk)

• Polynomial growth
• As input doubles, runtime will increase by 2k
• Quadratic and Cubic are examples of polynomial growth
• Polynomial growth with k > 3 may become impractical

– O (2n)

• Exponential growth
• When input size double, runtime will increase by the square
• Indicative of a really “hard” problem.
• Algorithm is intractable except for very small input

Algorithm Efficiencies Algorithm Efficiency Asymptotic Analysis

• Examples:

– 100n = O(n) – 100n = Θ (n) – n log n = O(n2) – n log n ≠ Θ (n2) – 0.000000000001n log n ≠ O (n)

Asymptotic Analysis

• More examples

– n = O(n) – n = Θ (n) – 1000000000n = O(n) – 1000000000n = Θ(n) – 0.0000000001n = O(n) – 0.0000000001n = Θ(n) – n = O(n2) – n ≠ Θ (n2)

Asymptotic Analysis Adding Functions

• Let’s say that your algorithm does 3 tasks, one

after the other

– For task 1, T(n) = 200 – For task 2, T(n) = 20n – For task 3, T(n) = 0.2n2

• When determining the asymptotic runtime for the

complete algorithm, only the “greatest” complexity term need be considered.

– In the long run, the runtime will be dominated by task 3.

Adding Functions

• By the same token

– Polynomial functions need only consider the highest polynomial term

• Example

– n2 + 3544442134n + 20 = O(n2) – Eventually the n2 term will dominate the other terms

• Another example

– n + 200000log(n) = O(n) – Eventually the n term will dominate the log n term

Adding functions Asymptotic Analysis and Loops

• Loop within a loop

– Running times are multiplicative

for (int i=0; i < n; i++) for (int j = 0; j < n; j++) // do something

Something is done n times in the inner loop The inner loop is done n times Something is done n2 times Done n times

Asymptotic Analysis and Loops

• Loop after a loop

– Additive

for (int i=0; i < n; i++) // do something for (int j = 0; j < n; j++) // do something else

Something is done 2n times = O(n)

Best, Worst, Average Case

• Usually, an algorithm’s performance will

depend on the data on which it operates.

– Example:

• A Linear Search for an integer in an array, x

… Works pretty quickly if the item searched for is here Works not as well if the item is not in the array

Best, Worst, Average Case

• Can categorize runtimes based on the property of

the data the algorithm is run on:

– Best case

• The best that it can do
• Linear search: when int searched for is at x[0]
• Best case performance: 1 operation

– Worst case

• The worst that it can do
• Linear search: when the int searched for is not in the array
• Worst case performance: n operations

Best, Worst, Average Case

• Can categorize runtimes based on the

property of the data the algorithm is run on:

– Average case

• The average performance over all possible datasets
• Usually what you are looking for
• Unfortunately, difficult to calculate unless you know

the distribution of your data.

Best, Worst, Average Case

• Algorithms tend to be tweaked to get good

best case performance

– This doesn’t really help us if our data is not always arranged to get us the best case – Average case is usually what we want, but difficult to calculate – Best to look at worst-case since the algorithm can only do better.

Asymptotic Analysis

• Any questions?

Summary

• Analysis of Algorithms
• Complexity

– Space / Time

• Asymptotic Analysis

– Big-O / Big Theta

• Best case, worst case, average case

Next time

• Searching and sorting

– What algorithms are out there – How do they compare