Week 1 Oliver Kullmann Organisation Computationa Getting Started - - PowerPoint PPT Presentation

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Week 1 Getting Started

1

Organisation

2

Computational problems and algorithms

3

Insertion sort

4

Algorithm analysis

5

Exactly analysing Insertion-Sort

6

Discrete mathematics

7

Exercises

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Organisation

Lecturer Dr Oliver Kullmann Lectures Tuesday 9-10 Vivian 404 Tuesday 15-16 Glyndwr E Wednesday 10-11 Vivian 404 Thursday 11-12 Glyndwr E Labs (choose one of the following two sessions) Thursday 12-13 Faraday 217 (Linux Lab) Thursday 13-14 Faraday 217 (Linux Lab)

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Course home page

http://www.cs.swan.ac.uk/~csoliver/Algorithms201213/index.html

with messages general information up-to-date versions of the slides (which serve as the script) lab sheets course works.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Assessment

There will be two pieces of coursework, each worth 10%. Labs are an essential part of the course. Successful participations is worth 10%. The written examination counts 70%, it will take place in January.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Lectures

The script will usually be handed out at the beginning of each week. Corrected version of the script are available on the course home-page. Usually, they will not be printed out again, except in cases where substantial changes have

• ccurred.

Left-over print-outs of the script can be found in the student’s office (Faraday Building Room 206) for those who could not attend the lecture. You also have to sign the module attendance form.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

References

There are very many algorithms textbooks in print. In this module, we will follow (and refer to as CLRS) Introduction to Algorithms (Third Edition) by Cormen, Leiserson, Rivest and Stein, The MIT Press, 2009. LIS Call Number: QA76.6. I58 2009 (10 copies on 1 week loan, 2 copies on 1 night loan)

http://en.wikipedia.org/wiki/Introduction_to_Algorithms

Reading from CLRS for week 1

Chapter 1 Chapter 2, Sections 2.1, 2.2

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Computational problems

Computational Problem: A specification in general terms of inputs and outputs and the desired input/output relationship. Problem Instance: An actual set of inputs for a given problem.

Example 1

Problem: Sorting (numbers). Input: A sequence of n numbers a1, a2, . . . , an . Output: A permutation (reordering) a′

1, a′ 2, . . . a′ n

• f the input such that a′

1 ≤ a′ 2 ≤ . . . ≤ a′ n.

Example instance: Input 10, −15, 3, 3, 70, −17, −30 Output −30, −17, −15, 3, 3, 10, 70

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Algorithmic solution

Algorithm: a well-defined computational procedure that takes some value, or sequence of values, as input and produces some values, or sequence of values, as output. There will always be many different algorithms for any given problem. A program is a particular implementation of some algorithm. A program is not the same as an algorithm. There are many different implementations of any given algorithm.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Expressing algorithms

We express algorithms in whatever way is the clearest and most concise. English is sometimes the best way. When issues of control need to be made perfectly clear, we often use pseudocode. Pseudocode is similar to any typical imperative programming language, such as Java, C, C++, Pascal, . . . Pseudocode expresses algorithms to humans. Software engineering issues of data abstraction, modularity, error handling are often ignored. We sometimes embed English statements into pseudocode. Therefore, unlike for “real” programming language, we cannot create a compiler that translates pseudocode to machine code.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Insertion Sort

Example 2

Insertion Sort is a good algorithm for sorting a small number of elements. It works the way you might sort a hand of playing cards. Insertion-Sort(A) 1 for j = 2 to A.length 2 key = A[j] 3 / / Insert A[j] into sorted sequence A[1 . . j−1]. 4 i = j−1 5 while i > 0 and A[i] > key 6 A[i+1] = A[i] 7 i = i−1 8 A[i+1] = key

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Algorithm analysis

We want to predict the resources that the algorithm requires. Typically, we are interested in runtime memory number of basic operations, such as:

arithmetic operations (eg, for multiplying matrices) bit operations (eg, for multiplying integers) comparisons (eg, for sorting and searching)

In order to predict resource requirements, we need a computational model.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Random-access machine (RAM) model

Instructions are executed one after another. No concurrent

• perations.

It’s too tedious to define each of the instructions and their associated time costs. Instead, we recognise that we’ll use instructions commonly found in real computers:

Arithmetic: add, subtract, multiply, divide, remainder, floor,

• ceiling. Also, shift left/shift right (good for

multiplying/dividing by 2k). Data movement: load, store, copy. Control: conditional/unconditional branch, subroutine call and return.

Each of these instructions takes a constant amount of time.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Random-access machine (RAM) model (cont’d)

The RAM model uses integer and floating-point types We don’t worry about precision, although it is crucial in certain numerical applications. There is a limit on the word size: when working with inputs

• f size n, assume that integers are represented by c lg n bits

for some constant c ≥ 1. (lg n is a very frequently used shorthand for log2 n.)

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

How do we analyse running time?

The time taken by an algorithm depends on the input. Sorting 1000 numbers takes longer than sorting 3 numbers. A given sorting algorithm may even take differing amounts

• f time on two inputs of the same size.

For example, we’ll see that insertion sort takes less time to sort n elements when they are already sorted than when they are in reverse sorted order. The input size depends on the problem studied. Usually the number of items in the input. Like the size n of the array being sorted. But could be something else. If multiplying two integers, could be the total number of bits in the two integers. Could be described by more than one number.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

The running time on a particular input is the number of primitive operations (steps) executed. Want to define steps to be machine-independent. Each line of pseudocode requires a constant amount of time. One line may take a different amount of time than another, but each execution of line i takes the same amount of time ci. Assumption: lines consist only of primitive operations.

If the line is a subroutine call, then the actual call takes constant time, but the execution of the subroutine being called might not. If the line specifies operations other than primitive ones,then it might take more than constant time.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Analysing Insertion-Sort

Insertion-Sort(A) cost repetitions 1 for j = 2 to A.length c1 n 2 key = A[j] c2 n−1 3 / / Insert A[j] into sorted n−1 sequence A[1 . . j − 1]. 4 i = j−1 c4 n−1 5 while i > 0 and A[i] > key c5 n

j=2 tj

6 A[i+1] = A[i] c6 n

j=2(tj−1)

7 i = i−1 c7 n

j=2(tj−1)

8 A[i+1] = key c8 n−1

where: n = A.length, and tj = number of times the while -loop test in line 5 is executed in the jth iteration of the for -loop.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Applying the exact analysis

Best possible situation: tj = 1 for all j, i.e., when A is already sorted. Runtime: (c1 + c2 + c4 + c5 + c8)n − (c2 + c4 + c5 + c8) . Worst possible situation: tj = j for all j, i.e., when A is sorted in reverse order. Runtime: ( c5

2 + c6 2 + c7 2 ) n2

+ (c1 + c2 + c4 + c5

2 − c6 2 − c7 2 + c8) n

− (c2 + c4 + c5 + c8) . Average situation: tj = j

2 for all j, i.e., every permutation is

equally likely, so expected value of tj is j

2.

Runtime: ( c5

4 + c6 4 + c7 4 )n2

+ (c1 + c2 + c4 + c5

4 − 3c6 4 − 3c7 4 + c8)n

− (c2 + c4 + c5

2 − c6 2 − c7 2 + c8) .

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Remarks

Types of Analysis: exact analysis (rarely achieved) best-case worst-case average-case. Worst-case versus average-case analysis: We usually concentrate

• n finding the worst-case running time. Average case is often

as bad as the worst case. Order of growth is another abstraction to ease analysis and focus on the important features. Will be discussed next week.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Mathematical Functions

We shall assume you are comfortable with standard math functions, like exponentiation bx and its inverse logb x. logb a is the number x such that bx = a. We shall usually work with binary logarithms lg x = log2 x. Some Useful Identities

logb(xy) = logb x + logb y logb(xy) = y logb x logb x logc x = logb c

We shall often use floor ⌊x⌋ and ceiling ⌈x⌉ functions: ⌊x⌋ is the largest integer ≤ x, e.g., ⌊5.3⌋ = 5 ⌈x⌉ is the smallest integer ≥ x, e.g., ⌈5.3⌉ = 6 as well as summation notation:

n

• i=1

ai = a1 + a2 + a3 + · · · + an .

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Two Revealing Tables

Time to solve a problem instance of size n using a T(n)-time algorithm.

T(n) n=10 n=20 n=50 n=100 n=500 n=1000 n lg n 33 µs 86 µs 282 µs 664 µs 4.5 ms 10 ms n2 100 µs 400 µs 2.5 ms 10 ms 250 ms 1 s 2n 1 ms 1 s 36 yr 1016 yr 10137 yr 10287 yr n! 4 s 105 yr 1051 yr 10144 yr 101120 yr 102554 yr

Largest problem instance solvable in 1 minute.

T(n) 2× faster machine 103× faster machine n lg n 3.9×106 7.5×106 2.7×109 n2 7 745 10 954 244 948 2n 25 26 35 n! 11 11 13

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Running insertion sort

Organise in pairs, write down the following sequence 12, 14, 5, 4, 1, 14, 5, 3, 13, 10 and sort it by insertion-sort, counting the comparisons and the assignments — as usual in this context,

• nly considering the “real objects”.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Running insertion sort (cont.)

Showing number of comparisons and assignments for each insertion step, together with the next element to be inserted: 12, 14, 5, 4, 1, 14, 5, 3, 13, 10

1 12, 14, 5, 4, 1, 14, 5, 3, 13, 10 : (1, 1) 2 5, 12, 14, 4, 1, 14, 5, 3, 13, 10 : (2, 4) 3 4, 5, 12, 14, 1, 14, 5, 3, 13, 10 : (3, 5) 4 1, 4, 5, 12, 14, 14, 5, 3, 13, 10 : (4, 6) 5 1, 4, 5, 12, 14, 14, 5, 3, 13, 10 : (1, 1) 6 1, 4, 5, 5, 12, 14, 14, 3, 13, 10 : (4, 5) 7 1, 3, 4, 5, 5, 12, 14, 14, 13, 10 : (7, 8) 8 1, 3, 4, 5, 5, 12, 13, 14, 14, 10 : (3, 4) 9 1, 3, 4, 5, 5, 10, 12, 13, 14, 14 : (5, 6)

: (30, 40)

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Logarithms

The powers 2n for 0 ≤ n ≤ 32: 1 131072 2 262144 4 524288 8 1048576 16 2097152 32 4194304 64 8388608 128 16777216 256 33554432 512 67108864 1024 134217728 2048 268435456 4096 536870912 8192 1073741824 16384 2147483648 32768 4294967296 65536

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Some binary logarithms

1 lg(103) ≈ 10 (since 210 = 1024) 2 lg(128 · 103) = lg(128) + lg(103) = 7 + lg(103) ≈ 17 3 lg(106) = lg(103 · 103) = lg(103) + lg(103) ≈ 20 4 lg(7 · 106) = lg(7) + lg(106) ≈ 2.8 + 20 = 22.8 5 lg(109) = lg(103) + lg(106) ≈ 10 + 20 = 30 6 lg(1010) = lg(10) + lg(109) ≈ 3.3 + 30 = 33.3

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Number of binary digits

Consider a natural number n ≥ 0: Develop a formula for the number b(n) ≥ 1

• f binary digits of n.

Hint: use lg.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Number of binary digits (cont.)

b(n) =

• 1

if n = 0 ⌊lg(n)⌋ + 1 if n ≥ 1 Here are the values of b(n) for n ∈ { 0, . . . , 16 }: 1, 1, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 5

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Sums

Develop a formula for f (n) :=

n

• i=1

i. Start by creating a table of values.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Sums (cont.)

The list of values of f (n) for n = 0, . . . , 10: 0, 1, 3, 6, 10, 15, 21, 28, 36, 45, 55. The differences between these values are 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Since these differences grow, the original function is not linear (not something like a · n + b). However this list of differences is linear, and so the original function is quadratic (something like a · n2 + b · n + c). Playing around yields f (n) =

n

• i=1

i = 1 2n(n + 1).

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics Exercises

Other algorithms (homework)

Can you imagine other algorithms for sorting? In what sense could they improve insertion-sort? How good could they be?