Week 1 Oliver Kullmann Organisation Computationa Getting Started - - PowerPoint PPT Presentation

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Week 1 Getting Started

1

Organisation

2

Computational problems and algorithms

3

Insertion sort

4

Algorithm analysis

5

Exactly analysing Insertion-Sort

6

Discrete mathematics

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Organisation

Lecturer Dr Oliver Kullmann Lectures Monday 13 Glyndwr C Tuesday 9 Glyndwr B Wednesday 10 Grove 330 Thursday 11 Wallace JLT Labs (choose one of the following two sessions) Thursday 12 Faraday 217 (Linux Lab) Thursday 13 Faraday 217 (Linux Lab)

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Course home page

http://www.cs.swan.ac.uk/~csoliver/Algorithms201112/index.html

with messages general information up-to-date versions of the slides (which serve as the script) lab sheets course works.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Assessment

There will be two pieces of coursework, each worth 10%. Labs are an essential part of the course. Successful participations is worth 10%. The written examination counts 70%, it will take place in January.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Lectures

The script will usually be handed out at the beginning of each week. Corrected version of the script are available on the course home-page. Usually, they will not be printed out again, except in cases where substantial changes have

• ccurred.

Left-over print-outs of the script can be found in the student’s office (Faraday Building Room 206) for those who could not attend the lecture. You also have to sign the module attendance form.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

References

There are very many algorithms textbooks in print. In this module, we will follow (and refer to as CLRS) Introduction to Algorithms (Third Edition) by Cormen, Leiserson, Rivest and Stein, The MIT Press, 2009. LIS Call Number: QA76.6. I58 2009 (10 copies on 1 week loan, 2 copies on 1 night loan) http://en.wikipedia.org/wiki/Introduction_to_Algorithms

Reading from CLRS for week 1

Chapter 1 Chapter 2.1, 2.2

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Computational problems

Computational Problem: A specification in general terms of inputs and outputs and the desired input/output relationship. Problem Instance: An actual set of inputs for a given problem.

Example 1

Problem: Sorting (numbers). Input: A sequence of n numbers a1, a2, . . . , an . Output: A permutation (reordering) a′

1, a′ 2, . . . a′ n

• f the input such that a′

1 ≤ a′ 2 ≤ . . . ≤ a′ n.

Example instance: 10, −15, 3, 3, 70, −17, −30.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Computational problems

Computational Problem: A specification in general terms of inputs and outputs and the desired input/output relationship. Problem Instance: An actual set of inputs for a given problem.

Example 1

Problem: Sorting (numbers). Input: A sequence of n numbers a1, a2, . . . , an . Output: A permutation (reordering) a′

1, a′ 2, . . . a′ n

• f the input such that a′

1 ≤ a′ 2 ≤ . . . ≤ a′ n.

Example instance: 10, −15, 3, 3, 70, −17, −30.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Algorithmic solution

Algorithm: a well-defined computational procedure that takes some value, or set of values, as input and produces some values, or set of values, as output. A program is a particular implementation of some algorithm. A program is not the same as an algorithm. There will always be many different algorithms for any given problem.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Algorithmic solution

Algorithm: a well-defined computational procedure that takes some value, or set of values, as input and produces some values, or set of values, as output. A program is a particular implementation of some algorithm. A program is not the same as an algorithm. There will always be many different algorithms for any given problem.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Expressing algorithms

We express algorithms in whatever way is the clearest and most concise. English is sometimes the best way. When issues of control need to be made perfectly clear, we often use pseudocode. Pseudocode is similar to any typical imperative programming language, such as Java, C, Pascal, . . . Pseudocode is designed for expressing algorithms to

• humans. Software engineering issues of data abstraction,

modularity, error handling are often ignored. We sometimes embed English statements into pseudocode. Therefore, unlike for “real” programming language, we cannot create a compiler that translates pseudocode to machine code.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Expressing algorithms

We express algorithms in whatever way is the clearest and most concise. English is sometimes the best way. When issues of control need to be made perfectly clear, we often use pseudocode. Pseudocode is similar to any typical imperative programming language, such as Java, C, Pascal, . . . Pseudocode is designed for expressing algorithms to

• humans. Software engineering issues of data abstraction,

modularity, error handling are often ignored. We sometimes embed English statements into pseudocode. Therefore, unlike for “real” programming language, we cannot create a compiler that translates pseudocode to machine code.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Expressing algorithms

We express algorithms in whatever way is the clearest and most concise. English is sometimes the best way. When issues of control need to be made perfectly clear, we often use pseudocode. Pseudocode is similar to any typical imperative programming language, such as Java, C, Pascal, . . . Pseudocode is designed for expressing algorithms to

• humans. Software engineering issues of data abstraction,

modularity, error handling are often ignored. We sometimes embed English statements into pseudocode. Therefore, unlike for “real” programming language, we cannot create a compiler that translates pseudocode to machine code.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Expressing algorithms

We express algorithms in whatever way is the clearest and most concise. English is sometimes the best way. When issues of control need to be made perfectly clear, we often use pseudocode. Pseudocode is similar to any typical imperative programming language, such as Java, C, Pascal, . . . Pseudocode is designed for expressing algorithms to

• humans. Software engineering issues of data abstraction,

modularity, error handling are often ignored. We sometimes embed English statements into pseudocode. Therefore, unlike for “real” programming language, we cannot create a compiler that translates pseudocode to machine code.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Insertion Sort

Example 2

Insertion Sort is a good algorithm for sorting a small number of elements. It works the way you might sort a hand of playing cards. Insertion-Sort(A) 1 for j = 2 to A.length 2 key = A[j] 3 / / Insert A[j] into sorted sequence A[1 . . j−1]. 4 i = j−1 5 while i > 0 and A[i] > key 6 A[i+1] = A[i] 7 i = i−1 8 A[i+1] = key

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Insertion Sort

Example 2

Insertion Sort is a good algorithm for sorting a small number of elements. It works the way you might sort a hand of playing cards. Insertion-Sort(A) 1 for j = 2 to A.length 2 key = A[j] 3 / / Insert A[j] into sorted sequence A[1 . . j−1]. 4 i = j−1 5 while i > 0 and A[i] > key 6 A[i+1] = A[i] 7 i = i−1 8 A[i+1] = key

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Algorithm analysis

We want to predict the resources that the algorithm requires. Typically, we are interested in runtime. memory. number of basic operations, such as:

arithmetic operations (eg, for multiplying matrices). bit operations (eg, for multiplying integers). comparisons (eg, for sorting and searching).

In order to predict resource requirements, we need a computational model.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Algorithm analysis

We want to predict the resources that the algorithm requires. Typically, we are interested in runtime. memory. number of basic operations, such as:

arithmetic operations (eg, for multiplying matrices). bit operations (eg, for multiplying integers). comparisons (eg, for sorting and searching).

In order to predict resource requirements, we need a computational model.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Algorithm analysis

We want to predict the resources that the algorithm requires. Typically, we are interested in runtime. memory. number of basic operations, such as:

arithmetic operations (eg, for multiplying matrices). bit operations (eg, for multiplying integers). comparisons (eg, for sorting and searching).

In order to predict resource requirements, we need a computational model.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Random-access machine (RAM) model

Instructions are executed one after another. No concurrent

• perations.

It’s too tedious to define each of the instructions and their associated time costs. Instead, we recognise that we’ll use instructions commonly found in real computers:

Arithmetic: add, subtract, multiply, divide, remainder, floor,

• ceiling. Also, shift left/shift right (good for

multiplying/dividing by 2k). Data movement: load, store, copy. Control: conditional/unconditional branch, subroutine call and return.

Each of these instructions takes a constant amount of time.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Random-access machine (RAM) model

Instructions are executed one after another. No concurrent

• perations.

It’s too tedious to define each of the instructions and their associated time costs. Instead, we recognise that we’ll use instructions commonly found in real computers:

Arithmetic: add, subtract, multiply, divide, remainder, floor,

• ceiling. Also, shift left/shift right (good for

multiplying/dividing by 2k). Data movement: load, store, copy. Control: conditional/unconditional branch, subroutine call and return.

Each of these instructions takes a constant amount of time.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Random-access machine (RAM) model (cont’d)

The RAM model uses integer and floating-point types We don’t worry about precision, although it is crucial in certain numerical applications. There is a limit on the word size: when working with inputs

• f size n, assume that integers are represented by c lg n bits

for some constant c ≥ 1. (lg n is a very frequently used shorthand for log2 n.)

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

How do we analyse running time?

The time taken by an algorithm depends on the input. Sorting 1000 numbers takes longer than sorting 3 numbers. A given sorting algorithm may even take differing amounts

• f time on two inputs of the same size.

For example, we’ll see that insertion sort takes less time to sort n elements when they are already sorted than when they are in reverse sorted order. The input size depends on the problem studied. Usually the number of items in the input. Like the size n of the array being sorted. But could be something else. If multiplying two integers, could be the total number of bits in the two integers. Could be described by more than one number.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

How do we analyse running time?

The time taken by an algorithm depends on the input. Sorting 1000 numbers takes longer than sorting 3 numbers. A given sorting algorithm may even take differing amounts

• f time on two inputs of the same size.

For example, we’ll see that insertion sort takes less time to sort n elements when they are already sorted than when they are in reverse sorted order. The input size depends on the problem studied. Usually the number of items in the input. Like the size n of the array being sorted. But could be something else. If multiplying two integers, could be the total number of bits in the two integers. Could be described by more than one number.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

The running time on a particular input is the number of primitive operations (steps) executed. Want to define steps to be machine-independent. Each line of pseudocode requires a constant amount of time. One line may take a different amount of time than another, but each execution of line i takes the same amount of time ci. Assumption: lines consist only of primitive operations.

If the line is a subroutine call, then the actual call takes constant time, but the execution of the subroutine being called might not. If the line specifies operations other than primitive ones,then it might take more than constant time.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

The running time on a particular input is the number of primitive operations (steps) executed. Want to define steps to be machine-independent. Each line of pseudocode requires a constant amount of time. One line may take a different amount of time than another, but each execution of line i takes the same amount of time ci. Assumption: lines consist only of primitive operations.

If the line is a subroutine call, then the actual call takes constant time, but the execution of the subroutine being called might not. If the line specifies operations other than primitive ones,then it might take more than constant time.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Analysing Insertion-Sort

Insertion-Sort(A) cost repetitions 1 for j = 2 to A.length c1 n 2 key = A[j] c2 n−1 3 / / Insert A[j] into sorted n−1 sequence A[1 . . j − 1]. 4 i = j−1 c4 n−1 5 while i > 0 and A[i] > key c5 n

j=2 tj

6 A[i+1] = A[i] c6 n

j=2(tj−1)

7 i = i−1 c7 n

j=2(tj−1)

8 A[i+1] = key c8 n−1

where: n = A.length, and tj = number of times the while -loop test in line 5 is executed in the jth iteration of the for -loop.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Applying the exact analysis

Best possible situation: tj = 1 for all j, i.e., when A is already sorted. Runtime: (c1 + c2 + c4 + c5 + c8)n − (c2 + c4 + c5 + c8) . Worst possible situation: tj = j for all j, i.e., when A is sorted in reverse order. Runtime: ( c5

2 + c6 2 + c7 2 ) n2

+ (c1 + c2 + c4 + c5

2 − c6 2 − c7 2 + c8) n

− (c2 + c4 + c5 + c8) . Average situation: tj = j

2 for all j, i.e., every permutation is

equally likely, so expected value of tj is j

2.

Runtime: ( c5

4 + c6 4 + c7 4 )n2

+ (c1 + c2 + c4 + c5

4 − 3c6 4 − 3c7 4 + c8)n

− (c2 + c4 + c5

2 − c6 2 − c7 2 + c8) .

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Remarks

Types of Analysis: exact analysis (rarely achieved) best-case worst-case average-case. Worst-case versus average-case analysis: We usually concentrate

• n finding the worst-case running time. Average case is often

as bad as the worst case. Order of growth is another abstraction to ease analysis and focus on the important features. Will be discussed next week.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Remarks

Types of Analysis: exact analysis (rarely achieved) best-case worst-case average-case. Worst-case versus average-case analysis: We usually concentrate

• n finding the worst-case running time. Average case is often

as bad as the worst case. Order of growth is another abstraction to ease analysis and focus on the important features. Will be discussed next week.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Remarks

Types of Analysis: exact analysis (rarely achieved) best-case worst-case average-case. Worst-case versus average-case analysis: We usually concentrate

• n finding the worst-case running time. Average case is often

as bad as the worst case. Order of growth is another abstraction to ease analysis and focus on the important features. Will be discussed next week.

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Mathematical Functions

We shall assume you are comfortable with standard math functions, like exponentiation bx and its inverse logb x. logb a is the number x such that bx = a. We shall usually work with binary logarithms lg x = log2 x. Some Useful Identities

logb(xy) = logb x + logb y logb(xy) = y logb x logb x logc x = logb c

We shall often use floor ⌊x⌋ and ceiling ⌈x⌉ functions: ⌊x⌋ is the largest integer ≤ x, e.g., ⌊5.3⌋ = 5 ⌈x⌉ is the smallest integer ≥ x, e.g., ⌈5.3⌉ = 6 as well as summation notation:

n

• i=1

ai = a1 + a2 + a3 + · · · + an .

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Two Revealing Tables

Time to solve a problem instance of size n using a T(n)-time algorithm.

T(n) n=10 n=20 n=50 n=100 n=500 n=1000 n lg n 33 µs 86 µs 282 µs 664 µs 4.5 ms 10 ms n2 100 µs 400 µs 2.5 ms 10 ms 250 ms 1 s 2n 1 ms 1 s 36 yr 1016 yr 10137 yr 10287 yr n! 4 s 105 yr 1051 yr 10144 yr 101120 yr 102554 yr

Largest problem instance solvable in 1 minute.

T(n) 2× faster machine 103× faster machine n lg n 3.9×106 7.5×106 2.7×109 n2 7 745 10 954 244 948 2n 25 26 35 n! 11 11 13

CS 270 Algorithms Oliver Kullmann Organisation Computationa problems and algorithms Insertion sort Algorithm analysis Exactly analysing Insertion- Sort Discrete mathematics

Two Revealing Tables

Time to solve a problem instance of size n using a T(n)-time algorithm.

T(n) n=10 n=20 n=50 n=100 n=500 n=1000 n lg n 33 µs 86 µs 282 µs 664 µs 4.5 ms 10 ms n2 100 µs 400 µs 2.5 ms 10 ms 250 ms 1 s 2n 1 ms 1 s 36 yr 1016 yr 10137 yr 10287 yr n! 4 s 105 yr 1051 yr 10144 yr 101120 yr 102554 yr

Largest problem instance solvable in 1 minute.

T(n) 2× faster machine 103× faster machine n lg n 3.9×106 7.5×106 2.7×109 n2 7 745 10 954 244 948 2n 25 26 35 n! 11 11 13