TDDE25 a baker, supply of ingredients, baking utensils, an oven, - - PowerPoint PPT Presentation

TDDE25

Fö 8 Chap 5: Algorithms Chap 12: Theory of Computation

https://liuonline.sharepoint.com/sites/Lisam_TDDE25_2020HT_FZ

Algorithmics and Computability Part I

Patrick Doherty Dept of Computer and Information Science Artificial Intelligence and Integrated Computer Systems Division 1

What is an Algorithm?

• Let’s focus on gastronomy!
• Given a kitchen containing
• a baker, supply of ingredients, baking utensils, an oven, etc.
• Baking a cake is
• a process that is carried out from the ingredients, by the

baker, with the aid of the oven, and most significantly, according to the recipe!

• Ingredients: input to the process
• Cake: output of the process
• Recipe: the algorithm (an abstract entity)
• the formal written version: program (Software)
• Utensils, oven, baker: (Hardware)

2

Baking a Cake

Ingredients Cake Recipe

Oven Utensils Baker

Effective Process

Hardware Software Input Output Computational Thinking!

3

A Real Recipe: Mousseline au chocolat

Melt chocolate and 2 tablespoons water in double boiler. When melted stir in powdered sugar; add butter bit by bit. Set aside. Beat egg yolks until thick and lemon-colored, about 5 minutes. Gently fold in chocolate. Reheat slightly to melt chocolate, if necessary. Stir in Rum and vanilla. Beat egg whites until foamy. Beat in 2 tablespoons sugar; beat until stiff peaks form.Gently fold whites into chocolate-yolk mixture. Pour into individual serving dishes. Chill at least 4 hours. Serve with whipped cream, if desired. 8 ounces of semisweet chocolate pieces, 2 tablespoons of water, 1/4 cup of powdered sugar, 6 separated eggs, etc. 6-8 servings of Mousseline au chocolat 4

Level of Detail: Basic Actions

“Stir in powdered sugar”

Basic action because it is assumed the hardware knows how to do it.

Level of detail is very important when it comes to an algorithm’s elementary instructions.

• Should be tailored to fit the hardware’s capabilities
• Should also be appropriate for the comprehension level
• f the potential reader or user of the algorithm.

Ambiguities or Fuzzy Phrases not allowed in specifying instructions to a computer!

“about 5 minutes”

“served with whipped cream if desired”

Is it the serving or addition of WC that is dependent on person’s desires?

5

Short Algorithmic Representations for Long Processes

Given a list of personnel records, one for each employee in a certain company, each containing the employee’s name, personal details, and salary, find the total sum of all salaries of all employees

• 1. Make a note of the number 0;
• 2. proceed through the list, adding each employee’s salary to the noted

number;

• 3. having reached the end of the list, produce the noted number as output.

Problem: Algorithm:

The text of the algorithm is short and fixed, yet the process it invokes varies with the length of the input list and can be very, very long! A fixed algorithm prescribing many processes of varying lengths, the precise duration and nature of the process being dependent

• n the inputs to the algorithm.

In fact, the potential choice of inputs may be infinite!

We have:

6

The Algorithmic Problem and its Solution

Characterization of all legal inputs Characterization of desired outputs as a function of inputs

Algorithmic Problem and Algorithmic Solution

Any legal input Desired Output Algorithm A 7

The Algorithmic Problem and its Solution

• An algorithmic problem consists of:
• a characterization of a legal, possibly infinite, collection of potential

input sets, and

• a specification of the desired outputs as a function of the inputs
• Assumption
• a description of allowed basic actions or a hardware configuration,

together with its built-in basic actions is provided in advance.

• Each of the actions must be carried out in a finite amount of time.
• An algorithmic solution consists of:
• an algorithm, composed of elementary instructions prescribing

actions from the agreed upon set

• the algorithm, when executed for any legal input set, solves the

problem, producing the output as required.

8

Solving Algorithmic Problems

• Algorithmic problems can be incredibly

complex and take years of work to solve satisfactorily

• Many problems do not admit satisfactory

solutions

• Many problems do not admit solutions at all!
• For many problems, the status as far as good

algorithmic solutions is still unknown.

9

Course Book: Definition of an Algorithm

An algorithm is an ordered set of unambiguous finitely executable steps that defines a terminating process

An algorithm must contain control instructions that determine the sequence or order in which the executable steps or instructions are carried out

10

Control Flow or Ordering in Program Execution

• Sequencing: Statements are to be executed (or expressions evaluated)

in a certain specified order.

• Selection: Depending on some runtime condition, a choice is to be

made among two or more statements or expressions

• The most common selection constructs are if and case (switch)
• Iteration: A given fragment of code is executed repeatedly, either a

certain number of times, or until a certain run-time condition is true.

• Iteration constructs include for/do, while, and repeat loops
• Procedural Abstraction: A potentially complex collection of control

constructs ( a subroutine) is encapsulated in a way that allows it to be treated as a single unit, usually subject to parameterization

• Recursion: An expression is defined in terms of (simpler versions of )

itself, either directly or indirectly

• the computational model requires a stack on which to save

information about partially evaluated instances of the expression.

11

Control Flow or Ordering in Program Execution cont’d

• Concurrency: Two or more program fragments are to be executed/evaluated

“at the same time”

• either in parallel on separate processors,
• or interleaved on a single processor in a way that achieves the same effect.
• Exception handling and speculation: A program fragment is executed
• ptimistically, on the assumption that the expected condition will be true. If the

condition turns out to be false

• execution branches to a handler that executes in place of the remainder of

the protected fragment ( in the case of exception handling), or

• in place of the entire protected fragment (in the case of speculation)
• In this case, the language implementation must be able to rollback or

undo any visible effects of the protected code.

• Nondeterminacy: The ordering or choice among expressions or statements is

deliberately left unspecified, implying that any alternative will lead to correct results.

12

Control Flow or Ordering in Program Execution cont’d

• These 8 principle categories cover all the

control-flow constructs and mechanisms found in most programming languages

• Though the syntax and semantics details vary

from language to language, thinking in these terms will make it easier to

• learn new languages
• evaluate the tradeoffs among languages
• and design and reason about algorithms in a

language-independent way!

13

Computational Problems

• An “algorithm” is an informal intuitive concept, but associated with it

is the concept of a computational process.

• Many attempts have been made to provide formal definitions of what

the most general mathematical notion of a computational process is.

• In other words, a formal equivalent of the informal notion of an

algorithm.

• For example:
• A Turing machine that halts on all inputs is the precise formal

notion corresponding to the intuitive notion of an algorithm.

• The area that studies such issues is called Computability Theory
• It asks the fundamental question:
• What can be computed by a computational device?

14

Computational Problems

Computational Problems that CAN be solved by an algorithm

Computational Problems that CANNOT be solved by any algorithm

Computability Theory

CANNOT be solved in any practical sense due to excessive time/space requirements

Computational Problems that CAN be solved by an algorithm

CAN be solved in a practical sense with reasonable time/space requirements

Computational Complexity Theory

Tractable Intractable Class NP Class P

15

Computability Theory

16

Some History and Context

17

Leibniz (1646-1716)

• A universal artificial mathematical language
• All human knowledge could be represented

in this language

• Calculational rules would reveal all logical

relationships among these propositions

• Machines would be capable of carrying
• ut such calculations

Calculus Ratiocinator Let us Calculate!

Addition Subtraction Multiplication Square root extraction

Binary Arithmetic

18

Boole (1815 - 1864)

Turned “Logic” into Algebra

Classes and terms (thoughts) could be manipulated using algebraic rules resulting in valid inferences

Logical deduction could be developed as a branch

• f mathematics

Subsumed Aristotle’s syllogisms In essence Leibniz’ calculus rationator (lite) Boolean Logic

19

Hilbert (1862 - 1943)

23rd Problem: Does there exist an algorithm that can determine the truth or falsity of any logical proposition in a system of logic that is powerful enough to represent the natural numbers? (Entscheidungsproblem)

2nd Problem: Establish the consistency of the axioms for the arithmetic of real numbers 1st Problem: Decide the truth of Cantor’s Continum Hypothesis

24 problems for the 20th century

20

Countability

Q R

Dense Continuous

∞

Q R

Countable Uncountable

Continuum Hypothesis (Cantor) - There is no set whose cardinality is strictly between that of the (positive) integers and that of the real numbers.

Are some infinite sets bigger than others? ℕ ⊆ ℚ ⊂ ℝ

21

Metamathematics

Inference Proof Theory Entailment Model Theory Semantics Syntax

Soundness Completeness Consistency

Can not infer both ω and its negation

Δ ⊢ ω Δ ⊧ ω Not too strong Strong enough Correct

22

Hilbert’s Program

Logic from the inside Formal axiomatic theories Peano Arithmetic

Logic from the

• utside

Metamathematics Proof Theory

Consistency Completeness Decidability, etc

Business as usual Only use Finitist Methods Is 1st-order logic complete? Is PA complete?

23

Frege (1848 -1925)

The 1st fully developed system of logic encompassing all of the deductive reasoning in ordinary mathematics.

• 1st example of formal artificial

language with formal syntax

• logical inference as purely mechanical
• perations (rules of inference)

Begriffsschrift “Concept Script”

Intention was to show that all of mathematics could be based on logic! (Logicism)

24

Gödel (1906 - 1978)

Showed the completeness

• f 1st-order logic in his PhD Thesis

The logic of PM (and consequently PA) is incomplete There are true sentences not provable within the logical system

As a consequence, the consistency of the mathematics of the real numbers can not be proven within any system as strong as PA

Develop metamathematics inside a formal logical system by encoding propositions as numbers Hilbert’s 2nd Problem

PM: Principia Mathematica (Russell) PA: Peano Arithmetic

25

Gödel’s Argument

U is a proposition that states that “U is not provable in PM”. Assume: Anything provable in PM is True

1. U is true: Suppose U were false. Then what it says would be false. So U would have to be provable, and therefore True (assumption above). This contradicts the supposition that U is false. 2. U is not provable in PM: Since U is true, what it says must be true. 3. The negation of U is not provable in PM: Because U is true, its negation (that U is provable) must be false, and therefore the negation of U is not provable in PM. U is a true (from the outside [1]) proposition, but an undecidable (from the inside [2,3]) proposition.

Self-referential:

26

Turing (1912-1954)

Turing wanted to disprove the 23rd problem

23rd Problem: Does there exist an algorithm that can determine the truth or falsity of any logical proposition in a system of logic that is powerful enough to represent the natural numbers? (Entscheidungsproblem) To do this, he had to come up with a formal characterization of the generic process underlying the computation of an algorithm

He then showed that there were functions that were NOT effectively computable including the Entscheidungsproblem!

As a byproduct he found a mathematical model

• f an all-purpose computing machine!

27

Effective Computation

What is it, that humans do, when humans “compute”? What actions are undertaken by a human agent when manipulating a finite set of symbols according to fixed rules? Strip this undertaking of any particular notion of mental activity, thought, imagination, creativity.... What are the essentials of human computation itself?

Observe!: This is 1930. There are no computers in existence!

28

Turing machine

• Blueprint for a Turing Machine (1st of 2 parts):
• Architectural Part:
• An infinitely long tape divided into squares stretching in two directions
• there will never be a shortage of space
• A finite set of symbols
• 0 and 1 (can use any finite alphabet)
• executes absolutely simple, primitive operations on these symbols.
• A reading/writing head
• Can scan squares one at a time
• Can move to the left or right one square at a time, or not at all
• Can inscribe, or erase symbols on squares
• A finite set of states
• The states correspond to various finite configurations of a Turing

machine’s reading/writing head.

29

Turing Machine

• Blueprint for a Turing Machine (2nd of 2 parts):
• Procedural Part:
• The behavior of a Turing machine is controlled by a finite series of

instructions

• Each instruction is governed by
• the state of the reading head and
• the current symbol being scanned
• The Turing machine understands only two commands:
• The first instructs the machine what to write or erase
• The second, whether it is to move to the left or the right by one

square, or stay where it is.

• Each command given to a Turing machine has 4 parts:
• If the machine is in this state, and it is scanning that symbol,

then it must write this symbol or that symbol, and move one square to the left or one square to the right, or not at all.

30

Effective Computability: Turing Machine

• finite alphabet of symbols
• finite set of states
• infinite tape marked off with squares

each of which is capable of carrying a single symbol

• mobile sensing-and-writing head

that can travel along the tape one square at a time

• state-transition diagram

containing the instructions that cause changes to take place at each step

Claim: Any effective computation could be described as a Turing machine

31

Course Book Example

A Turing Machine for Incrementing a Value

Uses a state transition table instead of finite state diagram * 1 1 *

1 *

32

Another Example: Palindromes

move-a move-b test-a test-b mark YES NO return

a/a, R b/b, R a/#, R b/#, R a/a, R b/b, R #/#, L #/#, L #/#, L #/#, L a/#, L b/#, L a/a, L b/b, L #/#, L #/#, R b/b, L a/a, L

a b b a # a b a a #

Examples:

Can generalize to any finite alphabet Can encode decision problems (recall hilbert’s 23rd problem!)

# #

33

Functions

• Function: A correspondence between a collection of possible

input values and a collection of possible output values so that each possible input is assigned a single output

• Computing a Function: Determining the output value for each input

value associated with a given set of input values

• Effectively Computable Function: Output values can be

determined algorithmically from input values.

• Turing Computable Function: Any function that can be computed

from a Turing machine in the following manner:

• Place input value in binary form on a tape
• Run the Turing machine associated with the function until it halts
• Read the output value from the tape

Claim: Any effectively computable function is Turing Computable

34

On Algorithms and Computable Functions

• The notion of an algorithm is a human artifact
• It stems from the human concept of effectively getting

something done (mentally).

• In essence, it may be synonymous with anything a

human mind can effectively do.

• Controversial!
• The notion of computability is a formally defined

mathematical concept.

• Turing Computability is one such formal definition.
• For any computable function, there is a Turing

machine which effectively computes that function.

35

Church-Turing Thesis

Turing machines are capable of solving any effectively computable algorithmic problem! Put differently, any algorithmic problem for which we can find an algorithm that can be programmed in some programming language, any language, running on some computer, any computer, even one that has not yet been built, and even one requiring unbounded amounts of time and memory space for ever larger inputs, is also solvable by a Turing machine!

Turing Machine

Scheme, C+ +, C, Ada, Phython, Java, Ruby, C#, LISP ,....... Pentium, Multicore, Big Blue, super computer, ... Partial Recursive Functions: Gödel,Kleene Lambda Calculus: Church Post Production Systems: Post Turing Machines: Turing Unlimited Register Machines: Cutland

A Truly Astonishing and Remarkable (formally well-grounded )Claim!

Note: This would apply to Quantum Computers too!

36

Uncomputable Functions?

• Are there well-defined functions that are not effectively computable?
• If so, how would we prove such a thing?
• Turing did just that!
• First he required a formal mathematical notion of effective computability.
• Turing Machine
• His well-defined problem was that asked by Hilbert:
• Does there exist an algorithm that can determine the truth or falsity of any

logical proposition in a system of logic that is powerful enough to represent the natural numbers? (Entscheidungsproblem)

• He formally proved that the function (decision problem) associated with this

question is not effectively computable. There is no Turing machine that can compute this function!

• The fact that there are functions which are not computable, places limitations on

what a computer can do.

• The result may in fact place limitations on what the human mind can do

also....but this is a controversial (but important) topic.

37

An Unsolvable Problem

R X Does R halt

• n X?

Yes No If R(X) terminates If R(X) diverges A Program Potential Input

Halting Problem

There is no effective algorithm that, given an arbitrary program and arbitrary input can determine if the program will halt on the input

38

Universal Turing machine

• One consequence of the Church-Turing thesis is the existence of

universal algorithms.

• A universal algorithm A has the ability to act like any algorithm

whatsoever

• Input: the description of any algorithm P
• Input: any legal input value X, to the algorithm P
• It runs, or simulates P(X), halting if P(X) halts with the outputs that

P(X) would normally give had P been running on X

• A computer or interpreter is very much like a universal algorithm
• It is presented with a program P in a language with legal input X, to P;

It executes P on X and outputs the result.

• Stored Program Concept (program as data)
• Universal though means that it is insensitive to the choice of language or

machine, which a computer or interpreter is not.

39

Universal Turing Machine

Formal mathematical abstraction of a general computing device

Turing’s Ace Computer algorithm A program P input X Run P

• n X

P implements A; is written in language L2 Universal program U, written in language L1; simulates the effect of a program in L2 on an input

LISP: Eval Programs as data

Interpreter for Turing Machines

Functional Programming: Python, LISP Observe: U can simulate any computer in existence Observe: L1 can simulate algorithms in its own language L1

40

Universal Programming Language

• It would be tedious to represent algorithms at such a low level
• f abstraction using Turing Machines.
• A Universal Programming Language is one in which a

solution to any computable function can be expressed

• Most popular programming languages are universal in

this sense.

• Turing Completeness
• A system of data-manipulation rules (such as a computer’s

instruction set, a programming language, etc.) is said to be:

• Turing complete or computationally universal if it can be

used to simulate any single-taped Turing machine. 41

Course Book: Bare Bones Language

• Variables:
• refer to bit patterns interpreted as non-negative

integers

• Three Assignment Statements:
• clear name [bind name to 0]
• incr name [increment name with 1]
• decr name [decrement name with 1]
• One control structure:
• while name not 0 do: <statement sequence> end;

42

Some Examples

Multiply X x Y Copy Today to Tomorrow 43

Universality of Bare Bones

• Bare Bones Language has been shown

formally to be Turing Complete

• Relation to a Turing machine:
• Input: Initial values of all variables
• Output: Final values of all variables
• The Bare Bones program itself directs the

computation of a function

44

Back to the Halting Problem

R

Encode the program R in bare bones as a bit sequence

We define a program as self-terminating if executing the program with all its variables initialized to the program’s encoded representation leads to a terminating process

Testing a Program for Self-Termination

Observe! Any program is either self-terminating or not self-terminating 45

Examples: Testing a Program for Self-Termination

While X not 0 do; incr X; end: Clear X While X not 0 do; incr X; end:

Binding X to the program’s encoded representation results in the program not halting

By definition it is not self-terminating

Binding X to the program’s encoded representation results in the program halting

By definition it is self-terminating Self-referential Programs

46 Proof Structure: Proving the Unsolvibility of the Halting Problem Assume: Existence of program P

Modify P to P’ Assume P’ is self-terminating Assume P’ is not self-terminating P’ is not self-terminating P’ is self-terminating

⊥ ⊥

Existence of program P is impossible 47

Proving the Unsolvability of the Halting Problem

First: Propose the existence

• f a program that,

given any encoded version of a program will halt with variable X equal to 1 if the input represents a self-terminating program, or with X equal to 0 otherwise.

Proposed Program

Assume:

Then: If such a program exists, we could modify it by adding a while-end structure

Proposed Program

to produce a new program

while X not 0 do; end;

Now: If this new program were self-terminating and execution would reach this point with X equal to 1

Proposed Program

we started it with its own encoding as its input

while X not 0 do; end;

so execution would become trapped in this loop forever i.e., if the new program is self-terminating, then it is not self-terminating!!!

Case 1

However: If this new program were not self-terminating and execution would reach this point with X equal to 0

Proposed Program

we started it with its own encoding as its input

while X not 0 do; end;

so execution of this loop would be skipped i.e., if the new program is not self-terminating, then it is self-terminating!!! and execution would halt;

Case 2

Consequently:

the existence of a new program

Proposed Program

The existence of the proposed program

while X not 0 do; end;

that is neither self-terminating nor non self- terminating So the existence of the proposed program is impossible! would lead to

Proposed Program

Contradiction Negate the assumption

48

Conclusions

• Gödel’s result:
• Any formal system expressive enough to axiomatize arithmetic is and always will be

incomplete!

• There are mathematical formulas which are true but can not be shown within the

formal system to be true!

• One such formula is the consistency of the axioms of arithmetic themselves.
• Turing’s result:
• There is no algorithm that can determine the truth or falsity of any logical proposition

in a system of logic that is powerful enough to represent the natural numbers (Entscheidungsproblem)

• As a side effect:
• Turing proposed a formal model of effective computability
• Turing defined a mathematical abstraction of a computing device
• Both Gödel and Turing formally showed the limits to formal systems and the degree of

mechanization possible.

• If mind is mechanism, what does this say about human intelligence?
• What does this say about the goals of artificial intelligence?

49

Computational Problems

Computational Problems that CAN be solved by an algorithm

Computational Problems that CANNOT be solved by any algorithm

Computability Theory

CANNOT be solved in any practical sense due to excessive time/space requirements

Computational Problems that CAN be solved by an algorithm

CAN be solved in a practical sense with reasonable time/space requirements

Computational Complexity Theory

Tractable Intractable Class NP Class P

50