Greedy Algorithms Chapter 16 1 Optimization Problems A class of - - PowerPoint PPT Presentation

Greedy Algorithms

Chapter 16

1

Optimization Problems

A class of problems in which we are asked to find a set (or a sequence) of “items” that satisfy some constraints and simultaneously

• ptimize (i.e., maximize or minimize) some
• bjective function

2

Example: Bin Packing

10

2 3 4 5 6

3

Example: Bin Packing

8

2 3 4 5 6

4

Example: Bin Packing

4

2 3 4 5 6

5

Example: Bin Packing

1

2 3 4 5 6

6

Example: Bin Packing

1

2 3 4 5 6

5

7

Example: Bin Packing

1

2 3 4 5 6

5 4

Number of bins = 3

8

Example: Bin Packing

2 3 4 5 6

(Optimal Solution) Number of bins = 2

9

The Greedy Method

Applied to optimization problems Adds items to the solution one-by-one Builds up towards the final solution No backtracking Not necessarily optimal!

10

Activity Selection

a.k.a. Task Scheduling

11

Activity Selection Problem

Given a set of activities 𝑇 = {𝑏1, 𝑏2, … , 𝑏𝑜} where each activity 𝑗 has a start time 𝑡𝑗 and a finish time 𝑔

𝑗, where 0 ≤ 𝑡𝑗 < 𝑔 𝑗 < ∞. An

activity 𝑏𝑗 happens in the half-open time interval [𝑡𝑗, 𝑔

𝑗). Two activities are said to be

compatible if they do not overlap. The problem is to find a maximum-size compatible subset, i.e., a one with the maximum number of activities.

12

Example of Activity Selection

13

A Solution

14

A Better Solution

15

An Optimal (Best) Solution

16

Another Optimal Solution

17

“Greedy” Strategies

Longest first Shortest first Early start first Early finish first …

18

Early Finish Greedy Strategy

• 1. Sort activities by finish time
• 2. Schedule the first activity
• 3. Remove all incompatible activities
• 4. If there are more activities, repeat 2

19

Early Finish

20

Early Finish

21

Early Finish

22

Early Finish

23

Early Finish

24

Early Finish

25

Early Finish

26

Early Finish

27

Optimality of the Greedy Choice

To prove optimality of the greedy choice, we have to prove the following two properties

• 1. Greedy Choice: The greedy choice is part
• f the answer
• 2. Optimal Substructure: The optimal solution

to the big problem contains the optimal solution to the sub-problem

28

Greedy Choice

Let 𝐵 ⊆ 𝑇 be an optimal solution. Let 𝑏𝑘 be the first element in 𝐵 and 𝑏𝑛 be the first element in 𝑇. We want to prove that 𝑏𝑛 is part of an

• ptimal solution.

If 𝑏𝑛 = 𝑏𝑘 then we are done Otherwise, we prove that there is another

• ptimal solution 𝐵` = 𝐵 − 𝑏𝑘 ∪ 𝑏𝑛

Is 𝐵` a solution? yes Is 𝐵` optimal? yes

29

Optimal Substructure

We want to prove that, if 𝐵 ⊆ 𝑇 is an optimal solution to 𝑇, then 𝐵 − 𝑏𝑛 is an optimal solution to S` = 𝑏𝑗: 𝑏𝑗 ∈ 𝑇 ∧ 𝑡𝑗 > 𝑔

𝑛

Proof by contradiction Assume that 𝐵 − 𝑏𝑛 is not optimal Then, there is another solution 𝐶 to 𝑇` such that 𝐶 > 𝐵 − 𝑏𝑛 ⇒ |𝐶| ≥ |𝐵| If this is the case, then the subset 𝐵` = 𝑏𝑛 ∪ 𝐶 is also a solution to 𝑇 𝐵` > 𝐶 ⇒ |𝐵`| > 𝐵 , which means that 𝐵 is not optimal which is a contradiction

30

Knapsack Problem

31

0-1 Knapsack Problem

$4,500 15LBs $1,500 3LBs $800 2LBs $3,000 10LBs $4,000 20LBs



45LBs

32

0-1 Knapsack Problem

$4,500 15LBs $1,500 3LBs $800 2LBs $3,000 10LBs $4,000 20LBs



45LBs

33

0-1 Knapsack Problem

$4,500 15LBs $1,500 3LBs $800 2LBs $3,000 10LBs $4,000 20LBs



45LBs

$300/LB $500/LB $400/LB $200/LB $300/LB

34

0-1 Knapsack Problem

$4,500 15LBs $1,500 3LBs $800 2LBs $3,000 10LBs $4,000 20LBs



45LBs

$300/LB $500/LB $400/LB $200/LB $300/LB

35

Total value = $9,800

0-1 Knapsack Problem

$4,500 15LBs $1,500 3LBs $800 2LBs $3,000 10LBs $4,000 20LBs



45LBs

$300/LB $500/LB $400/LB $200/LB $300/LB

36

Total value = $11,500

There isn’t a known solution to the 0-1 Knapsack problem using a greedy algorithm

Fractional Knapsack Problem

$4,500 15 oz $1,500 3 oz $800 2 oz $3,000 10oz $4,000 20oz



45 oz

$300/oz $500/oz $400/oz $200/oz $300/oz

37

Problem Formulation

Given a set 𝑇[1. . 𝑜] of items where each item 𝑗 has a weight 𝑥𝑗 and a value 𝑤𝑗, we would like to find the amount 𝑦𝑗 of each item that we can take to maximize the total value 𝑊 = ෍

𝑗=1 𝑜

𝑤𝑗 𝑦𝑗 𝑥𝑗 Under the following two constraints

0 ≤ 𝑦𝑗 ≤ 𝑥𝑗 σ𝑗=1

𝑜

𝑦𝑗 ≤ 𝑋

38

Pseudo-code

Fractional-Knapsack

Compute the value-per-weight 𝑧𝑗 =

𝑤𝑗 𝑥𝑗 for all items

Sort items by 𝑧𝑗 Set 𝑀 = 0 While (𝑀 < 𝑋)

Select the item 𝑘 with the highest 𝑧𝑘 Set 𝑦𝑘 = 𝑁𝑗𝑜 𝑋 − 𝑀, 𝑥

𝑘

Remove item 𝑘 Set 𝑀 = 𝑀 + 𝑦𝑘

Set all remaining 𝑦𝑗’s to 0

39

Greedy-choice Property

Given the set 𝑇 ordered by the value-per- weight (𝑧), taking as much as possible 𝑦𝑘 from the item 𝑘 with the highest value-per- weight will lead to an optimal solution Assume there is an optimal solution 𝑌′ where we take less amount of item 𝑘, say 𝑦𝑘′ < 𝑦𝑘. We prove that there is another solution 𝑌′′ where we take 𝑦𝑘 of item 𝑘 and get a similar

• r a higher total value 𝑊

40

Greedy-choice Property

Since 𝑦𝑘′ < 𝑦𝑘, there must be another item 𝑙 which was taken in an amount that accounts for the difference, i.e., 𝑦′𝑙 We create another solution 𝑌′′ by doing the following changes in 𝑌′ Reduce the amount of item 𝑙 by a value 𝑨 𝑦𝑙

′′ = 𝑦𝑙 ′ − 𝑨

Increase the amount of item 𝑘 by a value 𝑨 𝑦𝑘

′′ = 𝑦𝑘 ′ + 𝑨 = 𝑦𝑘 (𝑦𝑘 is the greedy choice)

41

Greedy Choice Property

σ 𝑦𝑗

′ = σ 𝑦𝑗 ′′ ⇒ (both are valid solutions)

𝑊′ = σ 𝑦𝑗

′𝑧𝑗, 𝑊′′ = σ 𝑦𝑗 ′′𝑧𝑗

𝑊′′ − 𝑊′ = 𝑧𝑘𝑦𝑘

′′ + 𝑧𝑙𝑦𝑙 ′′ − 𝑧𝑘𝑦𝑘 ′ + 𝑧𝑙𝑦𝑙 ′

(All other items are the same) 𝑊′′ − 𝑊′ = 𝑧𝑘 𝑦𝑘

′′ − 𝑦𝑘 ′ − 𝑧𝑙 𝑦𝑙 ′ − 𝑦𝑙 ′′

But, xj

′′ − 𝑦𝑘 ′ = 𝑦𝑙 ′ − 𝑦𝑙 ′′ = 𝑨

𝑊′′ − 𝑊′ = 𝑧𝑘 − 𝑧𝑙 𝑨 Since 𝑧𝑘 ≥ 𝑧𝑙 and 𝑨 = 𝑦𝑘

′′ − 𝑦𝑘 > 0

𝑊′′ ≥ 𝑊′, hence, 𝑌′′ is also optimal

42

Greedy Choice Illustration

𝑇 𝑘 Ordered by 𝑧 𝑌′ 𝑘 𝑥

𝑘

𝑦𝑘

′

𝑦𝑙

′

𝑌′′ 𝑘 𝑦𝑘

′′ = 𝑦𝑘

𝑘 𝑦𝑙

′′

Input Optimal answer that does not have the greedy choice Optimal answer that has the greedy choice

43

Optimal Sub-structure

Given the problem 𝑇 with an optimal solution 𝑌, we want to prove that the solution 𝑌` = 𝑌 − 𝑦𝑘 is optimal to the problem 𝑇` after removing the item 𝑘 and updating the capacity 𝑋` = 𝑋 − 𝑦𝑘 Proof by contradiction Assume that 𝑌` is not optimal to 𝑇` There is another solution 𝑌`` to 𝑇` that has a higher total value 𝑊`` > 𝑊` This means we can have a better solution to the problem 𝑇 which is impossible because 𝑌 is

• ptimal

44

Optimal substructure

𝑇 𝑘 𝑥

𝑘

Input Ordered by 𝑧 𝑌 𝑘 Optimal answer for 𝑇 𝑇′ Reduced problem 𝑌′ Should be an

• ptimal answer for

𝑇′

45

Optimal Substructure

Input problem 𝑇 with an optimal answer 𝑌 of total value 𝑊 𝑌` = 𝑌 − 𝑦𝑘 , 𝑊` = 𝑊 − 𝑦𝑘𝑧𝑘 Assume 𝑌`` is an optimal solution to 𝑇` 𝑋`` = σ𝑦``𝑗 ≤ 𝑋 − 𝑦𝑘 and 𝑊`` > 𝑊` 𝑌`` + 𝑦𝑘 is a valid solution to 𝑇 because 𝑋`` + 𝑦𝑘 ≤ 𝑋 𝑊`` = 𝑊`` + 𝑦𝑘𝑧𝑘 > 𝑊′ + 𝑦𝑘𝑧𝑘 > V This is a contradiction with the assumption that 𝑌 is an optimal solution for 𝑇

46

Huffman Codes

47

Encoding

How data is represented? Fixed-size codes, e.g., ASCII

A: 1000001 B: 1000010

Variable-size codes, e.g., Morse Codes

A: ●▬ B: ▬●●● E: ● T: ▬

48

Example: Morse Code

49

Prefix Codes

No code is allowed to be a prefix of another code To encode, simply concatenate all the codes Decoding does not entail any ambiguity Example:

Message ‘JAVA’ a = “0”, j = “11”, v = “10” Encoded message “110100” Decoding “110100”

51

Trie

We can use a trie to find prefix codes the characters are stored at the external nodes a left child (edge) means 0 a right child (edge) means 1

A = 010 B = 11 C = 00 D = 10 R = 011 D B C R 1 1 1 1 A

52

Example of Decoding

encoded text: 01011011010000101001011011010 text: ABRACADABRA

ASCII: 88 bits Our encoding: 29 bits

A = 010 B = 11 C = 00 D = 10 R = 011 D B C R 1 1 1 1 A

53

Another Encoding

Message: ‘ABRACADABRA’ Encoded message: ‘001011000100001100101100’ Length: 24 bits

B R A D 1 1 1 1 C

54

Optimal Encoding Problem

Given a set 𝐷 of 𝑜 characters, for each character c ∈ 𝐷. Let c. 𝑔𝑠𝑓𝑟 be the frequency

• f 𝑑 in the file. We would like to find a prefix

encoding for each 𝑑 ∈ 𝐷 with a length 𝑒𝑈 𝑑 such that we minimize the total cost 𝐶 = ෍

𝑑∈𝐷

𝑑. 𝑔𝑠𝑓𝑟 × 𝑒𝑈 𝑑 Solution: Huffman Codes

55

Example

A,5 B,2 C,1 D,1 R,2

“ABRACADABRA”

56

Example

A,5 B,2 C,1 D,1 R,2 2

“ABRACADABRA”

57

Example

A,5 B,2 C,1 D,1 R,2 2 4

“ABRACADABRA”

58

Example

A,5 B,2 C,1 D,1 R,2 2 4 6

“ABRACADABRA”

59

Example

A,5 B,2

“ABRACADABRA”

C,1 D,1 R,2 2 4 6 11

60

Example

A,5 B,2

“ABRACADABRA”

C,1 D,1 R,2 2 4 6 11

1 1 1 1

61

Example

A,5 B,2

“ABRACADABRA”

C,1 D,1 R,2 2 4 6 11

1 1 1 1

A=0 B=10 C=1100 D=1101 R=111

62

Encoding

A,5 B,2 C,1 D,1 R,2 2 4 6 11

1 1 1 1

A=0 B=10 C=1100 D=1101 R=111

“ABRACADABRA” 0 10 111 0 1100 0 1101 0 10 111 0 Length= 23 Optimal!

63

Encoding

A,5 R,2 C,1 D,1 B,2 2 4 6 11

1 1 1 1

A=0 R=10 C=1100 D=1101 B=111

“ABRACADABRA” 0 111 10 0 1100 0 1101 0 111 10 0 Length= 23 Optimal!

64

Construction of Huffman Tree

Huffman(C)

n=|C| Q=C for i = 1 to n-1

allocate a new node z z.left = x = Extract-Min(Q) z.right = y = Extract-Min(Q) z.freq = x.freq + y.freq Insert(Q, z)

return Extract-Min(Q) // Root of the tree

Θ log 𝑜 Θ log 𝑜 n Θ 𝑜 log 𝑜

Note: Can also be done in linear time

𝑈 𝑜 = Θ 𝑜 log 𝑜 𝑈 𝑜 = Θ 𝑜 log 𝑜 + 𝑁

65

Optimality of Huffman Codes

Greedy-choice

The greedy choice yields an optimal solution.

Optimal sub-structure

The optimal solution for the bigger problem contains the optimal solution of the sub-problem.

Detailed proof in the textbook

66

Greedy Choice

Greedy choice: Choose the characters 𝑦 and 𝑧 with the highest frequencies and merge them under one internal node We need to prove that the optimal tree has 𝑦 and 𝑧 as siblings under one common node Assume there is an optimal tree 𝑈 where 𝑦 and 𝑧 are not siblings In the same tree, the two siblings at the deepest level are other characters 𝑏 and 𝑐 Assume 𝑦. 𝑔𝑠𝑓𝑟 ≤ 𝑧. 𝑔𝑠𝑓𝑟 and 𝑏. 𝑔𝑠𝑓𝑟 ≤ 𝑐. 𝑔𝑠𝑓𝑟

67

Greedy Choice

𝑧 𝑦

𝑏 𝑐 𝑈

𝑧 𝑏

𝑦 𝑐 𝑈`

𝑐 𝑏

𝑦 𝑧 𝑈``

68