Week 2: Greedy Algorithms Karan Singh 373F19 - Karan Singh 1 - - PowerPoint PPT Presentation

β–Ά
week 2 greedy algorithms
SMART_READER_LITE
LIVE PREVIEW

Week 2: Greedy Algorithms Karan Singh 373F19 - Karan Singh 1 - - PowerPoint PPT Presentation

May 28, 2023 171 likes β€’653 views

CSC373 Week 2: Greedy Algorithms Karan Singh 373F19 - Karan Singh 1 Recap Divide & Conquer Master theorem Counting inversions in ( log ) Finding closest pair of points in 2 in log Fast

slide-1
SLIDE 1

CSC373 Week 2: Greedy Algorithms

373F19 - Karan Singh 1

Karan Singh

slide-2
SLIDE 2

Recap

373F19 - Karan Singh 2

  • Divide & Conquer
  • Master theorem
  • Counting inversions in 𝑃(π‘œ log π‘œ)
  • Finding closest pair of points in ℝ2 in 𝑃 π‘œ log π‘œ
  • Fast integer multiplication in 𝑃 π‘œlog2 3
  • Fast matrix multiplication in 𝑃 π‘œlog2 7
  • Finding π‘™π‘’β„Ž smallest element (in particular, median) in

𝑃(π‘œ)

slide-3
SLIDE 3

Greedy Algorithms

373F19 - Karan Singh 3

  • Greedy (also known as myopic) algorithm outline
  • We want to find a solution 𝑦 that maximizes some
  • bjective function 𝑔
  • But the space of possible solutions 𝑦 is too large
  • The solution 𝑦 is typically composed of several parts (e.g.

𝑦 may be a set, composed of its elements)

  • Instead of directly computing 𝑦…
  • Compute it one part at a time
  • Select the next part β€œgreedily” to get maximum immediate benefit

(this needs to be defined carefully for each problem)

  • May not be optimal because there is no foresight
  • But sometimes this can be optimal too!
slide-4
SLIDE 4

Interval Scheduling

373F19 - Karan Singh 4

  • Problem
  • Job π‘˜ starts at time 𝑑

π‘˜ and finishes at time 𝑔 π‘˜

  • Two jobs are compatible if they don’t overlap
  • Goal: find maximum-size subset of mutually compatible jobs
slide-5
SLIDE 5

Interval Scheduling

373F19 - Karan Singh 5

  • Greedy template
  • Consider jobs in some β€œnatural” order
  • Take each job if it’s compatible with the ones already

chosen

  • What order?
  • Earliest start time: ascending order of 𝑑

π‘˜

  • Earliest finish time: ascending order of 𝑔

π‘˜

  • Shortest interval: ascending order of 𝑔

π‘˜ βˆ’ 𝑑 π‘˜

  • Fewest conflicts: ascending order of 𝑑

π‘˜, where 𝑑 π‘˜ is the

number of remaining jobs that conflict with π‘˜

slide-6
SLIDE 6

Example

373F19 - Karan Singh 6

  • Earliest start time: ascending order of 𝑑

π‘˜

  • Earliest finish time: ascending order of 𝑔

π‘˜

  • Shortest interval: ascending order of 𝑔

π‘˜ βˆ’ 𝑑 π‘˜

  • Fewest conflicts: ascending order of 𝑑

π‘˜, where 𝑑 π‘˜ is the number of

remaining jobs that conflict with π‘˜

slide-7
SLIDE 7

Interval Scheduling

373F19 - Karan Singh 7

  • Does it work?

earliest start time Counterexamples for shortest interval fewest conflicts

slide-8
SLIDE 8

Interval Scheduling

373F19 - Karan Singh 8

  • Implementing greedy with earliest finish time (EFT)
  • Sort jobs by finish time. Say 𝑔

1 ≀ 𝑔 2 ≀ β‹― ≀ 𝑔 π‘œ

  • When deciding whether job π‘˜ should be included, we

need to check whether it’s compatible with all previously added jobs

  • We only need to check if 𝑑

π‘˜ β‰₯ 𝑔 π‘—βˆ—, where π‘—βˆ— is the last added job

  • This is because for any jobs 𝑗 added before π‘—βˆ—, 𝑔

𝑗 ≀ 𝑔 π‘—βˆ—

  • So we can simply store and maintain the finish time of the last

added job

  • Running time: 𝑃 π‘œ log π‘œ
slide-9
SLIDE 9

Interval Scheduling

373F19 - Karan Singh 9

  • Optimality of greedy with EFT
  • Suppose for contradiction that greedy is not optimal
  • Say greedy selects jobs 𝑗1, 𝑗2, … , 𝑗𝑙 sorted by finish time
  • Consider the optimal solution π‘˜1, π‘˜2, … , π‘˜π‘› (also sorted by

finish time) which matches greedy for as long as possible

  • That is, we want π‘˜1 = 𝑗1, … , π‘˜π‘  = 𝑗𝑠 for greatest possible 𝑠
slide-10
SLIDE 10

Interval Scheduling

373F19 - Karan Singh 10

  • Optimality of greedy with EFT
  • Both 𝑗𝑠+1 and π‘˜π‘ +1 were compatible with the previous

selection (𝑗1 = π‘˜1, … , 𝑗𝑠 = π‘˜π‘ )

  • Consider the solution 𝑗1, 𝑗2, … , 𝑗𝑠, 𝑗𝑠+1, π‘˜π‘ +2, … , π‘˜π‘›
  • It should still be feasible (since 𝑔

𝑗𝑠+1 ≀ 𝑔 π‘˜π‘ +1)

  • It is still optimal
  • And it matches with greedy for one more step (contradiction!)

Another standard method is induction

slide-11
SLIDE 11

Interval Partitioning

373F19 - Karan Singh 11

  • Problem
  • Job π‘˜ starts at time 𝑑

π‘˜ and finishes at time 𝑔 π‘˜

  • Two jobs are compatible if they don’t overlap
  • Goal: group jobs into fewest partitions such that jobs in

the same partition are compatible

  • One idea
  • Find the maximum compatible set using the previous

greedy EFT algorithm, call it one partition, recurse on the remaining jobs.

  • Doesn’t work (check by yourselves)
slide-12
SLIDE 12

Interval Partitioning

373F19 - Karan Singh 12

  • Think of scheduling lectures for various courses

into as few classrooms as possible

  • This schedule uses 4 classrooms for scheduling 10

lectures

slide-13
SLIDE 13

Interval Partitioning

373F19 - Karan Singh 13

  • Think of scheduling lectures for various courses

into as few classrooms as possible

  • This schedule uses 3 classrooms for scheduling 10

lectures

slide-14
SLIDE 14

Interval Partitioning

373F19 - Karan Singh 14

  • Let’s go back to the greedy template!
  • Go through lectures in some β€œnatural” order
  • Assign each lecture to a compatible classroom (which?),

and create a new classroom if the lecture conflicts with every existing classroom

  • Order of lectures?
  • Earliest start time: ascending order of 𝑑

π‘˜

  • Earliest finish time: ascending order of 𝑔

π‘˜

  • Shortest interval: ascending order of 𝑔

π‘˜ βˆ’ 𝑑 π‘˜

  • Fewest conflicts: ascending order of 𝑑

π‘˜, where 𝑑 π‘˜ is the

number of remaining jobs that conflict with π‘˜

slide-15
SLIDE 15

Interval Partitioning

373F19 - Karan Singh 15

  • At least when you

assign each lecture to an arbitrary feasible classroom, three of these heuristics do not work.

  • The fourth one works!

(next slide)

slide-16
SLIDE 16

Interval Partitioning

373F19 - Karan Singh 16

slide-17
SLIDE 17

Interval Partitioning

373F19 - Karan Singh 17

  • Running time
  • Key step: check if the next lecture can be scheduled at

some classroom

  • Store classrooms in a priority queue
  • key = finish time of its last lecture
  • Is lecture π‘˜ compatible with some classroom?
  • Same as β€œIs 𝑑

π‘˜ at least as large as the minimum key?”

  • If yes: add lecture π‘˜ to classroom 𝑙 with minimum key, and

increase its key to 𝑔

π‘˜

  • Otherwise: create a new classroom, add lecture π‘˜, set key to 𝑔

π‘˜

  • 𝑃(π‘œ) priority queue operations, 𝑃(π‘œ log π‘œ) time
slide-18
SLIDE 18

Interval Partitioning

373F19 - Karan Singh 18

  • Proof of optimality (lower bound)
  • # classrooms needed β‰₯ maximum β€œdepth” at any point
  • depth = number of lectures running at that time
  • We now show that our greedy algorithm uses only these

many classrooms!

slide-19
SLIDE 19

Interval Partitioning

373F19 - Karan Singh 19

  • Proof of optimality (upper bound)
  • Let 𝑒 = # classrooms used by greedy
  • Classroom 𝑒 was opened because there was a schedule π‘˜

which was incompatible with some lectures already scheduled in each of 𝑒 βˆ’ 1 other classrooms

  • All these 𝑒 lectures end after 𝑑

π‘˜

  • Since we sorted by start time, they all start at/before 𝑑

π‘˜

  • So at time 𝑑

π‘˜, we have 𝑒 overlapping lectures

  • Hence, depth β‰₯ 𝑒
  • So all schedules use β‰₯ 𝑒 classrooms.
  • QED!
slide-20
SLIDE 20

Interval Graphs

373F19 - Karan Singh 20

  • Interval scheduling and interval partitioning can be

seen as graph problems

  • Input
  • Graph 𝐻 = (π‘Š, 𝐹)
  • Vertices π‘Š = jobs/lectures
  • Edge 𝑗, π‘˜ ∈ 𝐹 if jobs 𝑗 and π‘˜ are incompatible
  • Interval scheduling = maximum independent set

(MIS)

  • Interval partitioning = graph colouring
slide-21
SLIDE 21

Interval Graphs

373F19 - Karan Singh 21

  • MIS and graph colouring are NP-hard for general

graphs

  • But they’re efficiently solvable for interval graphs
  • Interval graphs = graphs which can be obtained from

incompatibility of intervals

  • In fact, this holds even when we are not given an interval

representation of the graph

  • Can we extend this result further?
  • Yes! Chordal graphs
  • Every cycle with 4 or more vertices has a chord
slide-22
SLIDE 22

Minimizing Lateness

373F19 - Karan Singh 22

  • Problem
  • We have a single machine
  • Each job π‘˜ requires π‘’π‘˜ units of time and is due by time π‘’π‘˜
  • If it’s scheduled to start at 𝑑

π‘˜, it will finish at 𝑔 π‘˜ = 𝑑 π‘˜ + π‘’π‘˜

  • Lateness: β„“π‘˜ = max 0, 𝑔

π‘˜ βˆ’ π‘’π‘˜

  • Goal: minimize the maximum lateness, 𝑀 = max

π‘˜

β„“π‘˜

  • Total lateness minimization is NP-complete
  • Contrast with interval scheduling
  • We can decide the start time
  • All jobs must be scheduled on a single machine
slide-23
SLIDE 23

Minimizing Lateness

373F19 - Karan Singh 23

  • Example

Input An example schedule

slide-24
SLIDE 24

Minimizing Lateness

373F19 - Karan Singh 24

  • Let’s go back to greedy template
  • Consider jobs one-by-one in some β€œnatural” order
  • Schedule jobs in this order (nothing special to do here,

since we have to schedule all jobs and there is only one machine available)

  • Natural orders?
  • Shortest processing time first: ascending order of

processing time π‘’π‘˜

  • Earliest deadline first: ascending order of due time π‘’π‘˜
  • Smallest slack first: ascending order of π‘’π‘˜ βˆ’ π‘’π‘˜
slide-25
SLIDE 25

Minimizing Lateness

373F19 - Karan Singh 25

  • Counterexamples
  • Shortest processing time first
  • Ascending order of processing time π‘’π‘˜
  • Smallest slack first
  • Ascending order of π‘’π‘˜ βˆ’ π‘’π‘˜
slide-26
SLIDE 26

Minimizing Lateness

373F19 - Karan Singh 26

  • By now, you

should know what’s coming…

  • We’ll prove

that earliest deadline first works!

slide-27
SLIDE 27

Minimizing Lateness

373F19 - Karan Singh 27

  • Observation 1
  • There is an optimal schedule with no idle time
slide-28
SLIDE 28

Minimizing Lateness

373F19 - Karan Singh 28

  • Observation 2
  • Earliest deadline first has no idle time
  • Let us define an β€œinversion”
  • 𝑗, π‘˜ such that 𝑒𝑗 < π‘’π‘˜ but π‘˜ is scheduled before 𝑗
  • Observation 3
  • By definition, earliest deadline first has no inversions
  • Observation 4
  • If a schedule with no idle time has an inversion, it has a

pair of inverted jobs scheduled consecutively

slide-29
SLIDE 29

Minimizing Lateness

373F19 - Karan Singh 29

  • Claim
  • Swapping adjacently scheduled inverted jobs doesn’t

increase lateness but reduces #inversions by one

  • Proof
  • Let β„“ and β„“β€² denote lateness before/after swap
  • Clearly, ℓ𝑙 = ℓ𝑙

β€² for all 𝑙 β‰  𝑗, π‘˜

  • Also, clearly, ℓ𝑗

β€² ≀ ℓ𝑗

slide-30
SLIDE 30

Minimizing Lateness

373F19 - Karan Singh 30

  • Claim
  • Swapping adjacently scheduled inverted jobs doesn’t

increase lateness but reduces #inversions by one

  • Proof
  • β„“π‘˜

β€² = 𝑔 π‘˜ β€² βˆ’ π‘’π‘˜ = 𝑔 𝑗 βˆ’ π‘’π‘˜ ≀ 𝑔 𝑗 βˆ’ 𝑒𝑗 = ℓ𝑗

  • 𝑀′ = max ℓ𝑗

β€², β„“π‘˜ β€², max 𝑙≠𝑗,π‘˜ ℓ𝑙 β€²

≀ max ℓ𝑗, ℓ𝑗, max

𝑙≠𝑗,π‘˜ ℓ𝑙 ≀ 𝑀

slide-31
SLIDE 31

Minimizing Lateness

373F19 - Karan Singh 31

  • Proof of optimality of earliest deadline first
  • Suppose for contradiction that it’s not optimal
  • Consider an optimal schedule π‘‡βˆ— which has fewest inversions

among all optimal schedules

  • We can assume it has no idle time
  • If π‘‡βˆ— has zero inversions, it’s exactly earliest deadline first
  • So assume π‘‡βˆ— has at least one inversion
  • So it must have an adjacent inversion (𝑗, π‘˜)
  • But swapping these jobs doesn’t increase lateness (so new schedule

stays optimal) and reduces the number of inversions by 1

  • Contradiction given that π‘‡βˆ— has fewest inversions among all optimal

schedules.

  • QED!
slide-32
SLIDE 32

Lossless Compression

373F19 - Karan Singh 32

  • Problem
  • We have a document that is written using π‘œ distinct labels
  • NaΓ―ve encoding: represent each label using 𝑙 = log π‘œ bits
  • If the document has length 𝑛, this uses 𝑛 log π‘œ bits
  • Say for English documents with no punctuations etc, we

have π‘œ = 26, so we can use 5 bits.

  • 𝑏 = 00000
  • 𝑐 = 00001
  • 𝑑 = 00010
  • 𝑒 = 00011
  • …
slide-33
SLIDE 33

Lossless Compression

373F19 - Karan Singh 33

  • Is this optimal?
  • What if 𝑏, 𝑓, 𝑠, 𝑑 are much more frequent in the

document than 𝑦, π‘Ÿ, 𝑨?

  • Can we assign shorter codes to more frequent letters?
  • Say we assign…
  • 𝑏 = 0, 𝑐 = 1, 𝑑 = 01, …
  • See a problem?
  • What if we observe the encoding β€˜01’?
  • Is it β€˜ab’? Or is it β€˜c’?
slide-34
SLIDE 34

Lossless Compression

373F19 - Karan Singh 34

  • To avoid conflicts, we need prefix-free encoding
  • Map each label 𝑦 to a bit-string 𝑑(𝑦) such that for all

distinct labels 𝑦 and 𝑧, 𝑑(𝑦) is not a prefix of 𝑑 𝑧

  • Then it’s impossible to have a scenario like this

………………………..

  • So we can read left to right, find the first point where it

becomes a valid encoding, decode the label, and continue

𝑑(𝑦) 𝑑(𝑧)

slide-35
SLIDE 35

Lossless Compression

373F19 - Karan Singh 35

  • Formal problem
  • Given π‘œ symbols and their frequencies (π‘₯1, … , π‘₯π‘œ), find a

prefix-free encoding with lengths (β„“1, … , β„“π‘œ) assigned to the symbols which minimizes σ𝑗=1

π‘œ

π‘₯𝑗 β‹… ℓ𝑗

  • Note that σ𝑗=1

π‘œ

π‘₯𝑗 β‹… ℓ𝑗 is the length of the compressed document

  • Example
  • (π‘₯𝑏, π‘₯𝑐, π‘₯𝑑, π‘₯𝑒, π‘₯𝑓, π‘₯𝑔) = (42,20,5,10,11,12)
  • No need to remember the numbers 
slide-36
SLIDE 36

Lossless Compression

373F19 - Karan Singh 36

  • Observation: prefix-free encoding = tree

𝑏 β†’ 0, 𝑓 β†’ 100, 𝑔 β†’ 101, 𝑑 β†’ 1100, 𝑒 β†’ 1101, 𝑐 β†’ 111

slide-37
SLIDE 37

Lossless Compression

373F19 - Karan Singh 37

  • Huffman Coding
  • Build a priority queue by adding 𝑦, π‘₯𝑦 for each symbol 𝑦
  • While |queue|β‰₯ 2
  • Take the two symbols with the lowest weight (𝑦, π‘₯𝑦) and (𝑧, π‘₯𝑧)
  • Merge them into one symbol with weight π‘₯𝑦 + π‘₯𝑧
  • Let’s see this on the previous example
slide-38
SLIDE 38

Lossless Compression

373F19 - Karan Singh 38

slide-39
SLIDE 39

Lossless Compression

373F19 - Karan Singh 39

slide-40
SLIDE 40

Lossless Compression

373F19 - Karan Singh 40

slide-41
SLIDE 41

Lossless Compression

373F19 - Karan Singh 41

slide-42
SLIDE 42

Lossless Compression

373F19 - Karan Singh 42

slide-43
SLIDE 43

Lossless Compression

373F19 - Karan Singh 43

  • Final Outcome

𝑏 β†’ 0, 𝑓 β†’ 100, 𝑔 β†’ 101, 𝑑 β†’ 1100, 𝑒 β†’ 1101, 𝑐 β†’ 111

slide-44
SLIDE 44

Lossless Compression

373F19 - Karan Singh 44

  • Running time
  • 𝑃(π‘œ log π‘œ)
  • Can be made 𝑃(π‘œ) if the labels are given to you sorted by

their frequencies

  • Proof of optimality
  • Induction on the number of symbols π‘œ
  • Base case: For π‘œ = 2, there are only two possible

encodings, both are optimal, assign 1 bit to each symbol

  • Hypothesis: Assume it returns an optimal encoding with

π‘œ βˆ’ 1 symbols

slide-45
SLIDE 45

Lossless Compression

373F19 - Karan Singh 45

  • Proof of optimality
  • Consider the case of π‘œ symbols
  • Lemma 1: If π‘₯𝑦 < π‘₯𝑧, then ℓ𝑦 β‰₯ ℓ𝑧 in any optimal tree.
  • Proof sketch: Otherwise, swapping 𝑦 and 𝑧 would strictly reduce

the overall length (exercise!).

  • Lemma 2: There is an optimal tree π‘ˆ in which the two

least frequent symbols are siblings.

  • Proof sketch: First prove that they must have the same longest

length assigned to them. Then, if they’re not siblings, chop and rearrange the tree to make them siblings (exercise!).

  • Now, we can compare the tree 𝐼 produced by Huffman

vs such an optimal tree π‘ˆ

slide-46
SLIDE 46

Lossless Compression

373F19 - Karan Singh 46

  • Proof of optimality
  • Let 𝑦 and 𝑧 be the two least frequency symbols
  • In Huffman, we combine them in the first step into β€œxy”
  • Let 𝐼′ and π‘ˆβ€² be trees obtained from 𝐼 and π‘ˆ by treating

𝑦𝑧 as one symbol with frequency π‘₯𝑦 + π‘₯𝑧

  • Use induction hypothesis: π‘€π‘“π‘œπ‘•π‘’β„Ž 𝐼′ ≀ π‘€π‘“π‘œπ‘•π‘’β„Ž(π‘ˆβ€²)
  • π‘€π‘“π‘œπ‘•π‘’β„Ž 𝐼 = π‘€π‘“π‘œπ‘•π‘’β„Ž 𝐼′ + π‘₯𝑦 + π‘₯𝑧 β‹… 1
  • π‘€π‘“π‘œπ‘•π‘’β„Ž π‘ˆ = π‘€π‘“π‘œπ‘•π‘’β„Ž π‘ˆβ€² + π‘₯𝑦 + π‘₯𝑧 β‹… 1
  • QED!
slide-47
SLIDE 47

Other Greedy Algorithms

373F19 - Karan Singh 47

  • If you aren’t familiar with the following algorithms,

spend some time checking them out!

  • Dijkstra’s shortest path algorithm
  • Kruskal and Prim’s minimum spanning tree algorithms