[PPT] - Lecture 17/18: Dynamic Programming - Matrix Chain Parenthesization PowerPoint Presentation

SLIDE 1

Lecture 17/18: Dynamic Programming - Matrix Chain Parenthesization

COMS10007 - Algorithms

Dr. Christian Konrad

01.04.2019 and 02.04.2019

Dr. Christian Konrad

Lecture 17/18: Matrix Chain Parenthesization 1 / 18

SLIDE 2

Matrix Multiplication

Problem: Matrix-Multiplication

1 Input: Matrices A, B with A.columns = B.rows 2 Output: Matrix product A × B

Example:     2 3 1 2 6 9     × 1 2 2

=

    6 2 4 1 2 12 2 4 18     p p q r r q Notation: p × q matrix: p rows and q columns p × q matrix times q × r matrix gives a p × r matrix (A × B)i,j = row i of A times column j of B

Dr. Christian Konrad

Lecture 17/18: Matrix Chain Parenthesization 2 / 18

SLIDE 3

Algorithm for Matrix-Multiplication

Algorithm: (A × B)i,j = row i of A times column j of B Require: Matrices A, B with A.columns = B.rows Let C be a new A.rows × B.columns matrix for i ← 1 . . . A.rows do for j ← 1 . . . B.columns do Cij ← 0 for k ← 1 . . . A.columns do Cij ← Cij + Aik · Bkj return C Algorithm Matrix-Multiply(A, B) Runtime: Three nested loops: O(A.rows · B.columns · A.columns) Number of Multiplications: A.rows · B.columns · A.columns Multiplying two n × n matrices: runtime O(n3)

Dr. Christian Konrad

Lecture 17/18: Matrix Chain Parenthesization 3 / 18

SLIDE 4

Background: Faster Matrix Multiplication

History: Multiplying two n × n matrices before 1969: O(n3) 1969: Strassen O(n2.8074) (divide-and-conquer) 1990: Coppersmith-Winograd O(n2.3755) 2010: Stothers O(n2.374) 2011: Virginia Williams O(n2.3728642) 2014: Le Gall O(n2.3728639) Important Problem: Many algorithms rely on fast matrix multiplication Better bound for matrix multiplication improves many algorithms

Dr. Christian Konrad

Lecture 17/18: Matrix Chain Parenthesization 4 / 18

SLIDE 5

The Matrix-chain Multiplication Problem

Problem: Matrix-Chain-Multiplication

1 Input: A sequence (chain) of n matrices A1, A2, A3, . . . , An 2 Output: The product A1 × A2 × A3 × · · · × An

Discussion: Ai.columns = Ai+1.rows for every 1 ≤ i < n Assume Ai has dimension pi−1 × pi, for vector p[0 . . . n] Matrix product is associative: (A1 × A2) × A3 = A1 × (A2 × A3) Exploit Associativity: Parenthesize A1 × A2 × A3 × . . . An so as to minimize the number of scalar multiplications (and thus the runtime)

Dr. Christian Konrad

Lecture 17/18: Matrix Chain Parenthesization 5 / 18

SLIDE 6

Order matters

Example: Three matrices A1, A2, A3 with dimensions A1 : 10 × 100 A2 : 100 × 5 A3 : 5 × 50 (p0 = 10, p1 = 100, p2 = 5, p3 = 50) Computation of (A1 × A2) × A3: A1 × A2 = A12 requires 10 · 100 · 5 = 5000 multiplications A12 × A3 requires 10 · 5 · 50 = 2500 multiplications Total: 7500 multiplications Computation of A1 × (A2 × A3): A2 × A3 = A23 requires 100 · 5 · 50 = 25000 multiplications A1 × A23 requires 10 · 100 · 50 = 50000 multiplications Total: 75000 multiplications

Dr. Christian Konrad

Lecture 17/18: Matrix Chain Parenthesization 6 / 18

SLIDE 7

The Matrix-Chain-Parenthesization Problem

Problem: Matrix-Chain-Parenthesization

1 Input: A sequence (chain) of n matrices A1, A2, A3, . . . , An 2 Output: A parenthesization of A1 × A2 × A3 × · · · × An that

minimizes the number of scalar multiplications How many Parenthesizations P(n) are there? We write: Aij for the product Ai × Ai+1 × · · · × Aj There is a final matrix multiplication: A1k × A(k+1)n, for some 1 ≤ k ≤ n − 1. Hence: P(n) =

1

if n = 1 , n−1

k=1 P(k)P(n − k)

if n ≥ 2 . Example: Four matrices A1, A2, A3, A4 A1 × A24 A12 × A34 A13 × A4

Dr. Christian Konrad

Lecture 17/18: Matrix Chain Parenthesization 7 / 18

SLIDE 8

Number of Parenthesizations

Example (continued): Four matrices A1, A2, A3, A4 A1 × A24 A12 × A34 A13 × A4 P(3) =

2

k=1

P(k)P(n − k) = P(1)P(2) + P(2)P(1) = 2 P(4) =

3

k=1

P(k)P(n − k) = P(1)P(3) + P(2)P(2) + P(3)P(1) = P(3) + 1 + P(3) = 2P(3) + 1 = 5 .

1 A1 × ((A2 × A3) × A4) 2 A1 × ((A2 × (A3 × A4)) 3 (A1 × A2) × (A3 × A4) 4 ((A1 × A2) × A3) × A4 5 (A1 × (A2 × A3)) × A4

Dr. Christian Konrad

Lecture 17/18: Matrix Chain Parenthesization 8 / 18

SLIDE 9

Number of Parenthesizations (2)

A Bound on the Number of Parenthesizations: P(n) =

1

if n = 1 , n−1

k=1 P(k)P(n − k)

if n ≥ 2 . 1, 1, 2, 5, 14, 42, 132, 429, 1430, 4862, 16796, 58786, 208012, 742900, . . . It can be seen that there are Ω(2n) possibilities An efficient algorithm thus cannot try out all possibilities We will give a dynamic programming algorithm

Dr. Christian Konrad

Lecture 17/18: Matrix Chain Parenthesization 9 / 18

SLIDE 10

Optimal Substructure

Optimal Substructure We say that a problem P exhibits optimal substructure if: An optimal solution to P contains within it optimal solutions to subproblems of P. Optimal Substructure in Matrix-Chain-Parenthesization Consider optimal solution to instance of size n Suppose that last product is A1k × A(k+1)n Then the optimal solution contains optimal parenthesizations

f A1 × A2 × · · · × Ak and Ak+1 × Ak+2 × . . . An
Proof. Suppose it did not contain optimal parenthesizations
f A1 × A2 × · · · × Ak and of Ak+1 × Ak+2 × . . . An. Then

picking optimal parenthesizations of the two subproblems would give better solution to initial instance.

Dr. Christian Konrad

Lecture 17/18: Matrix Chain Parenthesization 10 / 18

SLIDE 11

Recursive Solution

Optimal Solution to Subproblem: m[i, j] : minimum number of scalar multiplications needed to compute Ai × Ai+1 × · · · × Aj = Aij Observe that m[i, i] = 0 (chain consists of single matrix Ai) Suppose j > i. Suppose last multiplication in optimal solution is: Aik × A(k+1)j, for some k Then: m[i, j] = m[i, k] + m[k + 1, j] + pi−1pkpj

cost of multiplying Aik × A(k+1)j

(Aik: pi−1 × pk matrix, A(k+1)j : pk × pj matrix) Since we do not know k, we try out all possibilities and choose the best solution: m[i, j] =

if i = j ,

mini≤k<j{m[i, k] + m[k + 1, j] + pi−1pkpj} if i < j .

Dr. Christian Konrad

Lecture 17/18: Matrix Chain Parenthesization 11 / 18

SLIDE 12

Computing the Optimal Costs

m[i, j] =

if i = j ,

mini≤k<j{m[i, k] + m[k + 1, j] + pi−1pkpj} if i < j . Algorithmic Considerations: As in Pole-Cutting, we could implement this recursive formula directly. → exponential runtime Instead, we compute the table m[i, j] bottom up Observe that there are less than n2 subproblems m[i, j] (i and j take values in {1, . . . , n}) We will see that computing one value m[i, j] takes O(n) time This yields an O(n3) time algorithm

Dr. Christian Konrad

Lecture 17/18: Matrix Chain Parenthesization 12 / 18

SLIDE 13

Dynamic Programming Algorithm

Require: Integer n, vector of dimensions of matrices p so that matrix Ai has dimensions pi−1 × pi Let m[1 . . . n, 1 . . . n] be a new array for i ← 1 . . . n do m[i, i] ← 0 for l ← 2 . . . n do {chain length} for i ← 1 . . . n − l + 1 do {left position} j ← i + l − 1 {right position} m[i, j] ← ∞ for k ← i . . . j − 1 do m[i, j] ← min{m[i, j], m[i, k] + m[k + 1, j] + pi−1pkpj} return m Algorithm Matrix-Chain-Value(n, p) Runtime: O(n3) (by evaluating n

l=2

n−l+1

i=1

i+l−2

k=1

O(1))

Dr. Christian Konrad

Lecture 17/18: Matrix Chain Parenthesization 13 / 18

SLIDE 14

Runtime Evaluation

Useful Formula:

b

i=a

1 = b − a + 1

n

l=2

n−l+1

i=1

i+l−2

k=1

O(1) = O(1) ·

n

l=2

n−l+1

i=1

i+l−2

k=1

1 ≤ O(1) ·

n

l=1

n

i=1

n

k=1

1 = O(1) ·

n

l=1

n

i=1

n = O(1) · n

n

l=1

n

i=1

1 = O(1) · n

n

l=1

n = O(1) · n2

n

l=1

1 = O(1) · n2 · n = O(1)n3 = O(n3) .

Dr. Christian Konrad

Lecture 17/18: Matrix Chain Parenthesization 14 / 18

SLIDE 15