CS475/CM375 Lecture 8: Oct 6, 2011 Iterative Methods Reading: [Saad] - - PDF document

05/10/2011 1

CS475/CM375 Lecture 8: Oct 6, 2011

Iterative Methods Reading: [Saad] Chapt 4

CS475/CM375 (c) 2011 P. Poupart & J. Wan 1

SOR Iteration

• Successive Over Relaxation (SOR)
• Weighted average of

and

• 1

∑

• ∑
• SOR
• 1
• 1
• 1
• Select
• For a suitably chosen (>1), SOR can be much better

than GS

CS475/CM375 (c) 2011 P. Poupart & J. Wan 2

05/10/2011 2

Convergence Analysis

• Q1: Under what conditions does the iteration

converge?

• Q2: If the iteration converges, how fast is it?
• Def: is an eigenvalue and an eigenvector of if

( 0)

• Def: the spectral radius of , , is the largest

absolute value of the eigenvalues of

CS475/CM375 (c) 2011 P. Poupart & J. Wan 3

Convergence Analysis

• Theorem: the iterative method

is convergent for any and if and only if 1

• is called the iteration matrix
• is called the rate of convergence

CS475/CM375 (c) 2011 P. Poupart & J. Wan 4

05/10/2011 3

Minimization Formulation

• Assume is SPD. Consider the functional:

≡

• ∈
• Theorem: The solution of is equivalent to the

solution of the minimization problem: min

CS475/CM375 (c) 2011 P. Poupart & J. Wan 5

Minimization Formulation

• Proof: the minimizer satisfies 0 i.e.
• – Note
• ∑
• ∑
• – Hence
• CS475/CM375 (c) 2011 P. Poupart & J. Wan

6

05/10/2011 4

Minimization Formulation

• Since is convex, then local min = global min
• Picture:

CS475/CM375 (c) 2011 P. Poupart & J. Wan 7

Search Directions

• Idea: minimize along the direction 0
• Let current approximation
• Define
• Determine by min along

– i.e., min

CS475/CM375 (c) 2011 P. Poupart & J. Wan 8

05/10/2011 5

Search Directions

• Let ≡
• Hence 0

and

CS475/CM375 (c) 2011 P. Poupart & J. Wan 9

Search Directions

• Notes

1. is SPD ⟹ 0

• 2. What is the optimal search direction ?
• Picture:

CS475/CM375 (c) 2011 P. Poupart & J. Wan 10

05/10/2011 6

Steepest Descent

• Local optimal direction
• Consider
• Then 0

changes in at in the direction

• Idea: make ′0 as negative as possible by varying

CS475/CM375 (c) 2011 P. Poupart & J. Wan 11

Steepest Descent

• Assume
• 1. Then

′0 is max if steepest descent ′0 is min if steepest descent

• (where )

CS475/CM375 (c) 2011 P. Poupart & J. Wan 12

05/10/2011 7

Steepest Descent

• Steepest descent method:

step length

• The optimal

( /)

• Also

CS475/CM375 (c) 2011 P. Poupart & J. Wan 13

Steepest Descent

• Algorithm

Given , compute for 0,1,2, … end

CS475/CM375 (c) 2011 P. Poupart & J. Wan 14

05/10/2011 8

Steepest Descent

• Notes
• 1. Only 1 matrix‐vector product per iteration
• 2. “Nonlinear” iterative method:

i.e.,

• CS475/CM375 (c) 2011 P. Poupart & J. Wan

15