Systems of Nonlinear Equations CS3220 - Summer 2008 Jonathan - - PowerPoint PPT Presentation

Systems of Nonlinear Equations

CS3220 - Summer 2008 Jonathan Kaldor

From 1 to N

• So far, we have considered finding roots of

scalar equations f(x) = 0

• f(x) takes a single scalar argument,

computes a single scalar

• Much like for linear systems, we would like

to extend this to systems of n equations in n variables

From 1 to N

• We can write this as

f1(x1, x2, ... xn) = 0 f2(x1, x2, ... xn) = 0 ... fn(x1, x2, ... xn) = 0

• Or, better yet, f(x) = 0
• f(x) takes a vector of arguments, produces

a vector

Applications

• Physical Simulation
• Planetary Dynamics
• Structural Analysis
• Differential Equations

From 1 to N

• Just like removing the assumption of linearity

complicated things, so to does the addition

• f multiple equations in multiple variables
• In particular, we lose the ability to easily

bracket possible roots

Bracketing Roots (or not)

• Consider 2 variable, 2 equation case. What

does it mean to bracket a root? How can we refine our intervals based on evaluating the function at grid points?

• (chalkboard)

Bracketing Roots (or not)

• What this means is that we’ve lost our easy

and guaranteed convergence method

• That’s not to say that there are no methods

that guarantee global convergence, though

• Beyond scope of this course
• In general, more complicated than the

simple bisection we studied

Review of Vector Calculus

• Let f(x) be a function from Rn → Rn
• Define ∂fi/∂xj as the partial derivative of the

i-th function with respect to the j-th variable

• Derivative of the function with respect to

the j-th variable while holding all other variables constant

Review of Vector Calculus

• Let g(x, y) = x2y + x sin(y)
• Then ∂g/∂x = 2xy + sin(y)

∂g/∂y = x2 + x cos(y)

Review of Vector Calculus

• We can then define the jacobian (first

derivative) of the function by the matrix Jij = ∂fi/∂xj

• Note: n x n matrix when f(x) is system of n

equations in n variables

• Also: Jacobian is a function of x: Jf(x)

Review of Vector Calculus

• Compute the Jacobian of

f1(x, y, z) = x2 + y2 + z2 - 10 f2(x, y, z) = x sin(z) + y cos(z) f3(x, y, z) = ez

Newton’s Method Redux

• Newton’s Method is one of the single

variable methods that has an extension to n variables and n equations.

• Relies on linear approximation of our

function

• Recall for g(x), we can use the Taylor Series

to approximate the function as: g(x0+h) = g(x0) + hg’(x0) + h2 g’’(x0) + ...

Newton’s Method Redux

• The Taylor series has an extension to n

equations in n variables: f(x + s) = f(x) + Jf(x) s + ...

• Quickly gets complicated...
• ... but fortunately, we only need the linear

approximation

Newton’s Method Redux

• Start at a point x0
• Take the linear approximation of f() at x0:

f(x + s) ≈ f(x0) + Jf(x0) s

• Solve for root of linear approximation:

Jf(x0) s = -f(x0)

• Note: n x n analog of division is matrix

inversion

Newton’s Method Redux

• In order to find our next iterate, we need to

solve the linear system Jf(x0) s = -f(x0)

• Newton in one variable replaces a nonlinear

equation with a linear equation. In many variables, this simply replaces a system of nonlinear equations with a system of linear equations.

• This is often referred to as “linearizing” the

system

Newton’s Method Redux

• Of course, this is most likely not our root,

so we need to iterate: x1 = initial guess for k = 1, 2, ... Solve Jf(xk) sk = -f(xk) for sk xk+1 = xk + sk end

Newton’s Method Redux

• Note: this is why we limit ourselves to n

equations in n variables, so that we can compute the inverse of the Jacobian

• This also depends on our Jacobian being

invertible at each of our (n-dimensional analogue of our derivative being non-zero)

• Don’t need to actually compute inverse of

Jacobian (of course)

Newton’s Method Redux

• Example of Newton’s Method

Newton’s Method Redux

• Convergence is quadratic like the “other”

Newton’s method, under similar assumptions

• Namely, our initial guess should be “relatively

close” to the root

• Note that as n increases, harder and

harder to “guess” a good initial point without additional information about prob.

Newton’s Method Redux

• Newton’s Method is quite expensive per

iteration:

• Each iteration involves computing an n x n

Jacobian matrix

• And then solving a linear system with that

matrix: O(n3) flops

Newton’s Method Redux

• As a result, sometimes common to do a

limited number of Newton steps and tolerate the error

• Oftentimes, only one iteration

Newton’s Method Redux

• Minimizing the amount of work per iteration

has led to alternate methods, called quasi- Newton methods

• Simplest example: compute Jacobian matrix

at xk, use it for more than one iteration without recomputing

• Slows convergence, since we are not

always using the correct derivative

Slightly Safer Strategies

• We can also try and improve the quality of
• ur steps by using damped Newton steps:

xk+1 = xk + αk sk

• Choose αk intelligently so that we dont take

any large or dangerous steps (usually by choosing αk < 1). As we get close to the answer, choose larger αk for quick convergence

Slightly Safer Strategies

• Note: this still relies on picking a good αk

value

• We can try and choose αk so that

‖f(xk+1)‖ < ‖f(xk)‖

• We are slowly creeping towards the

answer

• Still no absolute guarantees it will

converge