Nonlinear Equations = 40 / How can we solve these equations? - PowerPoint PPT Presentation

Nonlinear Equations

𝑙 = 40 𝑂/𝑛 How can we solve these equations? • Spring force: 𝐺 = 𝑙 𝑦 𝜈 ! = 0.5 𝑂𝑡/𝑛 What is the displacement when 𝐺 = 2N? • Drag force: 𝐺 = 0.5 𝐷 ! 𝜍 𝐵 𝑤 " = 𝜈 ! 𝑤 " What is the velocity when 𝐺 = 20N ?

• Spring force: 𝑔 𝑦 = 𝑙 𝑦 − 𝐺 = 0 𝜈 ! = 0.5 𝑂𝑡/𝑛 • Drag force: 𝑔 𝑤 = 𝜈 ! 𝑤 " −𝐺 = 0 Find the root (zero) of the nonlinear equation 𝑔 𝑤 Nonlinear Equations in 1D Goal: Solve 𝑔 𝑦 = 0 for 𝑔: ℛ → ℛ Often called Root Finding

Bisection method Algorithm: 1.Take two points, 𝑏 and 𝑐 , on each side of the root such that 𝑔(𝑏) and 𝑔(𝑐) have opposite signs. 2.Calculate the midpoint 𝑛 = !"# $ 3. Evaluate 𝑔(𝑛) and use 𝑛 to replace either 𝑏 or 𝑐 , keeping the signs of the endpoints opposite.

Convergence • The bisection method does not estimate 𝑦 . , the approximation of the desired root 𝑦. It instead finds an interval smaller than a given tolerance that contains the root. • The length of the interval at iteration 𝑙 is /01 " ! . We can define this interval as the error at iteration 𝑙 𝑐 − 𝑏 |𝑓 "%& | |𝑓 "%& | 2 "%& lim |𝑓 " | = lim |𝑓 " | = lim = 0.5 𝑐 − 𝑏 "→$ "→$ "→$ 2 " • Linear convergence

Convergence An iterative method converges with rate 𝑠 if: ||𝑓 .=> || lim ||𝑓 . || ? = 𝐷, 0 < 𝐷 < ∞ .→< 𝑠 = 1: linear convergence 𝑠 > 1: superlinear convergence 𝑠 = 2: quadratic convergence Linear convergence gains a constant number of accurate digits each step (and 𝐷 < 1 matters! Quadratic convergence doubles the number of accurate digits in each step (however it only starts making sense once ||𝑓 . || is small (and 𝐷 does not matter much)

Example: Consider the nonlinear equation 𝑔 𝑦 = 0.5𝑦 " − 2 and solving f x = 0 using the Bisection Method. For each of the initial intervals below, how many iterations are required to ensure the root is accurate within 2 0@ ? A) [−10, −1.8] B) [−3, −2.1] C) [−4, 1.9]

Bisection Method - summary q The function must be continuous with a root in the interval 𝑏, 𝑐 q Requires only one function evaluations for each iteration! o The first iteration requires two function evaluations. q Given the initial internal [𝑏, 𝑐] , the length of the interval after 𝑙 iterations is /01 " ! q Has linear convergence

Newton’s method • Recall we want to solve 𝑔 𝑦 = 0 for 𝑔: ℛ → ℛ • The Taylor expansion: 𝑔 𝑦 . + ℎ ≈ 𝑔 𝑦 . + 𝑔′ 𝑦 . ℎ gives a linear approximation for the nonlinear function 𝑔 near 𝑦 . . 𝑔 𝑦 . + ℎ = 0 → ℎ = −𝑔 𝑦 . /𝑔′ 𝑦 . • Algorithm: 𝑦 A = 𝑡𝑢𝑏𝑠𝑢𝑗𝑜𝑕 𝑕𝑣𝑓𝑡𝑡 𝑦 .=> = 𝑦 . − 𝑔 𝑦 . /𝑔′ 𝑦 .

Newton’s method Equation of the tangent line: 𝑔′(𝑦 . ) = 𝑔 𝑦 . − 0 𝑦 . − 𝑦 .=> 𝑦 "%& 𝑦 "

Iclicker question Consider solving the nonlinear equation 5 = 2.0 𝑓 C + 𝑦 " What is the result of applying one iteration of Newton’s method for solving nonlinear equations with initial starting guess 𝑦 A = 0, i.e. what is 𝑦 > ? A) −2 B) 0.75 C) −1.5 D) 1.5 E) 3.0

Newton’s Method - summary q Must be started with initial guess close enough to root (convergence is only local). Otherwise it may not converge at all. q Requires function and first derivative evaluation at each iteration (think about two function evaluations) q What can we do when the derivative evaluation is too costly (or difficult to evaluate)? q Typically has quadratic convergence ||𝑓 .=> || lim ||𝑓 . || " = 𝐷, 0 < 𝐷 < ∞ .→<

Secant method Also derived from Taylor expansion, but instead of using 𝑔′ 𝑦 . , it approximates the tangent with the secant line: 𝑦 .=> = 𝑦 . − 𝑔 𝑦 . /𝑔′ 𝑦 . Secant line: 𝑔′(𝑦 . ) ≈ 𝑔 𝑦 . − 𝑔 𝑦 .0> 𝑦 . − 𝑦 .0> • Algorithm: 𝑦 ! , 𝑦 " = 𝑡𝑢𝑏𝑠𝑢𝑗𝑜𝑕 𝑕𝑣𝑓𝑡𝑡𝑓𝑡 𝑔 # 𝑦 $ = 𝑔 𝑦 $ − 𝑔 𝑦 $%" 𝑦 $ − 𝑦 $%" 𝑦 $&" = 𝑦 $ − 𝑔 𝑦 $ /𝑔′ 𝑦 $ 𝑦 "%& 𝑦 "'& 𝑦 "

Secant Method - summary q Still local convergence q Requires only one function evaluation per iteration (only the first iteration requires two function evaluations) q Needs two starting guesses q Has slower convergence than Newton’s Method – superlinear convergence ||𝑓 .=> || lim ||𝑓 . || ? = 𝐷, 1 < 𝑠 < 2 .→<

1D methods for root finding: Method Update Convergence Cost Check signs of 𝑔 𝑏 and Linear (𝑠 = 1 and c = 0.5) Bisection One function evaluation per 𝑔 𝑐 iteration, no need to compute derivatives 𝑢 ! = |𝑐 − 𝑏| 2 ! Superlinear 𝑠 = 1.618 , Secant 𝑦 !"# = 𝑦 ! + ℎ One function evaluation per local convergence properties, iteration (two evaluations for ℎ = −𝑔 𝑦 ! /𝑒𝑔𝑏 convergence depends on the the initial guesses only), no initial guess need to compute derivatives 𝑒𝑔𝑏 = 𝑔 𝑦 ! − 𝑔 𝑦 !$# 𝑦 ! − 𝑦 !$# Quadratic 𝑠 = 2 , local 𝑦 !"# = 𝑦 ! + ℎ Newton Two function evaluations per convergence properties, iteration, requires first order ℎ = −𝑔 𝑦 ! /𝑔′ 𝑦 ! convergence depends on the derivatives initial guess

Nonlinear system of equations

Robotic arms https://www.youtube.com/watch?v=NRgNDlVtmz0 (Robotic arm 1) https://www.youtube.com/watch?v=9DqRkLQ5Sv8 (Robotic arm 2) https://www.youtube.com/watch?v=DZ_ocmY8xEI (Blender)

Nonlinear system of equations Goal: Solve 𝒈 𝒚 = 𝟏 for 𝒈: ℛ D → ℛ D In other words, 𝒈 𝒚 is a vector-valued function 𝑔 > 𝒚 𝑔 > 𝑦 > , 𝑦 " , 𝑦 E , … , 𝑦 D 𝒈 𝒚 = ⋮ = ⋮ 𝑔 D 𝒚 𝑔 D 𝑦 > , 𝑦 " , 𝑦 E , … , 𝑦 D If looking for a solution to 𝒈 𝒚 = 𝒛 , then instead solve 𝒈 𝒚 = 𝒈 𝒚 − 𝒛 = 𝟏

Newton’s method Approximate the nonlinear function 𝒈 𝒚 by a linear function using Taylor expansion: 𝒈 𝒚 + 𝒕 ≈ 𝒈 𝒚 + 𝑲 𝒚 𝒕 where 𝑲 𝒚 is the Jacobian matrix of the function 𝒈 : FG " 𝒚 FG " 𝒚 … FC " FC # IJ = FG $ 𝒚 𝑲 𝒚 = or 𝑲 𝒚 ⋮ ⋱ ⋮ FC % FG # 𝒚 FG # 𝒚 … FC " FC # Set 𝒈 𝒚 + 𝒕 = 𝟏 ⟹ 𝑲 𝒚 𝒕 = −𝒈 𝒚 This is a linear system of equations (solve for 𝒕 )!

Newton’s method Algorithm: 𝒚 A = 𝑗𝑜𝑗𝑢𝑗𝑏𝑚 𝑕𝑣𝑓𝑡𝑡 Solve 𝑲 𝒚 . 𝒕 . = −𝒈 𝒚 . Update 𝒚 .=> = 𝒚 . + 𝒕 . Convergence: • Typically has quadratic convergence • Drawback: Still only locally convergent Cost: • Main cost associated with computing the Jacobian matrix and solving the Newton step.

Newton’s method - summary q Typically quadratic convergence (local convergence) q Computing the Jacobian matrix requires the equivalent of 𝑜 " function evaluations for a dense problem (where every function of 𝒈 𝒚 depends on every component of 𝒚 ). q Computation of the Jacobian may be cheaper if the matrix is sparse. q The cost of calculating the step 𝒕 is 𝑃 𝑜 E for a dense Jacobian matrix (Factorization + Solve) q If the same Jacobian matrix 𝑲 𝒚 . is reused for several consecutive iterations, the convergence rate will suffer accordingly (trade-off between cost per iteration and number of iterations needed for convergence)

Example Consider solving the nonlinear system of equations 2 = 2𝑧 + 𝑦 4 = 𝑦 " + 4𝑧 " What is the result of applying one iteration of Newton’s method with the following initial guess? 𝒚 A = 1 0

Finite Difference Find an approximate for the Jacobian matrix: %& % 𝒚 %& % 𝒚 … %( % %( & )* = %& ' 𝒚 𝑲 𝒚 = or 𝑲 𝒚 ⋮ ⋱ ⋮ %( ( %& & 𝒚 %& & 𝒚 … %( % %( & In 1D: 𝜖𝑔 𝑦 ≈ 𝑔 𝑦 + ℎ − 𝑔 𝑦 𝜖𝑦 ℎ In ND: G $ 𝒚=K 𝜺 % 0G $ 𝒚 IJ = FG $ 𝒚 𝑲 𝒚 FC % ≈ K