Nonlinear Optimization Practical Advice for Non-linear Least Square - - PowerPoint PPT Presentation

Nonlinear Optimization

Practical Advice for Non-linear Least Square Problems Niclas Börlin

Department of Computing Science Umeå University niclas.borlin@cs.umu.se

December 10, 2007

c 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems

Function call structure

Problem independent Problem dependent 1 Calculate a starting approxima- tion x0. 2 Repeat for k = 0, . . . 3 Calculate residual F(xk) and Ja- cobian J(xk). 4 Calculate termination criteria. 5 Calculate search direction pk. 6 Calculate step length αk. 6 Compute F(xk + αkpk). 7 Calculate xk+1 = xk + αkpk.

c 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems

Toy problem: Circle fitting

Given a number of points ˜ pi = [˜ xi, ˜ yi]T, i = 1, 2, . . . , m, “find the circle that fits the points best in the least squares sense”. (cx, cy) r p1 ˜ p1 p2 ˜ p2 p3 ˜ p3 p4 ˜ p4

c 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems

Objective function

Step 1: Decide what to minimize. Minimize the squared sum of Euclidian distances between each mea- sured point ˜ pi and the closest point pi on the circle, i.e. f(x) = 1 2

m

• i=1

pi − ˜ pi2.

c 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems

Models and parameters

Step 2: Formulate a mathematical model of the object and determine the unknowns. Step 2a: Formulate a local model for each “term” of the least squares sum. A point (xi, yi) on a circle with center (cx, cy) and radius r is modelled as Gi(x) = xi yi

• =

cx cy

• + r

cos θi sin θi

• for some “phase angle” θi.

Step 2b: Formulate a global model describing a vector with all terms

• f the least squares sum.

G(x) =      G1(x) G2(x) . . . Gm(x)      .

c 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems

Models and parameters

Step 2c: Determine which parameters to put in the vector of unknowns, and in what order. x =         cx cy r θ1 . . . θm         , where cx, cy, r are the “global” parameters, corresponding to the circle we wish to find and the θi are “local” parameters, corresponding to one point each. Step 2d: Implement the model function. Step 2e: Verify the model and implementation by calculating G(x) for realistic values of x.

c 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems

The residual

Step 3a: Implement the residual F(x) as “model minus data”. F(x; d) = G(x)−d, where d =      ˜ p1 ˜ p2 . . . ˜ pm      contain the “measurements”. Step 3b: Reality check. Verify that F(x∗; G(x∗)) = 0.

c 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems

The Jacobian

Step 4a: Derive an analytical expression for the Jacobian. Use symbolic tools, e.g. Maple, if necessary. Step 4b: Implement the Jacobian. J(x; d) =        1 cos θ1 −r sin θ1 1 sin θ1 r cos θ1 . . . . . . . . . ... 1 cos θm −r sin θm 1 sin θm r cos θm        . Step 4c: Compare the implemented Jacobian with a numerical approximation. J(x; d) ≈

• F(x+ǫ1)−F(x−ǫ1)

2ǫ

. . .

F(x+ǫn)−F(x−ǫn) 2ǫ

• ,

ǫ1 =

• ǫ

. . . T , ǫn =

• . . .

ǫ T . Step 4d: Verify that J(x) has full rank.

c 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems

Convergence check on synthetic, error-free data

Step 5: Tests on perfect data. Use a realistic x∗ and generate “measurements” without any errors, i.e. d = G(x∗). Step 5a: Call the optimization method with x0 = x∗ and verify that it returns x∗ as the solution after maximum 1 iteration. Step 5b: Generate starting approximations x0 as pertubations of the true solution x∗. Verify convergence from a reasonable large region.

c 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems

Pertubation sensitivity of solution

Step 5: Tests on data with known errors. Step 5a: Use a realistic x∗ and generate “measurements” with an added measurement error, i.e. d = G(x∗) + ε, ε ∈ N(0, σ2). where σ2 is chosen to give errors of a reasonable size. Step 5b: Solve the optimization problem with x0 = x∗. Call the solution to the new problem ˆ x. Step 5c: Compare the solution of the perturbed problem ˆ x with the solution of the original problem x∗. Step 5d: Repeat steps 5a-5c and analyze the deviation of ˆ x from x∗. Step 5e: Repeat steps 5a-5d with x0 = x∗, i.e. perturb the starting approximation also.

c 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems

Starting approximation calculation

Step 6a: Construct a function for the starting approximation that (optimally) only relies on measurements.

cx0 = ˜ xi, cy0 = ˜ yi, r0 =

• (˜

xi − cx0)2 + (˜ yi − cy0)2, θ0i = tan−1 ˜ yi − cy0 ˜ xi − cx0

Step 6b: Check the quality of the starting approximating function on error-free data. Step 6b.1: Generate realistic measurements d without any errors. Step 6b.2: Use the function in Step 6a to determine x0 from d and solve the optimization problem. Call the solution ˆ x. Step 6b.3: Compare ˆ x with x∗. Step 6c: Repeat steps 6b.1 to 6b.3 on data with realistic errors.

c 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems

Sensitivity analysis

“Truth” “Real world” 1 Start with a “true” parameter vector x∗ = [cx, cy, r, θ1, . . . , θm]. 2 Calculate points pi on the circle. 3 Generate simulated measure- ments ˜ pi = pi + ǫi, ǫi ∈ N(0, σ2). 4 Construct a starting approxima- tion x0 from the measurements. 5 Solve the parameter estimation

• problem. Call the solution ˆ

x. 6a Study the deviation of ˆ x for re- peated simulations. Determine precision, repeatability. 6b Compare the true parameter vector x∗ with the estimated ˆ x. Determine accuracy.

c 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems