Constructing optimal designs on finite experimental domains using - - PowerPoint PPT Presentation

Constructing optimal designs on finite experimental domains using methods of mathematical programming

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a

Faculty of Mathematics, Physics and Informatics Comenius University, Bratislava

mODa8 Almagro, Spain 4th-8th June 2007

Overview of the talk

Very brief introduction to optimal design problem on a finite experimental domain. The formulation of the problem of c-optimality as a special linear programming problem. Specification of the simplex algorithm of linear programming for constructing c-optimal designs. Example: Optimal designs for estimating individual coefficients for Fourier regression on a partial circle. The formulation of the problem of optimal designs with respect to various other criteria as a special problem of semidefinite programming or the so-called maxdet programming.

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

The model and basic assumptions

The regression model for a single observation y: E(y) = f ′(x)β; x ∈ X f ... vector of regr. functions, f : X → Rm, lin. independent on X β ... vector of parameters, β ∈ Rm X = {x1, ..., xk} ... experimental domain The observations are homoscedastic (Var(y) ≡ σ2) and uncorrelated. As is usual, an (asymptotic) design is a probability ξ on X and its information matrix is M(ξ) =

• x∈X

ξ(x)f(x)f ′(x). A design ξ is an exact design of size n, iff it can be realized by n

• bservations, i.e., iff ξ(x) = nx/n for all x ∈ X and some nx ∈ N0.

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

Definition and properties of c-optimal designs

Let c ∈ Rm. Let Ξc be the set of all designs ξ under which is c′β estimable, i.e., such that c belongs to the range of M(ξ). If we perform n measurements according to an exact design ξ ∈ Ξc of size n, then the variance of the Gauss-Markov estimator of c′β is: Var(c′ ˆ βξ) = σ2n−1c′M−(ξ)c. Definition (c-optimality) A design ξ∗

c ∈ Ξc is said to be c-optimal iff it minimizes c′M−(ξ)c

among all designs ξ ∈ Ξc. The value c′M−(ξ∗

c )c is then called the

c-optimal variance and M(ξ∗

c ) is a c-optimal information matrix.

There exists a c-optimal design supported on m points or less. For some models and some choices of c, the c-optimal information matrix is not unique and it can be singular.

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

Special cases of c-optimality and its relations

Examples of specific choices of c: If c is the i-th unit vector (c = ei), then ξ∗

c is optimal for estimating

βi. If c = f(x), then ξ∗

c is optimal for estimating the mean value of

response in x. If c = (c1, ..., cm)′, where ci =

• A fi(x)dx then ξ∗

c is optimal for

estimating

• A f ′(x)βdx.

Relations of c-optimality to other criteria: The c-optimal variance for c = ei is used in the so-called standardized optimality criteria (Dette 1997). The c-optimal values for c ranging in the unit sphere have implications for the E-optimal value. For every Loewner isotonic criterion Φ, there exists an optimal design supported only on those design points x such that the singular design in x is c-optimal for c = f(x).

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

The Elfving set and the Elfving theorem

The Elfving set of the model (f, X) is E = conv {f(x1), ..., f(xk), −f(x1), ..., −f(xk)} , which is a compact polytope symmetric around 0m. Theorem (Elfving) The design ξ is c-optimal for the model (f, X) and 1/h2 is the c-optimum variance if and only if hc =

x∈X ǫ(x)f(x)ξ(x) ∈ ∂E for

some selection of signs ǫ(x) ∈ {−1, 1}. The Elfving theorem motivated an algorithm for c-optimality in L´

• pez-Fidalgo and Rodr´

ıguez-D´ ıaz (2004) requiring nonlinear

• ptimization routines and practical for m ≤ 4.

Our approach is based on the observation that 1/h2 is the

• ptimal variance iff h is the maximum possible scalar such that

hc is a convex combinations of 2k vectors ±f(xi), which is a linear programming problem.

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

Linear programming approach

Theorem Consider the linear programming (LP) problem: max

• h|
• F

−c 1′

2k

α h

• =

0m 1

• ,

α h

• ≥ 0
• ,

(1) where F = (f(x1), ..., f(xk), −f(x1), ..., −f(xk)). Then a design ξ is c-optimal for the model (f, X) and h−2 is the c-optimum variance if and only if ξ(xi) = αi + αi+k for a solution (α′, h)′ of the problem (1). Consequences: Theory of LP specifies to results about c-optimality on finite experimental domains (e.g., duality theorem of LP corresponds to the ”equivalence theorem” for c-optimality). Algorithms (such as the simplex method) for solving LP problems can be applied to construct c-optimal designs.

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

Simplex algorithm for c-optimal designs (SAC)

Input: The design points x1, ..., xk, the vector of regression functions f and a sequence L = (l1, ..., lm) of indices from {1, ..., k}, such that f(xl1), ..., f(xlm) are independent vectors.

• 1. Set F ← (f(x1), ..., f(xk), −f(x1), ..., −f(xk)), Bm+1 ← 2k + 1.
• 2. For all i = 1, ..., m do: If (F−1

L c)i ≥ 0 then Bi ← li else Bi ← li + k.

• 3. Set h ← (1T

mF−1 B c)−1, ˜

α ← hF−1

B c, and sj ← F−1 B F(j) for all j /

∈ B.

• 4. Set j∗ ← min
• argmaxj /

∈B1T msj

• . If 1T

msj∗ ≤ 1 go to Step 7.

• 5. Set i∗ ← min
• argmini∈{1,...,m},di>0
• ˜

αi di

• , where d = sj∗ − ˜

αrj∗.

• 6. Update Bi∗ ← j∗ and return to Step 3.
• 7. For all i = 1, ..., m do: If Bi ≤ k then x∗

i ← xBi else x∗ i ← xBi−k.

Output: The c-optimal design ξ assigning weights ˜ α1, ..., ˜ αm to design points x∗

1 , ..., x∗ m, and c-optimal variance h−2.

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

Simplex algorithm for c-optimal designs (SAC)

The simplex algorithm for c-optimality is an exchange algorithm where the ”pivot rules” for the entering and leaving variables correspond to rules for the choice of design points to be

• exchanged. At each step, the algorithm calculates optimal

weights on the actual set of independent support points (e.g., Pukelsheim and Torsney 1991); the optimal weights correspond to the values of ”basic variables”. The general mechanism of the simplex algorithm for c-optimality is analogous to the so-called Remez procedure published by Studden and Tsay (1976). However, the Remez procedure requires strong nonsingularity assumptions (for instance it cannot deal with singular c-optimal designs, which are very common). On the other hand, the simplex algorithm for c-optimality can be applied for any model and any vector c, was observed to converge in all test problems, and can be modified to a version with guaranteed convergence using an ”anticycling” pivot rule.

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

Example: Fourier regression on a partial circle

Fourier (trigonometric) regression of degree d: E(y) = β1 +

d

• k=1

β2k sin(kx) +

d

• k=1

β2k+1 cos(kx), x ∈ X = [−a, a] (2) If a < π we say that the experimental domain X is a ”partial circle”. Optimal designs for (2) on a partial circle were studied in a series of recent papers by Dette, Melas, Pepelyshev (D, E, and c-optimality). The paper Dette and Melas (2003) describes methods of constructing es-optimal designs, either by an explicit formula or by means of Taylor expansion for a ≤ Ld,s and a ≥ Ud,s, where Ld,s and Ud,s are some critical constants; for many d and s is Ld,s strictly less than Ud,s. Example: Numeric results obtained by SAC for the cubic (d = 3) model and ”all” values of a on a dense discretization of X.

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

Example: Fourier regression on a partial circle

Π

• 3

Π

• 2

5 Π

• 6

0.6881 Π Π

• 2

Π

• 2

Π

• 3

Π

• 2

5 Π

• 6

0.6881 Π Π

• 2

Π

• 2

s 1

Figure: Support points of e1-optimal designs (vertical axis) for the cubic trigonometric model on a partial circle with the half-length a of the experimental domain (horizontal axis).

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

Example: Fourier regression on a partial circle

Π

• 3

Π

• 2

5 Π

• 6

Π Π

• 2

Π

• 2

Π

• 3

Π

• 2

5 Π

• 6

Π

• 2

Π

• 2

s 7

Figure: Support points of e7-optimal designs (vertical axis) for the cubic trigonometric model on a partial circle with the half-length a of the experimental domain (horizontal axis).

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

Example: Fourier regression on a partial circle

Π

• 3

Π

• 2

2 Π

• 3

5 Π

• 6

Π a 0.1 0.2 0.3 0.4 time s Implemented SM SM for coptimality

Figure: Computation times (vertical axis) of SAC and a general LP routine applied to the problem of e7-optimal designs in the cubic trigonometric model depending on the half-length a of the experimental domain (horizontal axis).

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

Semidefinite and Maxdet programming problems

Let z ∈ Rk. Let F0, F1, ..., Fn and G0, G1, ..., Gn be given symmetric matrices, and let F, G be functions of x ∈ Rm defined by F(x) = F0 +

n

• i=1

xiFi, and G(x) = G0 +

n

• i=1

xiGi Semidefinite programming problem (SDP): min

x∈Rk

• zTx | F(x) is positive semidefinite
• Maxdet programming problem (MDP):

min

x∈Rk

• zTx + log det G−1(x)
• F(x) is positive semidefinite

G(x) is positive definite

• Methods for solving these problems: interior point algorithms (MDP:

Vandenberghe, Boyd, Wu 1997).

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

Examples of SDP and MDP problems

D, A, E-optimal design problems on a finite experimental domain can be formulated as SDP or MDP problems. (Vandenberghe, Boyd, Wu 1997). Other design problems? Yes. Many convex design problems on a finite experimental domain can be formulated as SDP or MDP problems, and thus be solved by general interior point algorithms. Standardized A- and M-optimal designs (Dette 1997): min

ξ∈Ξ m

• j=1

(M−1(ξ))jj (M−(ξ∗

j ))jj

, and min

ξ∈Ξ

max

j=1,...,m

• (M−1(ξ))jj

(M−(ξ∗

j ))jj

• ,

where ξ∗

j is the optimal design for estimating the individual

coefficient βj, j = 1, ..., m.

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

Examples of SDP and MDP problems

Optimal designs (D, E, A, c,...) with general linear constraints on the ”makeup” of design, e.g., for D-optimality: max

ξ∈Ξ {det(M(ξ)) | Aw(ξ) ≥ b}

where w(ξ) = (ξ(x1), ..., ξ(xk))T are the weights of design points, A is a given matrix of coefficients and b is a given vector. In particular we can impose bounds on weights of individual design points, or on the cost of the experiment (cf. Cook, Fedorov 1995). D-optimal design in the set of all designs with a lower bound on the minimum eigenvalue of the information matrix (or with a given lower bound on the E-efficiency): max

ξ∈Ξ {det(M(ξ)) | λmin(M(ξ)) ≥ λ0 }

Geometrically: Design that minimizes volume of the confidence ellipsoid for β with an upper bound for its diameter.

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming

Concluding opinions

There is a high potential for mutual enrichment of modern mathematical programming methods and design of experiments. From the point of view of DE, modern optimization methods (and the increasing power of computers) will allow us to focus more on meaningful statistical aspects and practical use of designs. ”Pure” optimality criteria will always be fundamental, but the corresponding optimal designs will probably serve more as benchmarks for designs tailored to specific practical requirements involving various constrains on efficiency and ”makeup” of designs.

Thank you for attention!

Radoslav Harman, Tom´ aˇ s Jur´ ık, M´ aria Trnovsk´ a Constructing optimal designs using methods of mathematical programming