Numer. Math. 34, 247-270 (1980) Numerische Mathematik 9 by Springer-Verlag 1980 Global Optimization Using Interval Analysis- The Multi-Dimensional Case Eldon Hansen Lockheed Missiles and Space Company Sunnyvale, CA 94086, USA Summary. We show how interval analysis can be used to compute the global minimum of a twice -continuously differentiable function of n variables over an n-dimensional parallelopiped with sides parallel to the coordinate axes. Our method provides infallible bounds on both the globally minimum value of the function and the point(s) at which the minimum occurs. Subject Classification: AMS(MOS): 65K05, 90C30. 1. Introduction Consider the function f(x) in C 2 of n variables x I , x,. We shall describe a method for computing the minimum value f* off(x) over a box X (~ A box is defined to be a closed rectangular parallelopiped with sides parallel to the coordinate axes. We assume the number of points in X ~~ at which f(x) is globally minimum is finite. Our method provides infallible bounds on f* and on the point(s) x* for which f(x*)=f*. That is, our algorithm produces bounds on x* and f* which are always correct despite the presence of rounding errors. How sharp these bounds can be depends on the function f and the precision of the computer used. For a highly oscillatory function f, our algorithm could be prohibitively slow. Presumably this wilt always be the case for any future global optimization algorithm. However, our algorithm is sufficiently fast for 'reasonable' functions. We assume that interval extensions (see [8]) of f and its derivatives are known. This is the case if every function in terms of which f and its derivatives are defined have known rational approximations with either uniform or rational error bounds for the arguments of interest. Since the initial box can be chosen as large as we please, our algorithm actually solves the unconstrained minimization problem provided it is known that the solution occurs in some finite region (which we enclose in the initial box). 0029-599 X/80/0034/0247/$04.80 248 E. Hansen There is a common misconception among researchers in optimization that it is impossible to obtain infallible bounds on x* and f* computationally. The argument is that we can only sample f(x) and a few derivatives off(x) at a finite number of points. It is possible to interpolate a function having the necessary values and derivatives values at these points and still have its global minimum at any other arbitrary point. The fallacy of this argument is that interval analysis can provide bounds on a function over an entire box; that is over a continuum of points. It is only necessary to make the box sufficiently small in order to make the bounds arbitrarily sharp. This is what our algorithm does. It narrows the region of interest until the bound is as sharp as desired (subject to roundoff restrictions). In a previous paper [5], we gave a method of this type for the one- dimensional case. The method never failed to converge provided f'(x) and f"(x) had only a finite number of isolated zeros. Our method for the n-dimensional problem appears to always converge also; but we have not yet attempted to prove it. When it does converge, there is never a question that x* and f* satisfy the computed bounds. Recently, R.E. Moore [9] published a method for computing the range of a rational function of n variables over a bounded region. (See also [14].) Although he does not note the fact, his method will serve to bound the global minimum value f* of a rational function. However, our algorithm is more efficient. Moreover, it is designed to bound x* as well as f*. We suggest the reader read the previous paper [5] before the current one. The one-dimensional case therein serves as an easier introduction. However, the current paper is essentially self contained. It would be better if the reader had some familiarity with the rudiments of interval analysis such as can be found in the first three chapters of [8]. However, we shall review some of its relevant properties. Our method will find the global minimum (or minima). Because of computer limitations of accuracy, it may also find near-global minima such that rounding errors prevent determination of which is the true minimum. However, if the termination criteria are sufficiently stringent, our algorithm will always elim- inate a local minimum whose value is substantially larger than f*. Our algorithm is composed of four separate parts. One part uses an interval version of Newton's method to find stationary points. A second part eliminates points of X t~ where f is greater than the smallest currently known value J~ A third part of our algorithm tests whether f is monotonic in a sub-box X of X (~ If so, we delete part or all of X depending on whether X contains boundary points of X~~ A fourth part checks for convexity of f in a sub-box X of X t~ If f is not convex .anywhere in X, there cannot be a stationary minimum off in X. The first part of the algorithm, if used alone, would find all stationary points in X(~ The second part serves to eliminate stationary points where f>f*. Usually they are eliminated before they are found with any great accuracy. Hence computational effort is not wasted using the first part to accurately find an unwanted stationary point. The second part also serves to eliminate bound- ary points of X ~~ and to find a global minimum if it occurs on the boundary. Global Optimization Using Interval Analysis 249 The second part of the algorithm used alone would find the global minimum (or minima) but its asymptotic convergence is relatively slow compared to that of the Newton method. Hence the latter is used also. The third and fourth parts of the algorithm merely improve convergence. 2. Interval Analysis The toot which allows us to be certain we have bounded the global minimum is interval analysis. We bound rounding errors by using interval arithmetic. More importantly, however, we use interval analysis to bound the range of a function over a box. Let g(x) be a rational function of n variables x 1 , x,. On a computer, we can evaluate g(x) for a given x by performing a sequence of arithmetic operations involving only addition, subtraction, multiplication, and division. Let X~ (i = 1 , n) be closed intervals. If we use X~ in place of x~ and perform the same sequence of operations using interval arithmetic (see [8]) rather than ordinary real arithmetic, we obtain a closed interval g(X) containing the range {g(x): xieXi(i = 1 n)} of g(x) over the box X. This result will not be sharp, in general, but if outward rounding (see [8]) is used, then g(X) will always contain the range. The lack of sharpness results from other causes besides roundoff. With exact interval arith- metic, the lack of sharpness diappears as the widths of the intervals decrease to zero. If g(x) is not rational, we assume an algorithm is known for computing an interval g(X) containing the range of g(x) for x~X. Methods for deriving such algorithms are discussed in [8]). 3. Taylor's Theorem We shall use interval analysis in conjunction with Taylor's theorem in two ways. First, we expand f as f(y) =f(x) + (y - x)r g (x) + 89 (y - x)r H (x, y, 4)(Y - x) (3.1) where g(x) is the gradient off(x) and has components gi(x)=Of(x)/Ox~. The quantity H(x, y, ~) is the Hessian matrix to be defined presently. For reasons related to the use of interval analysis, we shall express it as a lower triangular matrix instead of a symmetric matrix so that there are fewer terms in the quadratic form involving H(x, y, 4). We define the element in position (i,j) of H(x, y, ~) as [02f/Ox~ for j=i(i=l, ,n), hq={2O2f/dx~Oxj for j<i(i=l, ,n;j=l ,i-l), (3.2) [0 otherwise. 250 E. Hansen The arguments of hij depend on i and j. If we expand f sequentially in one of its variables at a time, we can obtain the following results illustrating the case n = 3 l -hi 1(~11, x2, x3) 0 0 ] H(x,y, ~)= hzl(~2x ,x2,x3) hz2(Y1,~22, x3) 0 . [h31(~31, x2, x3) h3z(Yl, ~32, x3) h33(Yl, Y2, ~33) Assume xieX i and y~eX~ for i= 1 , n. Then ~ij~Xj for each j= 1 , i. For general n, the arguments of H~j are (Yx, , Yj- 1, ~j, xj+ 1 , x,). Other arrange- ments of arguments could be obtained by reordering the indices. Let x be a fixed point in X. Then for any point yeX, H(x, y, ~)~H(x, X, X); that is, for i>j, hiJ(Yl , Y i- 1, ~i.i, x2+ 1 x,,)Ehl.i(X1 X i , x j+ 1 x,,). In the sequel, we shall shorten notation and use H(~) to denote H(x, y, ~) and H(X) to denote H(x, X, X). The purpose of this particular Taylor expansion is to obtain real (non- interval) quantities for as many arguments of the elements of H(X) as possible. The standard Taylor expansion would have intervals for all arguments of all elements of H(X). This type of expansion was introduced in [3]. A more general approach of this kind is discussed in I-4]. The other Taylor expansion we shall want is of the gradient g. Each element gi(i= i, , n) of g can be expanded as gi(Y) gi(X) + (Yl x1) dil (t]l, x2, -", xn) + (Y2 - xz) Ji2(Y l, 1~ 2, x3 Xn) +(Y3 -x3) Ji3(Yl, Y2, rl3, x4 x,,)+ +(yn-xn) Jin(Yl , Y,- 1, rl,), where 0.3) Ji2 = c~2f/c3xi c3xj (i,j = 1 , n). This Jacobian matrix J and the Hessian H introduced above are, of course, essentially the same. However, they will be evaluated with different arguments depending on whether we are expanding f or g. Also, H is lower triangular while d is a full matrix. Let J(x, y, Yl) denote the Jacobian matrix with elements J~i(Yl , Y.i 1, ~lj, x j+ 1 , x,). Then g(y)= g(x) + J (x, y, rl) (y- x). (3.4) If xeX and yeX, then theX ~ for all i 1, , n. Hence g(y)eg(x) + J (x, X, X)(y- x). (3.5) We shall again shorten notation and denote J(x, y, tl) by J(t/) and J(x, X, X) by J(X). Global Optimization Using Interval Analysis 251 Note that the elements of H(X) on and below the diagonal have the same arguments as the corresponding elements of J(X). Thus we need only calculate J(X); then H(X) follows easily. 4. The Approximate Value of the Global Minimum As we proceed with our algorithm, we shall evaluate f(x) at various points in X (~ Let f denote the currently smallest value off found so far. The very first step is to evaluate f at the center of X ~~ This value serves as the first one for J~ One part of our algorithm deletes sub-boxes of X ~~ wherein f >f since this implies inff >f*. (See Sect. 7.) In practice we cannot generally evaluate f(x) exactly because of rounding errors. Hence we do the evaluation using interval arithmetic. Suppose we obtain the interval if L, fR]. Then we know that f(x)<=fR and hence that f_<fR. Hence when we evaluate f(x), we update f by replacing it by fR only iff R is less than the previous value off In this way, we assure that f is always an upper bound for f*. 5. A Test for Convexity As our algorithm proceeds, we dynamically subdivide X ~~ into sub-boxes. Let X denote such a sub-box. We evaluate hii(X ~ X,) for i= 1, , n, where hii is the diagonal element of the Hessian. Note that every argument of hi~ is an interval and hence the resulting interval contains the value of h~i(x ) for every xeX. That is, if [ui, v~] denotes the computed interval h~(X 1 , X,), then hii(x)E [[,l i, vii for all xeX. Suppose we find vi<0 for some value of i. Then h~i(x)<0 for every xeX. Hence there is no point in X at which the real (non-interval) Hessian is positive definite. Hence f is not convex and cannot have a minimum which is a stationary point in X. Hence we can delete all of X except for any boundary points of X ~~ which might lie in X. When we evaluate h~(Xt X,), we may find that the left endpoint ui>0 for all i= 1, , n. When this occurs, we know from inclusion monotonicity (see [8]) that we will find each u~>0 for any sub-box of X. Hence we could save some computational effort by noting when a box is a sub-box of one for which u i > 0 for all i = 1 n. We would skip this test for such a box. Note that an element h~ with arguments (X1 X,) is not obtained when we compute H(X) since the diagonal elements of H(X) have arguments different from (X~, , X,) except for the element in position (n, n). Hence our test for convexity requires recalculation of the diagonal of the Hessian. 6. The Interval Newton Method For each sub-box X of X (~ that our algorithm generates, we can apply an interval Newton method to the gradient g. Such methods seek the zeros of g and 252 E. Hansen hence the stationary points off. Such a method produces from X a new box or boxes N(X). Any points in X not in N(X) cannot contain a zero ofg and can be discarded unless they are boundary points of X (~ These methods, in effect, solve (3.5) for points y where g(y)=0. The first such method was derived by Moore [8]. Variants of Moore's method can be found in [3, 8, 12, 13]. The most efficient variant is described below. Krawczyk's method [8] is a suitable alternative to the method in [6]. Discussions of Krawczyk's method can be found in [10] and [11]. We now give a brief synopsis of our method. We wish to solve the set of equations g(x) + J(~)(y-x)- 0 (6.1) for the set of points y obtained by letting ~ range over X. We shall find a subset Y of X containing this set. Let Jc be the matrix whose element in position (i,j) is the midpoint of the corresponding interval element J~j(X) of the Jacobian J(X). Let B be an approximate inverse of Jc. As pointed out in [3], a useful first step in solving for Y is to multiply (6.1) by B giving Bg(x) + BJ(~.)(y- x) = 0. (6.2) Note that the product BJ(~) approximates the identity matrix. However it may be a very poor approximation when X is a large box. We 'solve' (6.2) by a process similar to a single sweep of the Gauss-Seidel method. Write BJ(X)=L+D+U where L, D, and U are the lower triangular, diagonal, and upper triangular part of BJ(X), respectively. The interval matrix D- 1 =diag [1/D 11, 1/D22 1/D,,] (6.3) contains the inverse of every matrix in D. The box Y'solving' (6.2) is obtained as Y = x ~ D- ~ [B g (x) + L(Y- x) + U (X - x)]. (6.4) When obtaining the component Y~ of Y, the components Y1, , Y~-1 appearing in the right member of this equation have already been obtained. This formulation presupposes that the intervals D~ (i=1 n) do not contain zero. When X is a small box, BJ(X) is closely approximated by the identity matrix and hence D is also. However, for X large, it is possible to have 0eD u for one or more values of i. This case is easily handled. We simply use an extended interval arithmetic which allows division by an interval containing zero. A detailed discussion of this new method will be published elsewhere. Note that we cannot allow the Newton procedure to delete boundary points of X ~~ since the global minimum need not be a stationary point if it occurs on the boundary. We discuss this point further in Sect. 10. Global Optimization Using Interval Analysis 253 If we were to use this Newton method only, we would in general find stationary points of f which were not minima. Moreover, we would find local minima which were not global minima. To avoid this, we use an additional procedure to delete points where f exceeds the smallest known value ~ This procedure is described in the next section. In some applications, it may be desirable to find all the stationary points off in a given box. This can be done using the Newton method alone or in conjunction with the monotonicity check of Sect. 9. If, in addition, the convexity check of Sect. 5 were used, all stationary points except maximum would be found. 7. Bounding f We now consider how to delete points y~X where we know f(y)>f and hence where f(y) is not a global minimum. We retain the complementary set which is a sub-box (or sub-boxes) Y c X wherein f(y) may be <j~ As pointed out in [5], if we only wish to bound f* and not x*, we can delete points where f(y)>f -e I (7.1) for some e 1 >0. We can allow e 1 to be nonzero only if we do not need to know the point(s) x* at which f is globally minimum. We want to retain points where (7.1) is not satisfied. From (3.1), this is the case for points y if f(x) + (y- x) T g(x) + 89 (y - x) r H({)(y - x) <f- el because the left member equals f(y). Denote E =f-f(x)-e 1. Then ~r g(x) + 89 H(~) ~ < E (7.2) where ~=y-x. We shall use this relation to reduce X in one dimension at a time to yield the sub-box(es) Y resulting from deleting points where f(y)>f-~1. We shall illustrate the process for the case n = 2. The higher dimensional case follows in the same way. For n = 2, (7.2) becomes ~191(x)d-~Y292(x)-~89 hll(~)~-~l~Y2h21(~)-F~Y2 h22(~)J~E. (7.3) We first wish to reduce X in the xl-direction. Thus we solve this relation for acceptable values of Yl. After collecting terms in y~, we replace Y2 by X 2. In the higher dimensional case we would also replace Yi by Xi for all i= 3 , n. We also replace ~ by X (since ~eX). We obtain ~1[gl(x)+89189 hll(X)+ X292(x)+89163 h22(X)-E~O (7.4) where X2=X2-x2. 254 E. Hansen We solve this quadratic for the interval or intervals of points Yl as described below. Call the resulting set Z s. Since we are only interested in points with yleX~, we compute the desired set Ys as I11 =Xa c~Z~. For the sake of argument, suppose II1 is a single interval. We can then try to reduce X 2 the same way we (hopefully) reduced X1 to get Ys. We again rewrite (7.3). This time we replace Yl by Y~ and (as before) ~ by X. We could obtain better results by replacing ~ by II1 rather than X 1 but this would require re- evaluation of the elements of H. We obtain .v2 [g2(x) +89 J~s h2 ~ (X)] +~Y2' -z h22(X) + f's gl (x) +89 f'? hls(X)-E<O= (7.5) where 91 = IIi -xl. If the solution set Y2 is strictly contained in X 2, we could replace X 2 by Y2 in (7.4) and solve for a new Y1- We have not tried to do this in practice. Instead, we start over with the box Y in place of X as soon as we have tried to reduce each X i to Y~ (i = 1 , n). Note this means we re-evaluate H(X). We now consider how to solve the quadratic equation (7.4) or (7.5). These have the general form A + Bt + Ct2 <O (7.6) where A, B, and C are intervals and we seek values of t satisfying this inequality. Denote C= [c 1, c2] and let c be an arbitrary point in C. Similarly, let asA and beB be arbitrary. Suppose t is such that (7.6) is violated; that is Q(t)>0, where Q(t)=a+bt +ct 2. If this is true for c=c~, then it is true for all c6C. Hence if we wish to find the complementary values of t where (7.6) might hold we need only consider A + Bt +c 1 t2~0. (7.7) If c~ =0, this relation is linear and the solution set T is as follows: Denote A = [as, a2] and B = [bl, b2]. Then the set of solution points t is T= [-al/b2, ~] if a 1 _<0, b2<0 , [-al/bs, ~] if as>O, bs <O, b2 <O, [-~, ~] if as<O, bl<O<b2, [-~,-aa/bz]w[-al/bs,~ ] if ax>O, bl<O<b2, [-~, -al/bl] if a s<0, b s>0, [ ~, -ax/b2] if a I >0, b s >0, b2>0 , empty set if a s >0, b 1 =b 2 =0. i. Thus although T may Recall that we will intersect T with X i for some value of be unbounded, the intersection is bounded. If c1:# 0, the quadratic (7.6) may have no solution or it may have a solution set T composed of either one or two intervals. In the latter case, the intervals may be semi-infinite. However, after intersecting T with Xi, the result is finite. Global Optimization Using Interval Analysis 255 Denote Ql(t)=a+bt +q t 2 where aeA, bEB, and c 1 is the left endpoint of C. We shall delete points t where Qa(t)>0 for all a~A and beB. Thus we retain a set T of points where Ql(t)=<0, as desired. But we also retain (in T) points where, for fixed t, Qa(t)>0 for some aeA and beB and Qa(t)<O for other aEA and beB. This same criterion was used to obtain T when c a =0. This assures that we shall always retain points in Xi where f(x) is a minimum. Denote qa(t)={aa+b2t+clt2 if t<0, a,+bat+cat 2 if t>0 and qe(t)={a2+btt+Cat2 if t=<0, a2+b2t+cit 2 if t>=0. Then we can write the interval quadratic as Q1 (t) = [aa, a2] q- [b I, be] t + C 1 t 2 = [q, (t), q2 (t)]. Thus for any finite t, qa(t) is a lower bound for Qa(t) and q2(t) is an upper bound for Ql(t) for any aeA and any beB. For a given value of t, if ql(t)>0, then 01(0>0 for all aeA and beB. Hence we need only to solve the real quadratic equation qa (t)=0 in order to determine intervals wherein, without question, Q1 (t)> 0. This is a straightforward problem. The function qa(t) is continuous but its derivative is discontinuous at the origin when b1=t = b 2 which will generally be the case in practice. Hence we must consider the cases t < 0 and t => 0 separately. If c a >0, the curve qa(t) is convex for t=<0 and convex for t=>0. Consider the case t=<0. If q~(t) has real roots, then Qa(t)>0 outside these roots, provided t <0. Hence, we retain the interval between these roots. We need only examine the discriminant of q~ (t) to determine whether the roots are real or not. Hence it is a simple procedure to determine which part (if any) of the half line t <= 0 can be deleted. The same procedure can be used for t >_0. For c a <0, qa(t) is concave for t__<0 and for t>=0. In this case we can delete the interval (if any) between the roots of q~(t) in each half line. The set T is the complement of this interval. It is composed of two semi-infinite intervals. In determining T for either the case c 1 <0 or in the case c I >0, it is necessary to know whether the discriminant of qa (t) is non-negative or not. Denote A a =b2-4al ca, A2=b2-4aa C 1. These are the discriminants of qa (t) when t >__ 0 and t _-< 0, respectively. When we compute A a or A2, we shall make rounding errors. Thus we should compute them using interval arithmetic to bound these errors. When computing Ai=(i= 1, 2), suppose we obtain the interval A[=[A],Aff] (i=1, 2). 256 E. Hansen We use the appropriate endpoint of A~ or A~2 to determine T which assures that we never delete a point t where Q~(t) could be non-positive. Thus we use the endpoint of A~ or A~ which yields the larger set T. When we compute the roots of ql(t), we shall make rounding errors. Hence we compute them using interval arithmetic and again use the endpoints which yield the larger set T to assure we do not delete a point in X~ where f is a minimum. For i= 1 and 2, denote and R + =(-b~ +_A1/2)/(2Cl) S + = 2a( - b i 4- A~/2). Note that R~ + = S/- and R i- = S~ +. As is well known, the rounding error is less if we compute a root in the form R~ + rather than in the form S 7 when b i <0. The converse is true when bi > 0. Similarly, the rounding error is less when using R~- rather than S~ + when bi>0. Hence we compute the roots of q~(t) as R~ + and S~ when bi<0 and as R~- and S~- when bi>0. Note that computing R~ or S~ involves taking the square root of the interval A[. In exact arithmetic this would bc the real quantity Ai. We would never be computing roots of q~(t) when A~ was negative. Hence if we find that the computed result A[ contains zero, we can replace it by its non-negative part. Thus we will never try to take the square root of an interval containing negative numbers. Given any interval 1, let I L and I R denote its left and right endpoint, respectively. We use this notation below. Using the above prescriptions on how to compute the set T, we obtain the following results: For b~ __>0 and c I >0, [0 (the empty set) T= ~[(R;) L, ($2) R] [[(R~)L, (Si-)R] For b2<0 and c1>0, T=[E(S~), (R~-) n] S+L [[(2),(R~) R] For b 1 <0=<b 2 and c 1 >0, r [(R2)L, (S~-)R] T= [(S~-) L, (R~-) R] [(R; )L, (S ~- )R] u [(S +)g, (R ~- )R] E(R y )L, (R ~-)R] if A2R <0, if al>0 and A~>0, if a~ <0. (7.8) if A~<0, if al>0 and A~>0, if a 1 <0. (7.9) if max(A~, A2g)<O, if [bl[<b 2 and min(A~, A2g)__<O =<max (A~, A2R), if [blI>b 2 and min(AIR, A2R)_ <O (7.10) ____max (A~, A2R), if al>O and min(A~,A2R)>O if a 1 <0. [...]... with Xi In the former case, Y~ = X~ c~Z~ can be empty or a single interval In the latter case, can be empty, a single interval, or two intervals We now consider the logistics of handling these cases The quadratic inequality to be solved for Z~ will have quadratic term 1"2hii(X ) so the interval C in (7.6) is 89 ) and the left endpoint is c]i) gY~ = [89 L If c(/)>0 the solution set is a single interval. .. components for which the left endpoint of Hii(X ) is non-positive Finally, complete the Newton step for those components with O~BJu(X ) At each stage of either method, when trying to improve the i-th component of the box, we use the currently best interval for the other components This may be the smallest interval containing two disjoint intervals in some cases In fact it would be possible for the quadratic... all the others by the smallest sub -interval containing the two disjoint parts We then divide the remaining part of the current box into two sub-boxes by deleting the sub -interval for the component in question We could do this for more than one component, but each deletion would double the number of boxes It seems better to keep the number of boxes small 8 Choice of el Suppose we want to bound the value... obtained using the algorithm described above The computations were done on the Amdahl 470V/6-II computer In each case, we assumed it was known that the global minimum occurred in the interior of the initial box This speeds up the algorithm since boundary points need not get special treatment This is consistent with the fact that we are really considering the unconstrained case We intend to treat the constrained... Towards global optimization Amsterdam: North Holland, 1975 2 Dixon L.C.W., Szeg6, G.P.: Towards global optimization 2 Amsterdam: North Holland, 1977 3 Hansen, Eldon: On solving systems of equations using interval arithmetic Math Comp 22, 374384 (1968) 4 Hansen, Eldon: Interval forms of Newton's method Computing 20, 153-163 (1978) 5 Hansen, Eldon: Global optimization using interval analysis - the one -dimension. .. that at some stage in our algorithm, the list L contains s boxes Denote these boxes by X (1), , X (s) Let X~.~ denote the interval defining thej-th component (dimension) of X ii') Let w(X~i); denote the width of the interval X~i) and let w i = max [w(X~~ 1 . 34, 24 7-2 70 (1980) Numerische Mathematik 9 by Springer-Verlag 1980 Global Optimization Using Interval Analysis- The Multi-Dimensional Case Eldon Hansen. In the latter case, the intervals may be semi-infinite. However, after intersecting T with Xi, the result is finite. Global Optimization Using Interval Analysis