Chapter Convex sets and convex functions taking the infinity value Chapter Convex sets and convex functions taking the infinity value tvnguyen (University of Science) Convex Optimization / 108 Chapter Convex sets and convex functions taking the infinity value Convex set Definition A subset C of IRn is convex if ∀x, y ∈ C Proposition are convex ∀t ∈ [0, 1] tx + (1 − t)y ∈ C If C is convex, then its interior int C and its closure C Convexity is preserved by the following operations : Let I be an arbitrary set If Ci ⊆ IR n , i ∈ I , are convex, then C = ∩i∈I Ci is convex Let C and D be two convex sets in IRn and let a and b be two real numbers Then the following set is convex : aC + bD := {ac + bd | c ∈ C , d ∈ D} tvnguyen (University of Science) Convex Optimization / 108 Chapter Convex sets and convex functions taking the infinity value Illustration X X Y Y convex tvnguyen (University of Science) non convex Convex Optimization / 108 Chapter Convex sets and convex functions taking the infinity value Examples of convex sets The following are some examples of convex sets : (1) Hyperplane : S = {x|p T x = α}, where p is a nonzero vector in IRn , called the normal to the hyperplane, and α is a scalar (2) Half-space : S = {x|p T x ≤ α}, where p is a nonzero vector in IRn , and α is a scalar (3) Open half-space : S = {x|p T x < α}, where p is a nonzero vector in IRn and α is a scalar (4) Polyhedral set : S = {x|Ax ≤ b}, where A is an m × n matrix, and b is an m vector (Here the inequality should be interpreted elementwise.) tvnguyen (University of Science) Convex Optimization / 108 Chapter Convex sets and convex functions taking the infinity value Examples of convex sets (5) Polyhedral cone : S = {x|Ax ≤ 0}, where A is an m × n matrix (6) Cone spanned a finite number of vectors : Pby m S = {x|x = j=1 λj aj |λj ≥ 0, j = 1, , m}, where a1 , , am are given vectors in IRn (7) Neighborhood : Nε (¯ x ) = {x ∈ IRn |kx − x¯k < ε}, where x¯ is a fixed vector in IRn and ε > tvnguyen (University of Science) Convex Optimization / 108 Chapter Convex sets and convex functions taking the infinity value Convex cone Some of the geometric optimality conditions that we will study use convex cones Definition A nonempty set C in IRn is called a cone with vertex zero if x ∈ C implies that αx ∈ C for all α ≥ If, in addition, C is convex, then C is called a convex cone xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx Convex cone tvnguyen (University of Science) xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx Nonconvex cone Convex Optimization / 108 Chapter Convex sets and convex functions taking the infinity value Convex combination and convex hull of a set Definition x is said to be a convex combination of x , , x m if there exist α1 ≥ 0, , αm ≥ such that x = α1 x + · · · + αm x m , and α1 + · · · + αm = The convex hull of C (denoted conv C ) is the intersection of all convex subsets containing C Proposition (Carath´eodory’s lemma) Let C ⊆ IRn Then each element of conv C is a convex combination of at most n + points of C tvnguyen (University of Science) Convex Optimization 10 / 108 Chapter Convex sets and convex functions taking the infinity value Illustration tvnguyen (University of Science) Convex Optimization 11 / 108 Chapter Convex sets and convex functions taking the infinity value Closed convex hull Remark The convex hull of an open subset is open But the convex hull of a closed set is not necessarily closed Example Let C = {(0, 0)} ∪ {(x, y ) | x ≥ 0, xy = 1} Then conv C = {(0, 0)} ∪ {(x, y ) | x > 0, y > 0} is not closed See the figure below So we have to use the closed convex hull tvnguyen (University of Science) Convex Optimization 12 / 108 Chapter Convex sets and convex functions taking the infinity value Closed convex hull Definition The closed convex hull of a subset C of IRn is the smallest closed convex subset containing C It is denoted conv C Proposition The closed convex hull of a subset C is equal to the closure of its convex hull, i.e., conv C = conv C Proposition The convex hull of a bounded set is bounded The convex hull of a compact set is compact tvnguyen (University of Science) Convex Optimization 13 / 108 Chapter Convex sets and convex functions taking the infinity value Domain and Epigraph Definition Let f : IR n → IR ∪ {+∞} The domain of f is the set dom f = { x ∈ IR n | f (x) < +∞ } The function f is proper if dom f is nonempty If f is proper, then the epigraph of f is the nonempty set defined by epi f = {(x, r )|f (x) ≤ r } R X tvnguyen (University of Science) Convex Optimization 14 / 108 Chapter Convex sets and convex functions taking the infinity value Convex function Definition IRn × IR Proposition f is said to be convex if its epigraph is a convex subset of Let f : IRn → IR ∪ {+∞} Then f is convex if and only if ∀x, y ∈ IRn and ∀ λ ∈ [0, 1] f (λx + (1 − λ)y ) ≤ λf (x) + (1 − λ)f (y ) R x tvnguyen (University of Science) z y Convex Optimization X 15 / 108 Chapter Convex sets and convex functions taking the infinity value Example The indicator function Let S be a nonempty subset of IR n The indicator function of S is defined by  if x ∈ S n δS : IR → IR ∪ {+∞} δS (x) = +∞ otherwise The indicator function δS is a proper function whose domain is S Since epi δS = S × IR + , we have that δS is convex if and only if S is convex Moreover,  f is proper and convex  S ⊆ IR n nonempty and convex =⇒ f + δS is proper and convex  dom f ∩ S 6= ∅ We have a correspondence between convex sets and convex functions : S convex set → δS convex function f convex function → epi f convex set tvnguyen (University of Science) Convex Optimization 16 / 108 Chapter Convex sets and convex functions taking the infinity value Jensen’s inequality Proposition Let f : IRn → IR ∪ {+∞} be proper and convex Then for all collections of points {x1 , , xk } in dom f and all α = (α1 , , αk ) in the unit simplex ∆k of IR k , the following inequality holds : k k X X f( αi xi ) ≤ αi f (xi ) i=1 i=1 Let us recall that P α = (α1 , , αk ) ∈ ∆k means all the αi , i = 1, , k are nonnegative and ki=1 αi = tvnguyen (University of Science) Convex Optimization 17 / 108 Chapter Convex sets and convex functions taking the infinity value Operations preserving convexity Proposition Let C be a convex subset of IRn and let f1 , , fm be convex functions Let also and w1 , , wm ≥ Then w1 f1 + · · · + wm fm and max1≤i≤m fi (x) are convex functions More generally, let {fi }i∈I be a family of convex functions Then f = supi∈I fi is a convex function tvnguyen (University of Science) Convex Optimization 18 / 108 Chapter Convex sets and convex functions taking the infinity value Passing from sets to functions A common device to construct convex functions on IRn is to construct a convex set F in IRn+1 and then take the function whose graph is the lower boundary of F in the following sense Proposition Let F be any convex set in IRn+1 and let f (x) = inf{µ | (x, µ) ∈ F } Then f is a convex function Example Let fi , i ∈ I be proper convex functions on IRn Then F = ∩i∈I epi fi is convex and F is the epigraph of the convex function f = supi∈I fi Let us introduce another operation called the infimal convolution tvnguyen (University of Science) Convex Optimization 19 / 108 Chapter Convex sets and convex functions taking the infinity value Infimal convolution We introduce the functional operation which corresponds to the addition of epigraphs as sets in IRn+1 Let f1 , f2 be proper convex functions on IRn Let Fi = epi fi and let F = F1 + F2 Then F is convex Now (x, µ) ∈ F ⇔ ∃ (xi , µi ) s.t µi ≥ fi (x), µ = µ1 + µ2 , x = x1 + x2 So the convex function f corresponding to F is f (x) = inf {f1 (x1 ) + f2 (x2 ) | x = x1 + x2 } Definition Let f1 and f2 be two functions from IRn to IR ∪ {+∞} Their infimal convolution is the function from IRn to IR ∪ {+∞} defined by (f1 ⊕ f2 )(x) := inf{f1 (x1 ) + f2 (x2 ) : x1 + x2 = x} = tvnguyen (University of Science) inf y ∈IRn [f1 (y ) + f2 (x − y )] Convex Optimization 20 / 108 Chapter Convex sets and convex functions taking the infinity value Infimal Convolution Proposition Let f1 and f2 be two proper convex functions If they have a common affine lower bound : for some (s, b) ∈ IRn × IR, fj (x) ≥ hs, xi − b for j = 1, and all x ∈ IRn , then their infimal convolution is also proper and convex Furthermore epis (f1 ⊕ f2 ) = epis (f1 ) + epis (f2 ) where epis (f ) = {(x, r ) ∈ IRn × IR | f (x) < r } tvnguyen (University of Science) Convex Optimization 21 / 108 Chapter Convex sets and convex functions taking the infinity value Convex hull of a function Let f : IRn → IR ∪ {+∞} be proper and minorized by an affine function, i.e., there exists (s, b) ∈ IRn × IR such that f (x) ≥ < s, x > −b for all x ∈ IRn Let F = conv epi f and g (x) = inf{r : (x, r ) ∈ conv epi f } Then g is convex It is denoted conv f and is called the convex hull of f Proposition The convex hull of f coincides with the following two functions g1 and g2 on IRn : g1 (x) = sup{h(x) : h is proper and convex, h ≤ f } P g2 (x) = inf{ kj=1 αj f (xj ) : k = 1, 2, ; P α ∈ ∆k , xj ∈ dom f , kj=1 αj xj = x} where ∆k = {(α1 , , αk ) : tvnguyen (University of Science) Pk j=1 αj = 1, αj ≥ for j = 1, , k} Convex Optimization 22 / 108 Chapter Convex sets and convex functions taking the infinity value Other concepts of convexity Definition convex and A function f : IRn → IR ∪ {+∞} is strictly convex if it is f (λx + (1 − λ)y ) < λf (x) + (1 − λ)f (y ) ∀x 6= y ∈ IRn and ∀ λ ∈ (0, 1) A function f : IRn → IR ∪ {+∞} is strongly convex (with modulus b) if ∃b > : ∀x, y ∈ IRn , ∀λ ∈ [0, 1], f (λx + (1 − λ)y ) ≤ λf (x) + (1 − λ)f (y ) − b λ (1 − λ) kx − y k2 Example : Let A be a positive definite symmetric matrix of dimension n Then the quadratic form f (x) = 21 hAx, xi is strongly convex with modulus the smallest eigenvalue of A A strongly convex function is strictly convex but the converse is not true (example : f (x) = x ) tvnguyen (University of Science) Convex Optimization 23 / 108 ... Chapter Convex sets and convex functions taking the infinity value Illustration X X Y Y convex tvnguyen (University of Science) non convex Convex Optimization / 108 Chapter Convex sets and convex functions. .. xxxxxxxxxxxxx Nonconvex cone Convex Optimization / 108 Chapter Convex sets and convex functions taking the infinity value Convex combination and convex hull of a set Definition x is said to be a convex. .. the infinity value Illustration tvnguyen (University of Science) Convex Optimization 11 / 108 Chapter Convex sets and convex functions taking the infinity value Closed convex hull Remark The convex