Chapter 4. Subdifferential calculus for convex functions Chapter 4. Subdifferential calculus for convex functions tvnguyen (University of Science) Convex Optimization 66 / 108 Chapter 4. Subdifferential calculus for convex functions Directional derivative Definition. Let f : IR n → IR ∪ {+∞}, x ∈ domf and d ∈ IR n . The directional derivative of f at x in the direction d is f  (x, d) = lim t↓0 f (x + td) − f (x) t if the limit exists (-∞ and ∞ are allowed) Proposition. Let f : IR n → IR ∪ {+∞} be a convex function, and let x ∈ dom f . For each d, the difference quotient in the definition of f  (x, d) is a non-decreasing function of t > 0, so that f  (x, d) = inf t>0 f (x + t d) − f (x) t If f is differentiable at x, then f  (x, d) = ∇f (x), d. tvnguyen (University of Science) Convex Optimization 67 / 108 Chapter 4. Subdifferential calculus for convex functions Subdifferential. Motivation To obtain the simplest possible dual representation of a closed convex set C, we have introduced the notion of supporting hyperplane. When taking C the epigraph of a closed convex proper function f , the corresponding notion is the exact minoration : an affine function l : IR n → IR is an exact minorant of f at x if l ≤ f and l(x) = f (x) Setting l(y) = x ∗ , y + α, this becomes f (y) ≥ x ∗ , y + α ∀y ∈ IR n , and f (x) = x ∗ , x + α which is equivalent to (α = f (x) − x ∗ , x) ∀y ∈ IR n f (y) ≥ f (x) + x ∗ , y − x This leads to the following definition. tvnguyen (University of Science) Convex Optimization 68 / 108 Chapter 4. Subdifferential calculus for convex functions Subgradient. Subdifferential Definition. Let f : IR n → IR ∪ {+∞} be a convex function. A vector x ∗ is said to be a subgradient of f at x if ∀y ∈ IR n f (y) ≥ f (x) + x ∗ , y − x. The set of subgradients of f at x is called the subdifferential of f at x and is denoted by ∂f (x). Proposition. The subdifferential ∂f (x) is a closed convex set. It may be empty. tvnguyen (University of Science) Convex Optimization 69 / 108 Chapter 4. Subdifferential calculus for convex functions Examples f (x) = |x| ∂f (0) = [−1, 1], ∂f (x) = {1} if x > 0, ∂f (x) = −1 if x < 0 f (x) = e x − 1 if x ≥ 0 and 0 if x < 0 ∂f (0) = [0, 1], ∂f (x) = {e x } if x > 0, ∂f (x) = 0 if x < 0 x |x| x e x − 1 0 tvnguyen (University of Science) Convex Optimization 70 / 108 Chapter 4. Subdifferential calculus for convex functions Proposition. Let f : IR n → IR ∪ {+∞} be a convex function and let x ∈ dom f . Then x ∗ is a subgradient of f at x if and only if f  (x, d) ≥ x ∗ , d, ∀d. In fact, the closure of f  (x, d) as a convex function of d is the support function of the closed convex set ∂f (x). Proposition. Let f : IR n → IR ∪ {+∞} be a convex function. Then ∂f (x) is empty when x ∈ domf , and nonempty when x ∈ ri(domf ). In that case f  (x, y) is closed and proper as a function of y , and f  (x, d) = sup{< x ∗ , d > |x ∗ ∈ ∂f (x)} = δ ∗ (d|∂f (x)). Finally, ∂f (x) is a non-empty bounded set if and only if x ∈ int(domf ), in which case f  (x, d) is finite for every d. tvnguyen (University of Science) Convex Optimization 71 / 108 Chapter 4. Subdifferential calculus for convex functions Proposition. For any proper convex function f and any vector x, the following four conditions on a vector x ∗ are equivalent to each other : (a) x ∗ ∈ ∂f (x) (b) x ∗ , z − f (z) achieves its supremum in z at z = x (c) f (x) + f ∗ (x ∗ ) ≤ x ∗ , x (d) f (x) + f ∗ (x ∗ ) = x ∗ , x If (cl f )(x) = f (x), three more conditions can be added to this list (e) x ∈ ∂f ∗ (x ∗ ) (f) z ∗ , x − f ∗ (z ∗ ) achieves its supremum in z ∗ at z ∗ = x ∗ (g) x ∗ ∈ ∂(clf )(x) tvnguyen (University of Science) Convex Optimization 72 / 108 Chapter 4. Subdifferential calculus for convex functions Subgradient and monotonicity Definition. An operator T : IR n → 2 IR n is said to be monotone, if for any x, y and any x ∗ , y ∗ belonging to Tx and Ty respectively, y ∗ − x ∗ , y − x ≥ 0. T is said to be maximal monotone if it is monotone and its graph cannot be properly included into the graph of another monotone operator on IR n Proposition. Let f : IR n → IR ∪ {+∞} be proper, closed and convex function. Then ∂f (x) is a maximal monotone mapping. tvnguyen (University of Science) Convex Optimization 73 / 108 Chapter 4. Subdifferential calculus for convex functions Derivability of convex functions Let f be a function from IR n to [−∞, +∞], and let x be a point where f is finite. According to the usual definition, f is differentiable at x if and only if there exists a vector x ∗ (necessarily unique) with the property that f (y) = f (x) + x ∗ , y − x + o(|y − x|) or in other words lim y→x f (y) − f (x) − x ∗ , y − x |y − x| = 0. Such a x ∗ , if it exists, is called the gradient of f at x and is denoted by ∇f (x) tvnguyen (University of Science) Convex Optimization 74 / 108 Chapter 4. Subdifferential calculus for convex functions Proposition. Let f be a convex function, and let x be a point where f is finite. If f is differentiable at x, then ∇f (x) is the unique subgradient of f at x, so far in particular ∀y ∈ IR n f (y) ≥ f (x) + ∇f (x), y − x. Conversely, if f has a unique subgradient at x, then f is differentiable at x and ∂f (x) = {∇f (x)}. Proposition. Let f be a convex function on IR n , and let x be a point at which f is finite. A necessary and sufficient condition for f to be differentiable at x is that the directional derivative function f  (x, ·) be linear. Moreover, this condition is satisfied if merely the n two-sided partial derivatives ∂f (x)/∂ξ j exist at x and are finite. tvnguyen (University of Science) Convex Optimization 75 / 108 [...]... Science) Convex Optimization f (xi ) → x ∗ } 76 / 108 Chapter 4 Subdifferential calculus for convex functions Calculus rules Proposition Let f1 , f2 be two proper closed convex functions on IRn For x ∈ dom f1 ∩ dom f2 , there holds ∂(f1 + f2 )(x) ⊃ ∂f1 (x) + ∂f2 (x) with equality when ri dom f1 ∩ ri dom f2 = ∅ or int dom f1 ∩ dom f2 = ∅ Proposition Let f1 , f2 be two proper closed convex functions. .. x) It can be proven that fc is a differentiable convex function and that its gradient is given by fc (x) = c(x − xc ) tvnguyen (University of Science) Convex Optimization 78 / 108 Chapter 4 Subdifferential calculus for convex functions Calculus rules Let {fj }j∈J be a collection of convex functions from IR n into I and let R f : I n → I ∪ {+∞} be defined, for all x, by R R f (x) = sup fj (x) j∈J Proposition... is finite Then the following inclusion holds for every x ∈ IRn : ∂ f (x) ⊃ conv { ∪ ∂fj (x) | j ∈ J(x) }, where J(x) = { j ∈ J | fj (x) = f (x) } To obtain the equality, we suppose that J is a finite subset In that case the set J(x) is nonempty tvnguyen (University of Science) Convex Optimization 79 / 108 Chapter 4 Subdifferential calculus for convex functions Calculus rules Proposition If J = { 1, ,... ∂f2 (y2 ) = ∅, the inf-convolution is exact at x = y1 + y2 and equality holds tvnguyen (University of Science) Convex Optimization 77 / 108 Chapter 4 Subdifferential calculus for convex functions Moreau-Yosida Regularization Definition Let c > 0 and let f : IRn → IR ∪ {+∞} be a proper closed convex function The function fc = f ⊕ c · 2 is called the 2 Moreau-Regularization of f One has fc (x) = min {f...Chapter 4 Subdifferential calculus for convex functions Proposition Let f be a proper convex function on IRn , and let D be the set of points where f is differentiable Then D is a dense subset of int(dom f ), and its complement in int(dom f ) is a set of measure... calculus for convex functions Calculus rules Proposition If J = { 1, , m }, then ∂ f (x) = conv { ∪ ∂fj (x) | j ∈ J(x) } Corollary Let f1 , , fm be differentiable convex functions from I n into R n, I Then f = sup1≤j≤m fj is convex and for all x ∈ IR R ∂f (x) = conv { fj (x) | j ∈ J(x) } Example : Consider the function f (x) = max{f1 (x), f2 (x), f3 (x)} where f1 (x) = −x1 − x2 , f2 (x) = −x1 + x2... −x1 + x2 and the point (4, 8) Since J((4, 8)) = {2, 3}, f3 (4, 8) = (1, 0)T , we have and f3 (x) = x1 , f2 (4, 8) = (−1, 1)T and ∂f (4, 8) = conv {(−1, 1)T , (1, 0)T } tvnguyen (University of Science) Convex Optimization 80 / 108 . 4. Subdifferential calculus for convex functions Chapter 4. Subdifferential calculus for convex functions tvnguyen (University of Science) Convex Optimization. finite for every d. tvnguyen (University of Science) Convex Optimization 71 / 108 Chapter 4. Subdifferential calculus for convex functions Proposition. For