Codex @codex  0

The squared distance to a convex set is the infimal convolution of the set's indicator functional of a constraint set and the half-squared Hilbert space norm. Its convex conjugate is the sum of the half-squared norm and the support function of the set.

The convex conjugate records a function using slopes rather than input positions. Let $f:{\mathbb{R}}^n\to \mathbb{R}\cup \{+\infty \}$ be a proper extended-real function: it is finite somewhere and never takes $-\infty$. Its convex conjugate iswhere $p\cdot x$ is the dot product and $\mathrm{dom}f=\{x:f(x)<+\infty \}$ is the effective domain. An infinite value of $f$ excludes that input from the supremum. The value $f^{\ast }(p)$ can be $+\infty$, but cannot be $-\infty$ because an input with finite $f(x)$ supplies a finite lower bound.

The convex conjugate is automatically a convex function, even when $f$ is not convex: each fixed input gives an affine function $p\mapsto p\cdot x-f(x)$, and their supremum is convex. It is also lower semicontinuous. It need not be proper: for $f(x)=-x^2$ on $\mathbb{R}$, $px-f(x)=px+x^2$ is unbounded above for every $p$. The definition and the automatic convexity are discussed in Boyd and Vandenberghe, Convex Optimization (2004), §3.3.1.

The best lower bound with a prescribed slope. Whenever $f^{\ast }(p)$ is finite, the affine functionis an affine minorant of $f$. Indeed, an affine function $p\cdot x+b$ lies below $f$ everywhere exactly when $b\le {\inf }_x(f(x)-p\cdot x)=-f^{\ast }(p)$. Thus $-f^{\ast }(p)$ is the largest possible intercept at slope $p$. The convex conjugate trades positions for slopes: $x$ specifies where to inspect $f$, whereas $p$ specifies which global affine minorant to use. Taking the supremum over all these affine minorants produces the biconjugate; the Fenchel-Moreau theorem says this recovers a proper convex function when it is lower semicontinuous.

A quadratic, computed by completing the square. For the convex quadratic function $f(x)=x^2/2$,Consequently $f^{\ast }(p)=p^2/2$, and the unique maximizing input is $x=p$. The affine minorant ${\ell }_p(x)=px-p^2/2$ is the tangent line at $(p,p^2/2)$: its slope is $p$ and its intercept is $-f^{\ast }(p)=-p^2/2$. For example, $p=2$ gives $f^{\ast }(2)=2$ and the tangent line $2x-2$, touching at $(2,2)$. This is the same curve described either by its point values or by the intercepts of its tangent lines.

A corner turns into a constraint on slopes. For the absolute value $f(x)=|x|$, if $|p|\le 1$ then $px-|x|\le 0$, with value zero at $x=0$. If $p>1$, taking $x\to +\infty$ makes $(p-1)x\to +\infty$; if $p<-1$, taking $x\to -\infty$ does the same. Therefore its convex conjugate is zero on $[-1,1]$ and $+\infty$ outside: it is the indicator functional of a constraint set for this interval. For $|p|<1$ the unique maximizing input is zero; for $p=1$ every $x\ge 0$ maximizes, and for $p=-1$ every $x\le 0$ maximizes. The corner's many supporting slopes are exactly the subdifferential $\partial f(0)=[-1,1]$.

A supremum need not be attained. For a differentiable convex function finite on all of ${\mathbb{R}}^n$, an input maximizes $p\cdot x-f(x)$ exactly when $p=\nabla f(x)$. If $f$ is strictly convex, there is at most one such input; existence is a separate issue. For the exponential function $f(x)=e^x$, the derivative is positive. At $p>0$, solving $p=e^x$ gives $x=\log p$ and $f^{\ast }(p)=p\log p-p$. At $p=0$, the supremum of $-e^x$ is zero as $x\to -\infty$, with no maximizing input. At $p<0$, $px-e^x\to +\infty$ as $x\to -\infty$. Thus a strictly convex differentiable function can have both unattained finite conjugate values and infinite conjugate values.

Maximize $px-x^2/2$ over $|x|\le 1$. The maximizer is $x=\max (-1,\min (p,1))$, giving the displayed Huber loss. The indicator functional of a constraint set is zero on the interval and infinity outside; it is not the zero-one indicator used in measure theory.

The effective domain of an extended-real function is $\mathrm{dom}E=\{u:E(u)<+\infty \}$. An infinite value can impose a hard constraint through an indicator functional of a constraint set.

The Fenchel–Young inequality says that every input and every slope obeywhere $f$ is any proper extended-real function and $f^{\ast }$ is its convex conjugate. No convexity assumption on $f$ is needed for this inequality: the defining supremum is at least the particular value $p\cdot x-f(x)$. If either term on the left is $+\infty$, the inequality still holds, but cannot be an equality because the dot product on the right is finite.

For a proper convex function, equality is equivalent to the subgradient conditionTo see this, with $f(x)$ finite, equality says that $x$ attains the supremum defining $f^{\ast }(p)$, so $p\cdot z-f(z)\le p\cdot x-f(x)$ for every $z$. Rearranging gives the subgradient inequality $f(z)\ge f(x)+p\cdot (z-x)$. Geometrically, the affine minorant ${\ell }_p(z)=p\cdot z-f^{\ast }(p)$ touches $f$ at $x$.

If the proper convex function is also closed, meaning its epigraph is a closed set, or equivalently it is lower semicontinuous, thenThe forward implication holds without closedness; the reverse one requires the additional hypothesis in general. Applying the equality criterion to $f^{\ast }$ relates $x\in \partial f^{\ast }(p)$ to $f^{\ast \ast }(x)+f^{\ast }(p)=p\cdot x$, and the Fenchel-Moreau theorem supplies $f^{\ast \ast }=f$ when $f$ is closed. See Bertsekas, Convex Optimization Theory: A Summary, §5.4.1, Proposition 5.4.3 for precisely this distinction.

A concrete failure of the reverse implication occurs for the indicator functional of a constraint set $f$ of the open interval $(-1,1)$. Its convex conjugate is $f^{\ast }(p)=|p|$: the endpoint gives the supremum, even though it is excluded from the interval. At $(x,p)=(1,1)$, $x\in \partial f^{\ast }(p)=\{1\}$, but $f(1)=+\infty$, so equality fails and $\partial f(1)$ is empty. Closing the interval changes the endpoint value and restores equality there.

For the convex quadratic function $f(x)=x^2/2$, the Fenchel–Young inequality is $x^2/2+p^2/2\ge px$, and equality holds exactly when $x=p$. For the absolute value $f(x)=|x|$, finite equality requires $p=\mathrm{sgn}(x)$ at $x\ne 0$, where $\mathrm{sgn}$ is the sign function; at $x=0$ every $p\in [-1,1]$ gives equality. These are exactly the subgradients of the absolute value, including its entire interval of slopes at the corner.

An infimal convolution is the cheapest way to split a prescribed total between two costs. Suppose two activities contribute $u$ and $v$ to a required total $x$, costing $f(u)$ and $g(v)$. Their combined cost is optimized over all splits $u+v=x$, rather than charging both activities for the entire total.

For proper extended-real functions $f,g:X\to \mathbb{R}\cup \{+\infty \}$ on a real vector space $X$, defineThe definition does not require convexity. An infinite cost forbids that contribution, as with an indicator functional of a constraint set. If no split has two finite costs, set $h(x)=+\infty$; if the costs of admissible splits are unbounded below, $h(x)=-\infty$. Thus proper inputs need not produce a proper extended-real function. The notation $E\square F$ denotes the same operation. See Parikh and Boyd, Proximal Algorithms, §3.1 for the standard convex analysis definition and its connection to the Moreau envelope.

An infimum is a best lower bound, whereas a minimum is a value realized by some split. Writing $\min$ in place of $\inf$ requires an attaining pair $(u_{\ast },v_{\ast })$. For convex functions, mixing two nearly optimal splits and applying convexity to each cost proves that their infimal convolution is convex whenever it is proper; one must use nearly optimal splits because exact minimizers need not exist. Its convex conjugate is $f^{\ast }+g^{\ast }$ whenever the extended real number arithmetic is well defined; see conjugate of an infimal convolution for the precise identity. This turns the optimization over splits into addition in the conjugate variables.

Two quadratic costs: share the load according to its price. Let $X=\mathbb{R}$ and let the convex quadratic functions beFor fixed $x$, eliminate $v=x-u$. The derivative of the total cost is $au-b(x-u)$, so equal marginal costs giveThe cheaper activity receives more of a positive total: when $a<b$ and $x>0$, $u_{\ast }>v_{\ast }$. Completing the square gives an exact certificate for every competing split:Consequently the infimum is a unique minimum, andThe effective quadratic coefficient is $c=ab/(a+b)$, with $1/c=1/a+1/b$. It is smaller than either $a$ or $b$: allowing both activities reduces the cost compared with assigning everything to either one. If $a=b$, each receives half the total and $c=a/2$. This is a simple quadratic optimization problem whose answer preserves the form of the original costs.

Smoothing a corner by splitting its cost. Take the absolute value function $f(u)=|u|$ and the convex quadratic function $g(v)=v^2/(2t)$, with $t>0$. This infimal convolution is the Moreau envelope of the absolute value function:Here $\mathrm{sgn}$ is the sign function, with $\mathrm{sgn}(0)=0$. The optimal $u_{\ast }$ is soft thresholding. In the central region, the quadratic function carries the whole total; outside it, $v_{\ast }$ stays at $t\mathrm{sgn}(x)$ and the absolute value function carries the excess. At $t=1$ the result is the Huber loss in the normalization $x^2/2$ near zero and $|x|-1/2$ outside. Its derivative is $x$ for $|x|\le 1$ and $\mathrm{sgn}(x)$ outside, so the corner becomes a continuous change of slope. Parikh and Boyd's ^[ref] uses twice this normalization for the Huber loss; the explicit formula here fixes the convention. This is also the one-dimensional case of the existing Moreau envelope of the Euclidean norm.

Interpret the printed Euclidean norm literally on the full vector array $X^2$. It is a single global norm, not the sum of pixelwise gradient lengths in discrete isotropic total variation. Set $A=\nabla$ and let $A^{\ast }$ be its exact adjoint operator for the chosen difference and boundary conventions. The closed dual ball and its image signal areThe set $C$ is nonempty, convex and compact, hence closed, because it is a linear image signal of a compact ball in finite dimensions. Euclidean norm duality givesthe support function of $C$.

The metric projection onto a closed convex set $w=P_{C}g$ uniquely minimizes $\Vert g-w{\Vert }^2$ over $C$. Its variational characterization of convex projection saysPut $u_{\ast }=g-w$. This inequality says precisely that ${\sigma }_C(u_{\ast })=⟨u_{\ast },w⟩$. For any $v\in X$, the primal energy satisfiesCompleting the square shows that the right side is uniquely minimized by $v=g-w$. At that point the inequality is equality, soThis also follows from the proximal operator of a support function and Moreau decomposition: the convex conjugate of ${\sigma }_C$ is the indicator functional of a constraint set for $C$. The squared fidelity is strictly convex and coercive, so the primal minimizer exists and is unique.

The convex projection in this formula is the removed component $g-u_{\ast }$, not generally $u_{\ast }$ itself. A metric convex projection onto one fixed closed convex set is idempotent. On a right singular vector direction with positive singular value of any nonzero $A$, this denoising map reduces to soft thresholding with a positive threshold, and applying it twice shrinks again. It therefore cannot be such a convex projection for all data. This qualifies the printed convex projection wording while giving the required global gradient-norm projection residual.

Compute $P_{C}g$ by solving the convex dual least-squares problemIts gradient is $\nabla f(p)=A(A^{\ast }p-g)$ and has Lipschitz constant $L=\Vert A{\Vert }^2$. Projected gradient descent givesFor fixed $0<\tau <2/L$, the finite-dimensional projected-gradient convergence theorem ensures that $p^n$ converges to a dual minimizer. Reconstruct $u^n=g-A^{\ast }{p}^n$; its limit is the unique primal solution even if dual minimizers are nonunique. If $A=0$, the primal solution is simply $g$ and no iteration is needed. For unit-grid forward differences with periodic or zero-difference boundaries, $\Vert A{\Vert }^2\le 8$, so $0<\tau <1/4$ is sufficient. Grid-spacing factors or other boundary stencils change this bound.

The normalization here is global. If a pixelwise sum of gradient lengths had instead been intended, the dual feasible set would be a product of pixelwise balls and convex projection would normalize each block separately; it is a different regularizer and should not be silently substituted.

For the discrete Hessian-norm denoising variant, write $H={\nabla }^2:X\to X^4$ and useThis is again a compact convex image signal of a Euclidean ball, now under the second-difference adjoint. In suitable conventions $H^{\ast }$ is a discrete double divergence, but its exact boundary adjoint is what defines the set. It lies in $(\ker H{)}^{\perp }$, so components in $\ker H$ are preserved by denoising. Interior second derivatives annihilate affine image signals; whether all such image signals remain in the kernel depends on the boundary convention. With a pixelwise Hessian norm, the corresponding balls would instead be four-component blocks.

For the absolute value function, $px-|x|\le 0$ for all $x$ if $|p|\le 1$, with equality at $x=0$. If $|p|>1$, taking $x=t\mathrm{sign}(p)$ with $t\to \infty$ makes the expression diverge. Thus its convex conjugate isThis is the indicator functional of a constraint set for the dual unit interval, not a zero-one indicator function.

The indicator functional of a constraint set restricts the conjugate supremum to $[-1,1]$. Complete the square:Its maximizer is $x=p$ when $|p|\le 1$, and the nearer endpoint $x=\mathrm{sign}(p)$ otherwise. The convex conjugate of a constrained quadratic is thereforeThis is the unit-threshold Huber loss; both branches and their first derivatives agree at the transition points. The original PDF constrains $|x|\le 1$, including both signs; the damaged TeX loses that absolute value.

Write $D=\nabla$ and use the real pixelwise inner products. Let $D^{\ast }$ be the adjoint operator, defined by $⟨Du,p⟩=⟨u,D^{\ast }p⟩$. For each nontrivial grid direction its one-dimensional formula iswith the analogous formula in $j$ for $D_y^{+\ast }{p}^2$. Their sum is $D^{\ast }p$; unused components $p_{N,j}^1$ and $p_{i,N}^2$ do not contribute. For $N=1$, $D=D^{\ast }=0$. If discrete divergence is defined, its sign convention is $\mathrm{div}=-D^{\ast }$, as in the adjoint of a discrete forward gradient.

Define the compact convex set $P_{\lambda }=\{p\in X^2:|p_{i,j}{|}_2\le \lambda \}$ and its image $C=D^{\ast }{P}_{\lambda }$. The image is compact and convex, hence closed, and contains zero. The finite-dimensional Euclidean duality formula $\lambda |a{|}_2={\max }_{|p{|}_2\le \lambda }a\cdot p$, applied independently at every pixel, givesThus the penalty is the support function ${\sigma }_C$. Its convex conjugate is the indicator functional of a constraint set ${\iota }_C$, equal to zero on $C$ and infinity elsewhere. For $w\in C$, the conjugate supremum is zero; for $w\notin C$, the separation theorem supplies a direction $u$ with $⟨u,w⟩>{\sigma }_C(u)$, and scaling that direction makes the supremum infinite.

The proximal operator is ${\mathrm{prox}}_{{\sigma }_C}(g)=\arg {\min }_u\{{\sigma }_C(u)+\frac{1}{2}\Vert u-g{\Vert }_2^2\}$. It exists uniquely because the objective is continuous, coercive and strictly convex. The Moreau decomposition for a proper closed convex function gives ${\mathrm{prox}}_{{\sigma }_C}(g)+{\mathrm{prox}}_{{\iota }_C}(g)=g$. The second term is the Euclidean projection onto a convex set, so the projection residual for discrete total variation isAlternatively, the variational characterization of convex projection says $w=P_{C}g$ exactly when $⟨g-w,z-w⟩\le 0$ for every $z\in C$. Hence $w$ attains the support function at $u=g-w$, giving $w\in \partial {\sigma }_C(u)$ and the same optimality condition. This supplies a direct verification of the Moreau decomposition step in this instance.

The printed wording needs “the residual after a projection”, rather than “the projection itself”. In general this denoising map is not a projection onto any fixed closed convex set. A metric projection onto a closed convex set is idempotent, while denoising a sufficiently large jump twice shrinks it twice. Concretely, for $N=2$ and data with rows $(0,0)$ and $(6\lambda ,6\lambda )$, the minimizer has rows $(\lambda ,\lambda )$ and $(5\lambda ,5\lambda )$. Reapplying the map gives rows $(2\lambda ,2\lambda )$ and $(4\lambda ,4\lambda )$, so it is not idempotent. These formulas have a global certificate: take $p_{1,j}^1=\lambda$, all other dual entries zero, giving $D^{\ast }p$ with rows $(-\lambda ,-\lambda )$ and $(\lambda ,\lambda )$; it saturates every positive row difference and yields the primal optimality condition. The qualification is therefore not merely a consequence of a restricted two-level ansatz.

The small-parameter limit depends on data compatibility with the regularizer domain. SetFor every such $v$, optimality givesTaking the upper limit as $\alpha \downarrow 0$ and then the infimum over $v$ proves ${\Phi }_{\alpha ,f}(u_{\alpha })\to m$. Its two nonnegative contributions above $m$ must vanish, yielding the general resultIn particular, the requested zero-misfit limit holds when $f\in \overline{K(\mathrm{dom}J)}$. A sufficient condition is an exact solution $v$ with $Kv=f$ and $J(v)<\infty$; thenMembership of $f$ in $\mathcal{R}(K)$ alone is insufficient. Take $\mathcal{U}=\mathcal{V}=\mathbb{R}$, $K=I$, $f=1$, and $J$ the indicator functional of a constraint set $\{0\}$. All positive-parameter objectives are proper extended-real functions with coercivity and sequential lower semicontinuity, with the unique minimizer $u_{\alpha }=0$, yet $D(u_{\alpha })=1/2$ for every $\alpha$. Thus the first printed limit needs compatibility with the regulariser's effective domain; the second limit remains true under the given assumptions.

Write ${\iota }_C$ for the indicator functional of a constraint set $C$ and $q(u)=\Vert u{\Vert }^2/2$. ThenBoth terms are convex, hence their infimal convolution is the convex squared distance to a convex set. The closest point theorem in a Hilbert space supplies the unique metric projection onto a closed convex set $P_{C}u$ and gives $d_C(u)=\Vert u-P_{C}u{\Vert }^2/2$.

Identify the Hilbert space with its dual space using the Riesz representation theorem. Completing the square yields $q^{\ast }(p)=\Vert p{\Vert }^2/2$, while ${\iota }_C^{\ast }(p)={\sup }_{v\in C}⟨p,v⟩={\sigma }_C(p)$ is the support function. The conjugate of the squared distance to a convex set is thereforeFor the closed unit ball, the metric projection onto a closed convex set is $P_{C}u=u$ if $\Vert u\Vert \le 1$, and $P_{C}u=u/\Vert u\Vert$ otherwise. ThereforeThe last equality uses the Cauchy-Schwarz inequality to compute the unit ball's support function, attained in the direction of $p$ when $p\ne 0$.

For a differentiable convex function $F$ with $L$-Lipschitz gradient and a proper lower-semicontinuous convex function $H$, the proximal gradient method isThe proximal operator is ${\mathrm{prox}}_{\tau H}(v)=\arg {\min }_z\{H(z)+\Vert z-v{\Vert }^2/(2\tau )\}$. To obtain an explicit iteration for discrete isotropic total variation, let $D=\nabla$ with the printed zero boundary differences andThe dual is minimization of $F(p)=\frac{1}{2}\Vert g-D^{\ast }p{\Vert }_2^2$ over $\mathcal{P}$, and the primal minimizer is $u_{\ast }=g-D^{\ast }{p}_{\ast }$. Here $D^{\ast }=-\mathrm{div}$ must be the exact adjoint operator of these differences. For example,with the analogous formula in $j$ for $(D_y^{+}{)}^{\ast }$; for $N=1$ both operators are zero. Equivalently use $p_{0j}^1=p_{Nj}^1=p_{i0}^2=p_{iN}^2=0$ in the backward-difference expression, ignoring unused edge components.

The gradient of the dual objective is $D(D^{\ast }p-g)$, with Lipschitz constant $\Vert D{\Vert }^2\le 8$. Taking $H$ to be the indicator functional of a constraint set makes its proximal operator the Euclidean projection onto a convex set. Thus a valid explicit algorithm isThe bound follows by summing $(a-b{)}^2\le 2a^2+2b^2$ over grid differences. Finite-dimensional proximal gradient method convergence gives a dual minimizer and convergence of the recovered $u^k$ to the unique primal minimizer. Projection is pointwise in the dual; applying pointwise soft thresholding directly to $Du$ would not compute the primal TV proximal operator.

Use the extended-real characteristic functionalFor $\eta <0$ its sublevel set is empty, while for every finite $\eta \ge 0$ its sublevel set is $X^{\prime }$. Both are closed when $X^{\prime }$ is closed, so part ii proves that this indicator functional of a constraint set is lower semicontinuous.

For a nonempty closed convex set $C$, the squared-distance function is the Moreau envelope of its indicator functional of a constraint set. It is convex and differentiable withand this gradient is one-Lipschitz.

For a nonempty closed convex set $C$, the finite-valued support function has subgradients exactly at the maximizing points of $C$. This follows from Fenchel–Young inequality and the conjugacy between the support function and the indicator functional of a constraint set. For compact $C$, the exposed maximizing set is nonempty.