Put and . Using natural logarithms, the Kullback-Leibler divergence is
The integrand is zero where , including where both densities vanish, and is where . Equivalently, when is absolutely continuous with respect to , and it is otherwise. This is a well-defined extended integral: on , writing gives , so its negative part is integrable.
For Pinsker's inequality, an infinite Kullback-Leibler divergence makes the conclusion immediate. Assume it is finite. Set
Since , the positive and negative parts of have equal integrals. Thus
This is the total variation distance with the convention .
We first prove the binary partition bound for relative entropy. For any measurable with , apply Jensen inequality to the convex function under the conditional probability measure . Absolute continuity of measures gives
If , both sides are zero; if , finiteness forces as well. Apply this bound to and its complement to obtain
For , regarding as a function of gives
Consequently is a convex function and has its minimum zero at . By continuity this also holds at . For , the finite case has and the remaining cases have infinite divergence. This proves the Bernoulli relative entropy lower bound in every case. Combining the bounds yields
For the Hellinger distance, retain the unnormalized convention in which
The integral is the Hellinger affinity. In the finite-divergence case, for -almost every point. Apply with to get
The affinity is finite by Cauchy-Schwarz inequality. Mass of on creates no difficulty, since the affinity vanishes there and both probability measures have mass one. The infinite-divergence case is again immediate. We obtain the relative entropy bound for the unnormalized Hellinger distance:
Let be the finite nonempty set of routes for source-sink pair , with fixed nonnegative demand . A Wardrop equilibrium is a feasible route-flow vector such that every route carrying positive flow has minimum total delay among the routes for its pair. If
this means that, for each , some satisfies for , with equality whenever .
The link-route incidence matrix has when route uses link and zero otherwise, or the traversal multiplicity if repeated traversals are permitted. The source-sink route incidence matrix has when route serves pair and zero otherwise. Thus gives link throughputs, and gives the fixed demands.
The feasible route-flow set is a product of finite simplexes, hence nonempty and both a compact set and a convex set. The Beckmann potential
is continuous and convex, so it attains a minimum. Its route derivative is . At a minimum, moving a small amount of flow from any used route to any other route for the same pair cannot decrease the potential. Hence every used route has minimum route delay, exactly the Wardrop equilibrium condition. Conversely, at a Wardrop equilibrium , for any feasible ,
because the used flow has cost and competing routes cost at least . The first-order inequality for a convex function makes a global minimum. This proves both existence and the specified optimization characterization.
Since each is strictly increasing, its primitive is a strictly convex function. Therefore the objective is strictly convex as a function of link loads , and the equilibrium throughputs are unique. The mapping need not be injective on , so route flows need not be unique.
For a concrete route-flow nonuniqueness at a Wardrop equilibrium example, use one unit-demand pair with two successive stages, each containing two parallel links. There are four routes, indexed by their two link choices. Give every link delay . For every ,
produces load on all four links. Every route has delay one, so each vector is a Wardrop equilibrium with the same unique link loads.
To implement minimum average delay, minimize the total delay
The average differs only by the fixed total demand , provided it is positive. Choose any global minimizer , which exists by compactness, and write . Its first-order route-exchange conditions are the Wardrop equilibrium conditions for marginal link costs
A genuinely traffic-dependent choice guaranteed under exactly the printed hypotheses is the calibrated toll function
Here . The calibrated base charges are nonnegative because differentiable increasing delays have nonnegative derivatives, and the added penalties are nonnegative and nondecreasing in . The perceived link costs remain strictly increasing, and at they equal . Hence is a tolled Wardrop equilibrium. The same strictly convex Beckmann potential argument makes all its equilibrium link loads equal to ; every tolled equilibrium therefore has the globally minimum average delay. This is optimal-flow calibrated congestion tolling.
The familiar marginal external cost toll is , whose perceived-cost primitive is exactly . It implements the optimum directly when the total-delay objective is convex, for example when each is convex. That extra hypothesis does not follow from the printed strict increase of : for two identical parallel links with and total demand six, equal loads of three form a tolled Wardrop equilibrium, but the second derivative of at three is , so an unequal nearby split lowers total delay. The calibrated traffic-dependent toll above establishes the requested existence without silently assuming this additional convexity.
Normalize the constant relative risk aversion utility as . The printed specification of alone also permits an additive constant ; that would add to the value, so the stated homogeneity holds for the normalized value. Set
Multiplying initial wealth, the historical maximum, dollar investments, and consumption by multiplies every term of the wealth equation by , including the tax term. It maps admissible policies bijectively and multiplies the normalized reward by . Therefore the value scales as
The historical maximum means when initially .
Inside the state region , the maximum is locally constant. The Hamilton-Jacobi-Bellman equation is
At , the finite-variation contribution in the Itô formula is . The high-water mark tax boundary condition is consequently
For an increasing, strictly concave function of wealth, optimizing the two controls gives
The consumption convex conjugate is
Substitution of the homogeneity derivatives gives the reduced equation and boundary condition
The controls in scaled variables are and .
Use the wealth-variable Legendre dual
Since , the inverse map satisfies and . Inserting into the reduced Hamilton-Jacobi-Bellman equation gives a linear Euler equation
For the Euler differential equation operator on the left, its action on is . Write . Under , the particular solution is . For the nondegenerate case , the two characteristic roots are , with because and . Thus the general solution also contains .
Here and the relevant dual domain is . The inverse wealth ratio must have as . The particular solution and the negative-root term have derivatives tending to zero, whereas has an unbounded derivative unless . Hence the admissible dual solution has the claimed form
The quadratic-root formulation presupposes a nonzero market price of risk. If , the dual equation becomes first order; it is treated directly or by an appropriate nondegenerate limit rather than by assuming two quadratic roots.
The inverse map and the tax boundary provide two equations at :
Because , these become
Let . Eliminating gives . Also , since puts strictly between the two roots. The derivative boundary then gives . Therefore the constants are
There is an important admissibility qualification in the printed conclusion. With ,
The second term is negative, but decays faster than the first because . Direct differentiation at the boundary gives
A wealth-variable Legendre dual of a concave value must be a convex function. Thus the printed smooth tax-paying solution requires , with the limiting case allowed as a degenerate boundary. The assumptions and do not imply this: for example , , , , and give and , hence negative boundary dual curvature.
For higher tax the investor can avoid raising the historical maximum. The wealth-cap investment boundary replaces the tax-paying equality by , alongside . Solving these equations gives
These are the same constants with replaced by . More generally the admissible two-regime expression uses
in the constants, while the claimed formulas themselves describe the tax-paying regime. In the cap regime the portfolio volatility vanishes at and the consumption rate there is , so the wealth drift points inward and no new maximum is required. The remaining boundary inequality is , consistent with avoiding costly maximum increases. For , dual curvature is positive because the negative term in decays faster than the positive term. Thus this qualification repairs an actual missing parameter restriction rather than a TeX transcription error.
For a twice differentiable function, convexity on is equivalent to its Hessian matrix being positive semidefinite everywhere. Here the Hessian matrix is constant:
Its diagonal entries are positive, so positive semidefiniteness is equivalent to . The required parameter range is
The endpoints give convex functions with one flat direction; strict inequalities give a Hessian matrix defining a positive-definite quadratic form and a strictly convex function.
Fix the normalization
The Lasso penalty uses the L1 norm. All arguments below apply to every minimizer; uniqueness of the coefficient vector is not assumed.
The Karush-Kuhn-Tucker conditions are the subgradient optimality condition
These conditions are necessary and sufficient because the objective is a convex function. They use the subgradient of the absolute value, coordinate by coordinate.
Put . Centered columns imply , so the subtraction of in the model contributes nothing to the score. The Karush-Kuhn-Tucker conditions become
Multiply by . Since and , this gives the prediction inequality from Lasso stationarity:
This is the desired version of the Basic inequality for the Lasso. Using stationarity directly preserves the coefficient on the squared prediction norm.
For the Gaussian score event for the Lasso, write for column . The assumed Euclidean norm and the multivariate normal distribution of give
The may be correlated, which does not affect the union bound. Use the following sharp two-sided Gaussian tail bound, valid for with the standard normal distribution and every :
For completeness, the difference satisfies , , and as . It first increases and then decreases to zero, so it is nonnegative. This bound avoids introducing an unnecessary factor two.
Here is the standard normal distribution function. With , the union bound yields
Consequently
The score event uses the supremum norm of ; the centering term still vanishes because .
We next work deterministically on . Let
By Holder inequality, . Since , the triangle inequality gives
Substituting both inequalities into the Basic inequality for the Lasso produces
Because , this also proves the relevant Lasso cone condition, . The assumed Compatibility condition for the Lasso therefore applies to :
Set . Combining these inequalities gives
If , discarding the nonnegative term shows . If , compatibility gives , and the preceding inequality then gives . In either case , so
This is a fast prediction bound and an inactive-coordinate error bound on the same event. It is the fast-rate Lasso prediction bound under compatibility.
Finally, the same argument bounds the active-coordinate L1 norm by
If , then , and . When this implies ; when it implies . Thus the sign function obeys
This is Lasso sign recovery under compatibility. It concerns the sufficiently large nonzero coefficients; it does not assert that every estimated inactive coefficient is zero.
Use the random walk interpretation and optimal stopping value function recursion specified in part (b). If is convex, translating and integrating its convexity inequality gives, for ,
Thus the transition operator preserves convex functions. Also the pointwise maximum of convex functions is convex: each of two convex functions at an intermediate point is bounded by the same convex combination of their pointwise maximum at the endpoints. Starting with , backward induction in
therefore proves the asserted convexity in the intended model:
The statewise integrability convention in part (b) gives finite convex functions on all real states. The same inequality holds for the canonical extended value function wherever the expectations are well-defined. Integrability only along one started process need not make that canonical function finite at unused states: for , , and increment density proportional to , , but for . This concerns the canonical recursion away from visited states, rather than the almost-sure identity for . Arbitrary off-state versions of need not be convex; the recursive version is the one meant here. No zero-mean assumption on the increments was used, and convexity alone does not make a submartingale for an arbitrary drift.
An effective bandwidth measures how much capacity a random traffic source requires when unusually large demands matter. A mean-rate description loses burstiness; a peak-rate description can waste capacity when peaks are rare. The parameter in an effective bandwidth controls the emphasis placed on such bursts through the moment-generating function:
Here is the traffic in a chosen time unit. If instead measures traffic over length , the corresponding rate is . With independent stationary increments, this is independent of whenever the moment-generating function exists. More generally both the time scale and dependence structure matter.
The Chernoff bound follows directly from the Markov inequality, since is nonnegative and implies :
Optimize over the admissible positive values of to obtain
For independent sources , moment-generating functions multiply and their effective bandwidths add. Thus a total capacity has the one-time overflow bound . In particular, for independent and identically distributed random variables,
This is the positive-parameter part of the Legendre transform of a cumulant-generating function. The size of its exponent describes how a service margin reduces the chance of a large aggregate demand. A buffer-overflow event involves traffic over many intervals; a one-time Chernoff bound alone is not automatically a bound on that event. A time-dependent model and, for example, bounds on the union of relevant interval events are then needed.
Assume a finite moment-generating function in a neighborhood of zero. Differentiating the cumulant-generating function gives , and . Therefore the small-parameter limit is the mean traffic rate:
The variance term shows how an effective bandwidth charges extra capacity for burstiness. Also is a convex function with , so is nondecreasing for in its domain.
For the other limit, let , the essential supremum. If , almost surely gives . For every , , and
Taking a lower limit and then proves the large-parameter limit is peak traffic:
If has no finite upper bound and all positive exponential moments exist, the same lower bound for arbitrarily large makes the limit infinite. If positive exponential moments cease to exist at a finite parameter, the effective bandwidth is infinite beyond that domain, and the infinite limit is understood in this extended sense. For example, a normal distribution with mean and variance has : its unbounded upper tail has no finite peak. Thus effective bandwidth interpolates between mean and peak demand while permitting independent sources to be aggregated by addition.
Use an ideal carrier-sense multiple access scheme on the interference graph. An active station finishes its transmission after an independent unit-rate exponential distribution waiting time. An inactive station has an independent attempt clock with an exponential distribution of rate ; an attempt succeeds only when all its neighbors are inactive. Attempts that are blocked do not change the state. Thus, on the independent sets represented by , the continuous-time Markov chain has transition rates
The empty independent set can be reached by successive deactivations, and any other independent set can be reached from it by successive activations. Hence this finite chain is an irreducible Markov chain. The weight satisfies
By detailed balance for a continuous-time Markov chain, its unique stationary law is
This is a finite exponential family. Its natural parameter of an exponential family is and its mean parameter of an exponential family is the vector of active fractions.
The throughput region of an interference graph is exactly . To see the possibly less immediate inclusion, suppose a random feasible schedule has mean . Retain each of its active vertices independently with probability , interpreting zero coordinates as always deleted. A subset of an independent set remains an independent set; the thinned schedule has mean exactly . Thus lies in the convex hull of . The opposite inclusion follows from the definition using equality. Moreover , so this convex hull has full dimension.
Now choose such that remains in the interior of . Use interior moment matching in a finite exponential family. Consider the convex function
There is a ball of radius around contained in . Maximizing a linear functional over the convex hull is the same as maximizing over its generating set, so, for every ,
The last inequality uses the point when . Hence is a coercive function and has a finite minimizer . Differentiating the finite normalizing sum gives
This covariance matrix is a positive-definite matrix: a linear functional constant on all states of positive weight must be constant on , so its coefficients vanish. Thus is a strictly convex function; its minimizer is unique. At that minimizer the desired strict service margins are
For a backlogged station that transmits at unit speed whenever active, the long-run throughput is , by the ergodic theorem for a finite continuous-time Markov chain. Therefore the ideal carrier-sense multiple access scheme can supply a strict service margin for every arrival vector in the interior of the throughput region of an interference graph. The local attempt rates can realize the whole interior capacity region in this sense. Boundary points may require parameters tending to infinity, and the argument gives no uniform delay or mixing-time bound near the boundary.
In congestion control, resource observes only its own aggregate load and generates the nonnegative feedback price . User receives the sum of prices along its route. Its desired increase competes with the congestion charge , while sets the adjustment speed. This is local information: a resource needs the rates passing through it, and a user needs the feedback along its own route, rather than the complete network state.
There is a hypothesis to make explicit before claiming global convergence. If “increasing” means nondecreasing, the printed assumptions allow , in which case and there is no equilibrium point. The intended theorem holds when every route contains at least one resource whose price is not identically zero, with nonempty routes and nonnegative, continuous, nondecreasing prices. Strictly increasing prices satisfy this condition automatically. We prove the theorem under this necessary qualification; without it the zero-price example disproves the literal claim.
Let . Each is a nonnegative convex function. For positive rates define the primal congestion potential
The weighted logarithmic utility is a strictly concave function, and subtracting the resource costs preserves strict concavity. Thus has at most one maximizer and at most one critical point on the positive orthant.
It also attains its maximum there. A nonzero nondecreasing price is bounded below by some positive constant at all sufficiently large loads; hence its primitive satisfies with . Choose at zero-price resources. The route coverage assumption gives , so
This bounds all superlevel sets of . Within a bounded set, approaching makes . Consequently every nonempty superlevel set is a compact set contained in the positive orthant. A maximizing sequence stays in one such set, giving a unique interior maximizer . Its critical-point equations are
The rate dynamics are the gradient flow with diagonal mobility . Along any positive trajectory,
with equality only at . Thus is a strict Lyapunov function. The initial superlevel set traps the trajectory away from both infinity and the boundary, ensuring global continuation. Positive initial rates remain positive: variation of constants in makes this explicit. Starting with finite nonnegative rates also gives positive rates at every later positive time.
For completeness, mere continuity of suffices for convergence; one need not silently assume differentiable prices. On the trapping compact set, both the vector field and are continuous and bounded. The trajectory satisfies a uniform Lipschitz condition in time, so is uniformly continuous. Also , since increases to a finite limit. The decay of a nonnegative uniformly continuous integrable function applies: otherwise disjoint intervals of a fixed positive width would each contribute a fixed positive amount. Every accumulation point therefore has and is . Compactness then proves the global convergence
This is the dissipation argument behind the LaSalle invariance principle. If uniqueness of trajectories is desired under continuous prices, put . The transformed equation is . The transformed potential is concave: it is a sum of positive multiples of , minus convex nondecreasing composed with nonnegative quadratic sums. Its gradient is monotone decreasing, so the squared distance between two solutions cannot increase. This proves uniqueness for positive initial data without a locally Lipschitz function hypothesis on .
For the alternative price dynamics, restrict first to positive prices on resources used by at least one route, and discard unused resources. Define the dual congestion potential
Its domain has and for every route. Strictly increasing make each strictly convex, and is convex. Thus is a strictly convex function. Each is eventually bounded below by a positive constant, so its integral grows at least linearly, dominating the negative logarithms as . If a route price tends to zero in a bounded set, . Therefore attains a unique minimum on its domain including the admissible boundary faces.
That minimum has every used-resource price positive. At a face , all route prices are still positive and the inward derivative is
Hence the minimum cannot lie on that face. At the unique interior minimum,
Since the price dynamics are , their positive equilibrium points are exactly these critical points. There is one equilibrium in the positive price domain. An unused resource instead has its only nonnegative equilibrium at zero.
The positive-domain qualification matters here even with strictly increasing . If zero prices are admitted, the multiplicative factor permits boundary equilibria of multiplicative price dynamics. Take one route using two resources, , and . Both and are equilibria, with , as is the positive equilibrium , with . Thus the literal assertion of uniqueness on the whole nonnegative orthant is false. A boundary equilibrium need not satisfy load equals supplied capacity at its zero-price resources.
The relationship between the two intended equilibria is inverse resource response. The first system has ; the positive equilibrium of the second has . If on the relevant ranges, these are the same resource equations, and both systems also impose . They therefore have the same rates and prices. For strictly increasing continuous with and unbounded range, the ordinary inverses satisfy all the stated conditions on . With arbitrary independently chosen and , the equilibria need not agree. Under inverse response, and are convex conjugates on the nonnegative half-line, and supplies the Lagrangian dual problem for maximizing the weighted logarithmic utility minus resource costs, up to constants independent of .
Use the convention
This proximal map is also the resolvent of a monotone operator . A proper lower semicontinuous convex function has an affine minorant, so the quadratic term makes this minimization coercive and strongly convex. A unique minimizer exists for every . Its subgradient optimality condition is
The Moreau–Yosida regularisation is . The conjugate of an infimal convolution and the quadratic conjugate give
The factor here is essential. Apply subgradient inversion under convex conjugacy, followed by the subdifferential sum rule with the everywhere differentiable quadratic:
The last equivalence is precisely the unique proximal minimization condition. Thus
This proves both existence and uniqueness of the subgradient, rather than only identifying a possible element. The finite convex function is therefore differentiable, with , the gradient of a Moreau envelope.
For completeness, monotonicity of applied to the two proximal conditions gives
Hence is firmly nonexpansive. Expanding the same inequality shows that is firmly nonexpansive too. In particular, is -Lipschitz continuous. None of this requires a bounded effective domain; the result applies to the next example as well.
Use the extended real numbers . A proper convex function never takes and is finite somewhere. Its effective domain is
For a proper extended-real function, convexity means
The endpoint cases are identities; taking avoids the ambiguous product . Equivalently, its epigraph is a convex set. In particular, the effective domain is a convex set. Properness is an additional condition: it rules out the identically function and all values, rather than requiring finiteness everywhere.
For , the subdifferential is the set of supporting slopes
Each element is a subgradient. Define outside . The defining subgradient inequality must hold globally, not merely in a neighborhood of . For a differentiable finite convex function, .
A firmly nonexpansive mapping satisfies
Writing and , this is equivalent to
It is also equivalent to , so is a nonexpansive mapping. Consequently firm nonexpansiveness is exactly -averagedness. A proximal map of a proper lower semicontinuous convex function is a basic example. A -averaged operator is also -averaged for every , by combining its nonexpansive part with the identity; the equivalence with firm nonexpansiveness specifically uses the parameter .
Here -smooth means that is differentiable on the entire Euclidean space and has an -Lipschitz gradient, with . Together with convexity, this implies -cocoercivity:
First, integration of the Lipschitz gradient along a line segment proves the descent lemma:
Indeed, subtract the linear term and integrate for ; its absolute value is at most .
Fix and define . This is a convex function, has the same Lipschitz gradient constant, and , so is a global minimizer. Put . The descent lemma at with gives
Therefore
Interchange and add the inequalities. The function values cancel, leaving the asserted cocoercivity. This is a proof of the Baillon–Haddad theorem without assuming a second derivative. Convexity is essential: has a -Lipschitz gradient but its gradient is not cocoercive.
Set . Completing the square in the -minimization gives
Thus, with , the steps in terms of the proximal map of are
The factor comes from the squared residual without a prefactor. The proximal map can equivalently be denoted . Since is a finite convex function, it is continuous, and the quadratic makes the proximal minimizer unique.
A general convex may be nondifferentiable, so this last example need not satisfy the full -smooth hypothesis of the preceding part. The formulas nevertheless remain well-defined: only the derivative is used explicitly. If has an -Lipschitz gradient, one may take in the smooth proof, where is the operator norm.
There is also a useful nonsmooth extension. Put , whose Lipschitz gradient constant is . For two updates, retain the mixed-point differences above, let , and let be the difference of the two selected subgradients of at . The input difference is and the output difference is . Monotonicity of gives . The same norm expansion now yields
For , discard the nonpositive term and use . The result is the firmly nonexpansive mapping inequality for the full residual . Fixed points satisfy and , exactly the subgradient optimality condition for the convex objective. Thus the stated algorithm also converges for nonsmooth under this sufficient step bound, provided a minimizer exists; this extension uses monotonicity, not an unavailable gradient of .
Write the objective as with . Each hinge loss is finite and convex, so the subdifferential sum rule applies without a qualification gap. Its subdifferential is
This formula also covers , if such a sample is allowed, because the loss is then constant. For a convex function, zero in the subdifferential is necessary and sufficient for global optimality. Thus is optimal exactly when there are weights such that
Equivalently, using and ,
The notation denotes the segment between and , also when . This avoids reversing interval endpoints for negative-class samples.
Because , the objective is strongly convex, and makes it coercive. It is continuous, so a minimizer exists and is unique for every finite data vector . The weights at exact margins need not be unique even though is. These are the hinge-loss optimality weights for the one-dimensional support vector machine.
Take with the usual proper closed convex assumptions, so the Fenchel-Moreau theorem identifies its conjugate with the supplied . Put . The first update is . By Moreau decomposition, the second update is
Also , so the argument in the final update contains . Eliminating the auxiliary variable gives
This is the primal-dual hybrid gradient method with extrapolation parameter one and the primal update performed first. Its saddle point function is , corresponding to the primal objective .
For proper lower semicontinuous convex functions and a nonempty saddle-point set in these finite-dimensional spaces, the standard convergence of primal-dual hybrid gradient theorem gives the sufficient parameter condition
Here is the operator norm, or largest singular value. For , one possible choice is with ; for any positive steps satisfy the condition. The existence assumption is necessary: step sizes alone cannot guarantee convergence to a saddle point that does not exist.