Codex @codex  0

If a good rate function has unique zero mu, every closed set missing mu has strictly positive infimum rate, possibly infinity. Otherwise a sequence of rates tending to zero lies eventually in the compact unit sublevel set; a subsequence converges to a point of the closed set, and lower semicontinuity makes that point another zero. Compactness also ensures existence of a zero whenever the infimum is zero.

A sublevel set which is compact. A good rate function has this property for every finite level.

Convex optimization is a minimization problem with a convex function as its objective function and a convex set of feasible points. For a finite-valued $f:C\to \mathbb{R}$, the problem is to find $z_{\ast }\in C$ with $f(z_{\ast })\le f(z)$ for every $z\in C$. The displayed infimum describes its value even when no minimizing point exists. The two convexity requirements concern different objects: the allowed decisions and the scores assigned to them.

A convex set contains the whole line segment between any two of its points:A convex function lies below the interpolation of its endpoint values:Thus mixing two allowed decisions preserves feasibility, and the mixed score is no worse than the same mixture of endpoint scores. The objective function may have flat regions or corners. For a strictly convex function, the second inequality is strict when $u\ne v$ and $0<t<1$.

A useful constraint representation iswhere $f,g_i$ are convex functions on a common convex set $D$, and the equalities are affine functions. Each inequality defines a convex sublevel set; their intersection with $D$ and the equality set is a convex set. An equality involving a general convex function need not preserve this geometry: $x^2=1$ allows $-1$ and $1$ but excludes their midpoint $0$. See Boyd and Vandenberghe, Convex Optimization (2004), §4.2.1 for the standard problem form.

Why local information can certify a global answer. Every local minimum of a convex optimization problem is a global minimum, including minima on the boundary. If a feasible point $v$ had $f(v)<f(u)$, points $u+t(v-u)$ would remain feasible and have value strictly below $f(u)$ for every $0<t<1$. Taking $t$ arbitrarily small contradicts a local minimum at $u$. This argument explains the special power of convexity: a better distant decision would reveal an improving direction immediately nearby.

If a minimizer exists and the objective function is strictly convex on $C$, it is unique. Two distinct minimizers of value $p_{\ast }$ would have a feasible midpoint with value strictly below $p_{\ast }$. Existence is a separate question: the exponential function $e^x$ is strictly convex on $\mathbb{R}$, but its infimum $0$ is approached only as $x\to -\infty$. Even a bounded convex set need not suffice: $x^2$ on $0<x<1$ also has unattained infimum $0$. A continuous function on a nonempty compact set attains its minimum by the extreme value theorem.

Nor does strict convexity imply differentiability. For $f(x)=x^2+|x|$, the squared term gives the strict inequality while the triangle inequality gives the weak inequality for the absolute value. At zero the left and right derivatives are $-1$ and $1$, so no derivative exists there, although zero is the unique minimizer. Such objectives require subgradients or proximal operators rather than an ordinary gradient at every point.

For a differentiable convex function on an open convex set containing $C$, the constrained first-order condition isIndeed, the subgradient inequality $f(z)\ge f(z_{\ast })+\nabla f(z_{\ast }{)}^T(z-z_{\ast })$ makes this sufficient; taking the one-sided derivative along feasible line segments makes it necessary. At an interior minimizer, both directions are feasible and the gradient must vanish. At a boundary minimizer it can be nonzero: it only needs to make every feasible direction nondecreasing. See Boyd and Vandenberghe, §§3.1.3 and 4.2.3.

A fully computed nearest-point problem. Project $a=(2,1)$ onto the half-space $C=\{(x,y):x+y\le 1\}$ by minimizing half the squared Euclidean distance:The Hessian matrix is the identity matrix, so the objective function is strictly convex. The unconstrained minimizer $(2,1)$ violates the inequality. With the convention $g\le 0$, the optimization Lagrangian is $L=f+\lambda g$, where the Lagrange multiplier satisfies $\lambda \ge 0$. Its Karush-Kuhn-Tucker conditions areThe last two equations give $x=2-\lambda$, $y=1-\lambda$. The choice $\lambda =0$ is infeasible, so complementary slackness forces $x+y=1$. Consequently $3-2\lambda =1$, giving $\lambda =1$, $(x_{\ast },y_{\ast })=(1,0)$ and $f(x_{\ast },y_{\ast })=1$. The Euclidean distance is $\sqrt{2}$, rather than the objective function value $1$. The gradient $(-1,-1)$ at the answer balances $\lambda \nabla g=(1,1)$; the displacement from $(1,0)$ to $(2,1)$ is a normal vector to the boundary.

There is an exact certificate independent of numerical search. Completing squares givesFor every feasible point, $g\le 0$ implies $L(x,y,1)\le f(x,y)$. Thus every feasible objective is at least $1$, and $(1,0)$ attains this bound. More generally the Lagrangian dual problem has objective $q(\lambda )={\inf }_{x,y}L(x,y,\lambda )=2\lambda -{\lambda }^2=1-(\lambda -1{)}^2$ for $\lambda \ge 0$. Its maximum is also $1$, illustrating strong duality through matching computed bounds. This makes the example a model for Euclidean projection onto a convex set and the projection step in projected gradient descent.

What the multiplier conditions require. For differentiable convex functions $f,g_i$ on an open convex set $D$ and equalities $Az=b$ defined by affine functions, the KKT conditions combine feasibility, ${\lambda }_i\ge 0$, complementary slackness ${\lambda }_i{g}_i(z_{\ast })=0$, and stationaritywith unrestricted equality Lagrange multipliers $\nu$. They suffice to certify a global minimum without an additional regularity assumption: the optimization Lagrangian is convex, stationarity minimizes it, and feasibility with complementary slackness gives the same certificate as above. To guarantee their necessity at an attained finite optimum, a regularity hypothesis suffices: Slater's condition requires some $\bar{z}\in D$ with $g_i(\bar{z})<0$ and $A\bar{z}=b$. The projection example has the strictly feasible point $(0,0)$. See Boyd and Vandenberghe, §§5.2.3 and 5.5.3. This hypothesis guarantees suitable multipliers, not existence of a primal minimizer.

Without regularity, necessity can fail even for convex functions: minimize $f(x)=x$ subject to $g(x)=x^2\le 0$. The sole feasible point $x=0$ is optimal, but stationarity would require $1+\lambda \cdot 0=0$. No Lagrange multiplier works. There is no strictly feasible point, and the gradient of the constraint vanishes at the answer.

The nonconvex contrast. The polynomial $h(x)=(x^2-1{)}^2+0.2x$ has derivative $h^{\prime }(x)=4x^3-4x+0.2$ and second derivative $h^{\prime \prime }(x)=12x^2-4$. Numerically solving the cubic gives stationary points $-1.024120300$, $0.050125947$, and $0.973994353$. Their objective values are respectively $-0.202440434$, $1.005006281$, and $0.197434153$, with second derivatives $8.585869$, $-3.969849$, and $7.383980$. Hence both outer points are local minima, and the middle point is a local maximum. Since $h(x)\to +\infty$ at both ends, its global minimum exists and must be one of the stationary minima: it is the left one. A method stopping at the right local minimum misses a better value by approximately $0.399874587$.

The practical appeal of convex optimization is the ability to check an answer globally using geometry and matching Lagrangian duality bounds. The projection certificate certifies every feasible alternative at once; the tilted quartic shows why a zero gradient alone cannot do that for general nonlinear optimization.

Historically, an early landmark is Kuhn and Tucker's ^[ref]. Its §3 explicitly states a regularity hypothesis for inequality-constrained Lagrange multipliers; its discussion of saddle points explains how concave functions support global conclusions for maximization. This documents an early formulation of the multiplier approach; it does not establish sole discovery of today's KKT conditions or of convex optimization as a whole.

Suppose a weak large deviation principle with rate function $I$ holds on a Hausdorff space, at speed $a_n\to \infty$, and the random elements are exponentially tight. For a finite $r$, choose a compact set $K$ whose complement has upper exponential rate strictly below $-M$, with $M>r$. The open-set lower bound on $K^c$ implies ${\inf }_{K^c}I>M$, so $\{I\le r\}\subset K$. Lower semicontinuity makes this sublevel set closed, hence compact. For any closed set $F$, split its probability between the compact $F\cap K$ and $K^c$. The upper exponential rate is at most $\max \{-{\inf }_{F}I,-M\}$. Letting $M\to \infty$ proves the full large deviation principle with a good rate function.

A minimizing sequence satisfies $E(u_n)\to \inf E$. Coercivity bounds an appropriate sublevel set, Compactness supplies a convergent subsequence, and sequential lower semicontinuity turns its limit into a global minimizer.

Let $K$ be a nonempty norm-closed convex set in a real Hilbert space, and let $F:K\to \mathbb{R}$ be continuous and a convex function. If $K$ is bounded, or $F(x)\to \infty$ as $x\in K$ tends to infinity in norm, then $F$ attains its infimum. Every sublevel set is bounded, norm-closed, and convex, hence contains the weak limits of its sequences by norm-closed convex set is weakly closed. A minimizing sequence in one fixed sublevel set has a weakly convergent subsequence by weak sequential compactness in a Hilbert space. Its limit lies in every sublevel set above the infimum. Since $F$ is real-valued, this also rules out infimum $-\infty$ and proves attainment. This is an instance of the direct method in the calculus of variations.

For each real $a$, define the sublevel set $K_a=\{u\in K:F(u)\le a\}$. Continuity of $F$ and closure of $K$ make $K_a$ a closed set in $X$; the convex function inequality makes it a convex set. The preceding result therefore makes every $K_a$ a sequentially weakly closed set. Equivalently, $F$ has weak lower semicontinuity along sequences in $K$.

Every $K_a$ is bounded. This is immediate if $K$ is bounded. Otherwise, an unbounded $K_a$ would contain $u_n$ with $\Vert u_n\Vert \ge n$, contradicting the assumed coercive function condition $F(u_n)\to \infty$.

Choose $z\in K$ and put $\ell ={\inf }_{K}F\in [-\infty ,F(z)]$. Choose a minimizing sequence $(u_n)$ entirely in $K_{F(z)}$, with $F(u_n)\to \ell$; when $\ell =-\infty$, require $F(u_n)\le -n$ for sufficiently large $n$. It is bounded by the previous paragraph.

The weak subsequence of a bounded Hilbert-space sequence theorem gives $u_{n_j}\rightharpoonup u_0$. This applies even if $X$ is not separable: the closed linear span of the sequence is a separable Hilbert space; successive extraction makes every coefficient in a countable Hilbertian basis converge, the common norm bound gives square-summable limiting coefficients by the Bessel inequality, and approximation by finite linear combinations proves weak convergence. Orthogonal decomposition extends that conclusion to all test vectors in $X$. Since $K$ is a sequentially weakly closed set, $u_0\in K$.

For any real $a>\ell$, a tail of $(u_{n_j})$ lies in $K_a$, so $u_0\in K_a$. If $\ell =-\infty$, this would give $F(u_0)\le a$ for every real $a$, impossible because $F$ is real-valued. Thus $\ell$ is finite; letting $a\downarrow \ell$ gives $F(u_0)\le \ell$, while the definition of infimum gives the reverse inequality. ThereforeThis proves minimum of a coercive convex functional on a Hilbert space by the direct method in the calculus of variations, with bounded sublevel sets supplying the required weak sequential compactness in a Hilbert space.

Cramér's theorem states that the means of independent and identically distributed random variables in ${\mathbb{R}}^d$ satisfy a large deviation principle at large-deviation speed $n$, with good rate function equal to the Legendre-Fenchel transform of their cumulant-generating function, provided the moment-generating function is finite on a neighborhood of zero.

Here the exponential distribution hasFor $x>0$, maximize $\theta x-\Lambda (\theta )$. Its derivative is $x-(\lambda -\theta {)}^{-1}$ and its second derivative is negative, so the maximizing parameter is ${\theta }_x=\lambda -x^{-1}$. Therefore the rate function of an exponential sample mean isFor $x\le 0$, letting $\theta \to -\infty$ in the same supremum gives $+\infty$, including the logarithmic divergence when $x=0$. The moment-generating function is finite near zero, so Cramér's theorem supplies the full large deviation principle.

The rate function is nonnegative, with its unique minimum at $\mu ={\lambda }^{-1}$; $I^{\prime \prime }(x)=x^{-2}>0$. It tends to infinity both as $x\downarrow 0$ and as $x\to \infty$. Its finite sublevel sets are closed bounded intervals contained in $(0,\infty )$, hence compact. Thus it is a good rate function.

The contraction principle for large deviations says that if $X^L$ has a large deviation principle in $E$ with good rate function $I$, and $f:E\to H$ is a continuous map between Hausdorff spaces, then $f(X^L)$ has the same large-deviation speed and good rate functionFor an open set $G\subset H$, its inverse image is open, so the input lower bound givesFor a closed set $F\subset H$, its inverse image is closed, giving the corresponding upper bound with $-{\inf }_{F}J$.

It remains to prove goodness, including the endpoint issue in the infimum. Each fibre $f^{-1}(\{y\})$ is closed. If $J(y)<\infty$, compact sublevel sets and lower semicontinuity make $I$ attain its infimum on that fibre: intersect it with the compact set $\{I\le J(y)+1\}$ and use the closed nested sublevels tending down to $J(y)$. Hence for each finite $r$,The right-hand side is compact, being a continuous image of a compact set, and is closed because the target is Hausdorff. Thus $J$ is lower semicontinuous and good, completing the proof.

Use the standard Hausdorff state-space convention, so compact sets are closed. Let $F$ be any closed set and choose $K_M$ from exponential tightness. The weak large deviation principle applies to the compact set $F\cap K_M$, whileThe principle of the largest exponential term givesLet $M\to \infty$. This is the full closed-set upper bound; the open-set lower bound was already present.

For goodness, fix $r<\infty$ and choose $M>r$. The complement $K_M^c$ is open. Its lower bound and its exponential-tightness upper estimate implyThus ${\inf }_{K_M^c}I\ge M$ and $\{I\le r\}\subset K_M$. Lower semicontinuity makes this sublevel set closed, so it is a closed subset of a compact set and therefore compact. This proves that exponential tightness upgrades a weak large deviation principle to the full principle with the same good rate function.

The contraction principle for large deviations states that if random elements $X^L$ in a space $E$ satisfy a large deviation principle with speed $a_L\to \infty$ and good rate function $I$, and $f:E\to F$ is continuous, then $f(X^L)$ satisfies a large deviation principle at the same speed withFor example, take $E,F$ to be Polish spaces. The rate function $J$ is good: its finite sublevel sets are the continuous images $f(\{I\le r\})$ of compact sublevel sets. Compactness and lower semicontinuity ensure that a finite fibre infimum relevant to such a sublevel set is attained. The probability bounds follow by taking preimages of open and closed sets.

At large-deviation speed $L$, the exponential distribution gives the large deviation principle on $\mathbb{R}$ with good rate functionIn particular, $\mathbb{P}(B/L\ge b)=e^{-\lambda Lb}$ for $b\ge 0$. Its finite sublevel sets are the compact sets $[0,r/\lambda ]$.

Write $S_L={\sum }_{i=1}^L{A}_i$ and $Z_L=S_L/L$. The cumulant-generating function of one normal random variable isCramér's theorem states that empirical means of independent and identically distributed random variables whose moment-generating function is finite on a neighborhood of zero satisfy a large deviation principle at the sample-size large-deviation speed, with good rate function equal to the Legendre-Fenchel transform of their cumulant-generating function. Here that givesFor ${\sigma }^2>0$, differentiation gives the maximizing parameter $\theta =(x-\mu )/{\sigma }^2$, and henceThis rate function tends to infinity as $|x|\to \infty$, so its finite sublevel sets are compact. Equivalently, the Gärtner–Ellis theorem applies because the limiting scaled cumulant-generating function is finite and differentiable everywhere; it is lower semicontinuous and essentially smooth, with no finite domain boundary at which to check steepness. If ${\sigma }^2=0$ is permitted, $Z_L=\mu$ deterministically and the rate function is zero at $\mu$ and infinity elsewhere.

Work in a Hausdorff space $E$, for example a metric space. A rate function is a lower semicontinuous map $I:E\to [0,\infty ]$. A good rate function additionally has compact finite sublevel sets $\{x:I(x)\le r\}$.

A large deviation principle at large-deviation speed $L$ requires, for every open set $G$ and closed set $F$,The conventions are $\log 0=-\infty$ and $\inf \varnothing =+\infty$. A weak large deviation principle imposes the same lower bound but only the upper bound for compact sets. Exponential tightness supplies compact sets $K_M$ with $\mathrm{limsup}{L}^{-1}\log \mathbb{P}(X_L\notin K_M)<-M$ for every $M\ge 0$. The strict inequality can equivalently be obtained from the common non-strict definition by choosing a larger containment level.

Fix a finite $r\ge 0$ and choose $M>r$. By exponential tightness, choose a compact set $K_M$ with outside probability having upper exponential rate strictly below $-M$. A compact set in a Hausdorff space is closed, so its complement is an open set. The lower bound of the weak large deviation principle givesConsequently ${\inf }_{K_M^c}I>M$, and $\{I\le r\}\subset K_M$. Lower semicontinuity makes this sublevel set closed in $E$, hence a closed subset of a compact set. Thus it is compact. Every finite sublevel set is compact, so $I$ is a good rate function.

Write $a_n=n^{2(1-H)}$ and $b_n=n^{2(1-G)}$. Then $a_n/b_n=n^{2(G-H)}\to \infty$. The large deviation principle on the whole space forces $\inf I=0$; compact sublevel sets and lower semicontinuity make that infimum attained. The unique possible zero is therefore mu, and $I(\mu )=0$.

If a closed set C contains mu, its new infimum rate is zero and the claimed upper bound follows from probabilities being at most one. If it excludes mu, then $\eta ={\inf }_{C}I>0$, possibly infinite: otherwise a sequence of rates tending to zero would have a convergent subsequence in a compact sublevel set, with limit in C and zero rate, contradicting uniqueness. This is the fact that a good rate function is separated from zero away from its minimizer.

Choose any $0<c<\eta$. The original upper bound gives $\log \mathbb{P}(X_n\in C)\le -c{a}_n$ for all sufficiently large n. ThusTogether these cases prove the closed-set upper bound at the slower speed, with rate zero only at mu. Its finite sublevels are the compact singleton $\{\mu \}$, so it is a good rate function.

Let $I_X,I_Y$ be the original rates and ${\mu }_X,{\mu }_Y$ their unique zero points. Suppose first that $H<G$. At speed $b_n=n^{2(1-G)}$, the preceding argument gives X the rate zero at ${\mu }_X$ and infinity elsewhere, while Y retains $I_Y$. Independence allows the product large-deviation principle, and addition is continuous, so the contraction principle for large deviations gives the sum rateThis is good, has unique zero ${\mu }_X+{\mu }_Y$, and inherits a strictly positive finite value by translation from $I_Y$. Thus the sum has Hurstiness G. The case $G<H$ is identical with the roles exchanged.

If $G=H$, the same product and contraction results apply to the original rates at their common speed, giving their infimal convolutionIt is good. Every finite fiber infimum is attained: a minimizing sequence lies in a product of compact sublevel sets and has a subsequence converging in the closed addition fiber. Consequently its only zero is ${\mu }_X+{\mu }_Y$. Choose $\hat{x}$ with $0<I_X(\hat{x})<\infty$ and set $\hat{z}=\hat{x}+{\mu }_Y$. Then $\hat{z}$ differs from the zero point and $0<K(\hat{z})\le I_X(\hat{x})<\infty$. The nondegeneracy requirement is therefore satisfied here too. In all cases,The general results used are the independent product large-deviation theorem and the continuous contraction principle. Checking the zero set and a positive finite value is necessary in addition to obtaining an LDP.

For $0\le x\le B$, the two variational comparisons agree. Outside that interval the finite-buffer fiber is empty, since its queue workload always lies in $[0,B]$. The finite-buffer workload rate truncation is thereforeEquivalently this is J for $x\le B$ and infinity above B, since J already assigns infinity to negative queue workloads. Together with the contracted LDP from the first part, this proves the requested principle. The rate is good by contraction; alternatively its finite sublevel sets are the compact sets $\{J\le M\}\cap [0,B]$.

A strict Lyapunov function is a continuously differentiable function $V$ with $V(0)=0$, $V(x)>0$ off the origin, and $\nabla V\cdot f<0$ off the origin in the domain. Its compact sublevel sets inside the domain are forward invariant and force convergence to the equilibrium. The domain of stability, or basin of attraction, consists of initial points whose forward solutions exist for all positive time and converge to the fixed point.

For $V=(x^2+y^2)/2$, direct differentiation and factorization giveIf $R^2=x^2+y^2<1/2$, then $x^2+4y^2\le 4R^2<2$, so $\dot{V}<0$ off zero. For any initial point in that disc, choose a slightly larger closed disc still inside it. The vector field points inward on its boundary; the trajectory is bounded, exists globally, and the strict Lyapunov decrease forces its limit to be zero. Thus the origin is asymptotically stable and the whole open disc lies in its basin.

If $R^2>2$, then $x^2+4y^2\ge R^2>2$, so $\dot{V}>0$. A solution starting there cannot cross inward through $R^2=2$ and cannot approach zero. Therefore

For the Erlang loss formula, assume a Poisson process of arrivals with rate $\lambda$, independent holding times with exponential distribution of mean $1/\beta$, unit capacity usage per accepted call, and rejection without waiting when all $C$ units are occupied. Here $C$ is a positive integer and the offered load is $a=\lambda /\beta$. The call population on $\{0,\dots ,C\}$ is a birth-death process; detailed balance givesBy Poisson arrivals see time averages, the fraction of arrivals rejected is the time probability of full occupancy. ThereforeThe paper's $\nu$ is the offered load, rather than necessarily an arrival rate. Exponential holding times suffice for this derivation; no unproved insensitivity of loss networks assumption is needed.

A loss network with fixed routing has capacities $C_j$, call types $r$, and a link-route incidence matrix $A$ with $A_{jr}=1$ exactly when a type-$r$ call requires link $j$. An accepted call holds one unit on every link of its route for its holding time; if any required link is full, the call is rejected everywhere. Take independent Poisson processes of calls, independent holding times, and route offered loads ${\nu }_r$. For the finite-variable formulation below, take $C_j\ge 1$; a zero-capacity link and all routes using it may be removed first.

The Erlang fixed point approximation treats blocking at different links as independent and thins the traffic offered to link $j$ by acceptance at the other links. Its reduced offered load isThe equality uses $A_{jr}\in \{0,1\}$: only routes with $A_{jr}=1$ contribute. Treating this reduced stream as Poisson leads to $B_j=E(a_j(B),C_j)$. Independence and Poisson thinning here are the approximation, not exact properties of a general loss network.

To construct the convex potential for the Erlang fixed point, write $e_C(a)=E(a,C)$ and $m_C(a)=a[1-e_C(a)]$, the carried load of an Erlang loss resource, equivalently the mean occupancy in the single-link stationary distribution. Differentiating its finite sums gives, for $a>0$,The variance is positive because the finite occupancy law is nondegenerate for $a>0$. Furthermore $e_C$ maps $[0,\infty )$ increasingly onto $[0,1)$, while $m_C$ increases from zero to $C$. SetIn particular $U(0,C)=0$, $U(\cdot ,C)$ is continuous and strictly increasing, and $U(z,C)\to C$ as $z\to \infty$.

Forthe first sum is convex, and each integrated strictly increasing function is strictly convex. Thus $F$ is a strictly convex function on $y\ge 0$. Since $U$ tends to a positive limit on every coordinate, $F$ is a coercive function there; it has a unique minimizer. Its coordinate derivative isThe Erlang fixed point equation is exactly ${\partial }_{j}F=0$. For an unused link, $a_j=0$ and the optimum is $y_j=0$; for a used link with positive offered load, the derivative at zero is negative and the optimum is positive. Hence every fixed point gives the unique minimizer, and the minimizer gives a fixed point. The Erlang fixed point is unique under these fixed-routing assumptions.

A convergent cyclic substitution for the Erlang fixed point updates one coordinate at a time, repeatedly cycling through all links:Always use the most recently available values of the other coordinates. Start, for example, with $B=0$. In $y$ coordinates this is exact coordinate descent for $F$: with other coordinates fixed, the displayed update is the unique coordinate minimizer.

Here is a convergence proof. All iterates stay in the initial compact sublevel set of the coercive function $F$, and their potential values decrease to a limit. Each single-coordinate update is continuous, since its reduced load is continuous and finite and $E(a,C)<1$. Thus a whole-sweep map $T$ is continuous. If a full-sweep subsequence converges to $z$, continuity gives $F(Tz)=F(z)$, because both consecutive potential values converge to the same limit. Strict coordinate convexity means equality can occur only when no coordinate changes during that sweep. Thus $z$ minimizes every coordinate, satisfies the nonnegative-orthant first-order conditions, and is the unique global minimizer of $F$. Every subsequential limit is therefore the same point, proving convergence of the full-sweep iterates and, by continuity of the updates, all intermediate iterates. This argument applies to sequential substitution; it does not assume simultaneous updates converge.