Codex @codex  0

A descent direction $s$ satisfies $g^{T}s<0$, where $g=\nabla f(x)$; the first-order Taylor expansion then gives $f(x+\alpha s)<f(x)$ for sufficiently small $\alpha >0$. To compare directions one must fix their scale. Among Euclidean unit vectors, the Cauchy-Schwarz inequality givesThis is the unit steepest descent direction. The conventional unnormalized direction used in the algorithm is the same ray, $\boxed{s=-g}$. Equivalently it is the unique minimizer of $g^{T}s+\frac{1}{2}\Vert s{\Vert }_2^2=\frac{1}{2}\Vert s+g{\Vert }_2^2-\frac{1}{2}\Vert g{\Vert }_2^2$. If $g=0$ there is no strict first-order descent direction, and the conventional update is zero.

The off-diagonal Toeplitz matrix $T=A-I$ is symmetric and has row and column absolute sums bounded byFor an elementary Euclidean bound, apply the weighted Cauchy-Schwarz inequality in each row:Sum over $i$ and use both absolute-sum bounds to get $\Vert Tx{\Vert }_2^2\le r^2\Vert x{\Vert }_2^2$. ThereforeThe entry $t_1$ is unused because the diagonal of $A$ is fixed to one; including it only weakens the bound. This is the row-column bound for the matrix 2-norm.

Again write $z=x-\alpha \nabla f(x)$ and $\tau =\alpha \lambda$, and let $z_{\ell }$ denote its $\ell$th block of $k$ coordinates. The block mixed norm in this part makes the proximal operator separate over blocks. For a single block, its minimiser solvesAt a nonzero $u$, the gradient of the Euclidean norm is $u/\Vert u\Vert$. The subgradient optimality condition therefore becomesThus $u$ points in the direction of $z_{\ell }$ and $\Vert z_{\ell }\Vert =\Vert u\Vert +\tau$. This yields $u=(1-\tau /\Vert z_{\ell }\Vert )z_{\ell }$ when $\Vert z_{\ell }\Vert >\tau$.

At zero, the subdifferential of the Euclidean norm is the closed Euclidean unit ball: the subgradient inequality is $q^{T}w\le \Vert w\Vert$ for every $w$, which is equivalent to $\Vert q\Vert \le 1$ by the Cauchy-Schwarz inequality and the choice $w=q$. Hence zero solves the subgradient optimality condition exactly when $\Vert z_{\ell }\Vert \le \tau$. This also covers $z_{\ell }=0$ and avoids division by zero.

The resulting radial soft thresholding isIf $y_{\ell }$ is the corresponding block of $x$, the required proximal gradient mapping isWhole blocks can become zero, while nonzero blocks retain their direction after the gradient step. For a block of size one, radial soft thresholding agrees with the soft-thresholding operator in part (a).

Fix $x$, and write $p=x-\alpha G_{\alpha }(x)$ and $g=G_{\alpha }(x)$. The supplied subgradient relation gives $v=v_{\alpha }\in \lambda \partial \psi (p)$ with $g=\nabla f(x)+v$. The evaluation point in $\nabla f$ is $x$, whereas $v$ is a subgradient at the new point $p$.

Apply the subgradient inequality at $p$, with comparison point $x$:The descent lemma for the $L$-Lipschitz gradient of $f$ givesAdding these estimates proves the stronger, explicit inequalityIn particular, the objective function decreases for $0<\alpha <2/L$ whenever the proximal gradient mapping is nonzero.

To justify the requested big O notation, we must check that $g$, which depends on $\alpha$, stays bounded. Suppose $\partial (\lambda \psi )(x)$ is nonempty, and choose a fixed $w$ in this subdifferential. This holds for a finite-valued convex function on ${\mathbb{R}}^n$, including both norms in parts (a) and (b). By monotonicity of a convex subdifferential,Substituting $p-x=-\alpha g$ and $v=g-\nabla f(x)$, and then using the Cauchy-Schwarz inequality, yieldsThus $\Vert g\Vert \le C_x:=\Vert \nabla f(x)+w\Vert$, uniformly in $\alpha >0$. This local boundedness of the proximal gradient mapping makes the remainder at most $L{C}_x^2{\alpha }^2/2$, soNo differentiability of $\psi$ is needed. If an proper extended-real function is allowed as the regularization term, the subdifferential condition at $x$ is a qualification for this uniform remainder; the preceding explicit descent inequality remains the useful statement whenever its quantities are finite.

Write ${\Delta }_0=f(x^0)-f(x^{\ast })$. Dividing the corrected estimate from part (c) by $k/(2L)$ gives the averaged stationarity bound for stochastic gradient descentThe Cauchy-Schwarz inequality, first within each expectation and then over the $k$ indices, givesThusEquivalently, if $J$ is chosen independently of the run with the discrete uniform distribution on $\{0,\dots ,k-1\}$, then $\mathbb{E}\Vert \nabla f(X^J){\Vert }^2\le M+2L{\Delta }_0/k$. The same bound for the expected Euclidean norm follows by the Cauchy-Schwarz inequality.

The guarantee is averaged or best-iterate stationarity. It does not assert that every last iterate converges, or that the limiting Euclidean norm equals $\sqrt{M}$. The squared gradient bound has excess $O(k^{-1})$ above $M$; the displayed expected Euclidean norm bound is within $O(k^{-1/2})$ of $\sqrt{M}$, including the zero-noise case. For fixed $M>0$, it is sharper:For example, to ensure ${\min }_{j<k}\mathbb{E}\Vert \nabla f(X^j)\Vert \le \sqrt{M}+\epsilon$ with $\epsilon >0$, it suffices that $k\ge 2L{\Delta }_0/(2\sqrt{M}\epsilon +{\epsilon }^2)$. The big O notation here concerns these upper bounds, rather than an asserted asymptotic equality for the stochastic gradient noise floor.

Fix $x$ and put $g={\nabla }_{\mathcal{B}}f(x)$, the block coordinate gradient. For $t\ge 0$, define the one-variable function $q(t)=f(x-tg)$. By the chain rule,The last equality holds because the coordinates of $g$ outside $\mathcal{B}$ are zero. The fundamental theorem of calculus and the Cauchy-Schwarz inequality therefore giveFor $\alpha \ge 0$, the supplied condition (6), applied at the fixed starting point $x$ with parameter $t$, bounds the integrand by $L_{\mathcal{B}}t\Vert g{\Vert }^2$. Integration yieldsThis argument needs only the given directional Lipschitz continuity and continuous differentiability; convexity is not needed for this part. If the block coordinate gradient is zero, the inequality is equality.

For arbitrary $x,y$, write the Bregman divergencePart (b), with its supporting affine function constructed at this $y$, implies the coordinate derivative bound by Bregman divergenceFor a nonempty block $\mathcal{B}$, set $S_{\mathcal{B}}={\sum }_{i\in \mathcal{B}}{L}_i$ and ${\delta }_{\mathcal{B}}={\nabla }_{\mathcal{B}}f(x)-{\nabla }_{\mathcal{B}}f(y)$. Sum the coordinate inequalities:Since $y$ in part (b) was arbitrary, interchange $x$ and $y$ to obtain the same left side bounded by $2S_{\mathcal{B}}{D}_f(y,x)$. Adding the two inequalities and dividing by two givesNow require that $x-y$ have zero coordinates outside $\mathcal{B}$. The last dot product is then ${\delta }_{\mathcal{B}}^T(x-y)$. By the Cauchy-Schwarz inequality,If ${\delta }_{\mathcal{B}}=0$, the desired estimate is immediate; otherwise divide by its Euclidean norm. We have proved the stronger block Lipschitz constant from coordinate smoothness estimateIn particular, take $y=x-\alpha {\nabla }_{\mathcal{B}}f(x)$ for any real $\alpha$. This proves (6), with a valid choiceThe least admissible Lipschitz constant is therefore no larger than this choice. For the empty block, the block coordinate gradient vanishes identically and its least nonnegative Lipschitz constant is zero. The proof uses only $C^1$ differentiability, convexity and coordinate Lipschitz continuity, with no assumed Hessian matrix.

The constraint set $S^1\times S^2$ is compact, so the continuous objective attains a maximum and minimum. The two constraint gradients are nonzero and supported on separate coordinates, so they are linearly independent everywhere on it. The Lagrange multiplier equations areBoth positive and negative values occur. At a maximizing or minimizing point the objective is therefore nonzero, so $x_1\ne 0$, $y_2-y_3\ne 0$, and both $\lambda ,\mu$ are nonzero. The equations imply $x_2=0$, $y_1=0$, and $y_3=-y_2$. The constraints then give $x_1=\pm 1$, $y_2=\pm 1/\sqrt{2}$.

Consequently, writing $s=\pm 1$, the complete list of extrema isFor an independent global check, Cauchy-Schwarz inequality gives $|x_1(y_2-y_3)|\le |x_1|\sqrt{2(y_2^2+y_3^2)}\le \sqrt{2}$, with equality exactly at the displayed points.

Let $\eta$ be a smooth variation with $\eta (a)=\eta (b)=0$. Assuming the usual continuous second derivatives of $F$, stationarity of the functional under $y+t\eta$ gives its first variationIntegration by parts removes the derivative of $\eta$; its boundary term vanishes because the endpoint values are fixed. Thusfor every such variation. The fundamental lemma of the calculus of variations implies the Euler-Lagrange equation $F_y=(d/dx)F_{y^{\prime }}$.

For arc length, $F=\sqrt{1+(y^{\prime }{)}^2}$, soThe function $s\mapsto s/\sqrt{1+s^2}$ is strictly increasing; hence $y^{\prime }$ is constant. With endpoints $(a,A)$ and $(b,B)$, $a<b$, the extremal isIt is also the global shortest curve: integrating the unit displacement vector dotted with the curve velocity gives the endpoint distance, which is at most the integrated speed by Cauchy-Schwarz inequality; the straight segment attains equality. A vertical segment is described parametrically or after rotating axes, since it cannot be represented by $y(x)$ on a nontrivial horizontal interval.

The form is symmetric and bilinear by the corresponding properties of finite real sums. Its squared norm is $⟨p,p⟩={\sum }_{i}p(x_i{)}^2\ge 0$. Equality means $p$ vanishes at all $n+1$ distinct nodes. A nonzero polynomial of degree at most $n$ has at most $n$ roots, so equality forces $p=0$. Thus this is an inner product on ${\Pi }_n$.

On ${\Pi }_{n+1}$ it is only positive semidefinite: the nonzero node polynomial $\omega (x)={\prod }_{i=0}^n(x-x_i)$ satisfies $⟨\omega ,\omega ⟩=0$. Hence it is not an inner product on ${\Pi }_{n+1}$. The finite-evaluation inner product on polynomials works exactly when the evaluation map is injective on the chosen space.

Finally, if $q\ne 0$, positivity applied to $p-tq$ and the choice $t=⟨p,q⟩/⟨q,q⟩$ giveRearranging proves Cauchy-Schwarz inequality in its required form:If $q=0$ both sides of the asserted inequality are zero. Equality holds exactly when $p,q$ are linearly dependent, since in the nonzero case the minimizing $p-tq$ must have zero norm.

For every $z\in B$, $z_3+3>2$ and $|z_1+z_2|<2$. Consequently each of the three partial derivatives in the preceding gradient has absolute value below $1/2$, soThe segment joining $x,y\in B$ lies inside $B$. The mean value theorem and Cauchy-Schwarz inequality therefore giveFor $x=y$ both sides vanish. This proves the required Lipschitz continuity uniformly on the ball.

For every $z\in B$, $z_3+3>2$ and $|z_1+z_2|<2$. Consequently each of the three partial derivatives in the preceding gradient has absolute value below $1/2$, soThe segment joining $x,y\in B$ lies inside $B$. The mean value theorem and Cauchy-Schwarz inequality therefore giveFor $x=y$ both sides vanish. This proves the required Lipschitz continuity uniformly on the ball.

Positive definiteness gives a positive smallest eigenvalue $c$ with $x^{t}Ax\ge c|x{|}^2$. Hence the set $x^{t}Ax=y^{t}Ay=1$ is compact, and $h$ attains a maximum there. Its two constraint gradients are $(2Ax,0)$ and $(0,2Ay)$; they are linearly independent because $x,y$ are nonzero and $A$ is invertible.

Put $s=x^{t}Ay$. At any constrained extremum the multiplier equations areMultiplication by $x^t$ and $y^t$ gives $\lambda =\mu =s^2$. If $s=0$ the objective is zero. Otherwise $Ay=sAx$, so $y=sx$ and normalization gives $s^2=1$. Thus every maximum is at most one, and the value one is attained at $x=y$. This proves $\boxed{(x^{t}Ay{)}^2\le 1}$ on the normalized constraints.

For nonzero arbitrary vectors apply that result to $x/\sqrt{x^{t}Ax}$ and $y/\sqrt{y^{t}Ay}$. If either vector is zero the conclusion is immediate. ThereforeEquality for nonzero vectors holds precisely when they are linearly dependent. This is the Cauchy-Schwarz inequality for the inner product induced by $A$.

In a regular one-parameter model with joint density $f_{\theta }(x)$, write the score function as $U_{\theta }={\partial }_{\theta }\log f_{\theta }(X)$ and the Fisher information as $I_n(\theta )={\mathbb{E}}_{\theta }[U_{\theta }^2]$. Assume a parameter-independent support, differentiability and sufficient domination to interchange parameter differentiation and integration, and $0<I_n(\theta )<\infty$. If $T$ has finite variance and is unbiased for a differentiable target $g(\theta )$, the Cramér-Rao lower bound isIndeed, differentiating total mass gives ${\mathbb{E}}_{\theta }{U}_{\theta }=0$, and differentiating ${\mathbb{E}}_{\theta }T=g(\theta )$ gives ${\mathrm{Cov}}_{\theta }(T,U_{\theta })=g^{\prime }(\theta )$. Cauchy-Schwarz inequality proves the bound. For $n$ independent identically distributed observations, information adds: $I_n=n{I}_1$. In particular for the parameter itself, $g(\theta )=\theta$, the bound is $1/I_n$. The assumptions are material: the same formula cannot simply be imposed on models with parameter-dependent support.

Differentiability at $x_0$ means that a linear map $A:{\mathbb{R}}^n\to {\mathbb{R}}^m$ exists withThe map $A=Df(x_0)$ is the total derivative. A linear map between these finite-dimensional spaces is bounded, so $\Vert f(x_0+h)-f(x_0)\Vert \le \Vert A\Vert \Vert h\Vert +o(\Vert h\Vert )\to 0$. Thus differentiability implies continuity.

For the displayed two-variable function,Writing $r=\sqrt{x^2+y^2}$ gives $|f(x,y)|/r\le r\to 0$. Therefore it is differentiable at the origin, with derivative zero.

Finally, put $h=x-x_0$, write $f(x_0+h)=Ah+r(h)$ with $r(h)=o(\Vert h\Vert )$, and $g(x_0+h)=g(x_0)+s(h)$ with $s(h)=o(1)$. Since $f(x_0)=0$,By the Cauchy-Schwarz inequality the last two terms are $o(\Vert h\Vert )$: $A$ is bounded and $g(x_0)+s(h)$ stays bounded. HenceOnly continuity of $g$ at $x_0$ is needed, because the vanishing of $f(x_0)$ removes the term which would otherwise require differentiating $g$.

Zero is a strict weak local minimizer and is not a strong local minimizer. Write $D={\int }_0^1{u}^{\prime 2}$. For zero endpoint values, Cauchy-Schwarz inequality applied togives $|u(x){|}^2\le x(1-x)D$, hence ${\int }_0^1{x}^2{u}^2\le D/20$. For sufficiently small slopes, Taylor expansion gives $\frac{1}{2}[\sin (1+u^{\prime 2})-\sin 1]\ge (\frac{1}{2}\cos 1-\epsilon )u^{\prime 2}$. Thereforefor a suitably small fixed $\epsilon$ and every nonzero sufficiently weakly close competitor.

For strong minimality take $a=\sqrt{3\pi /2-1}$ and $u_N(x)=a\mathrm{dist}(x,N^{-1}\mathbb{Z})$. Their amplitudes tend to zero but slopes are $\pm a$, so $\sin (1+u_N^{\prime 2})=-1$. Thus $I(u_N)\le -1/2<I(0)=\sin 1/2$. Uniform closeness therefore does not prevent a decrease in energy.

For observables $A,B$, nonzero $[A,B]$ obstructs a complete common eigenbasis. In a finite-dimensional system, commuting Hermitian operators can be simultaneously diagonalized; quantum degeneracy may require choosing an appropriate basis inside the degenerate eigenspace. Noncommuting observables can still share particular eigenvectors, so incompatibility is not a claim that every individual state must have nonzero spreads in both quantities.

The Robertson uncertainty relation follows by applying Cauchy-Schwarz inequality to $(A-⟨A⟩)|\psi ⟩$ and $(B-⟨B⟩)|\psi ⟩$:In particular $[x,p_x]=i\hslash$ gives $\Delta x\Delta p_x\ge \hslash /2$. These are preparation spreads, not necessarily measurement errors. A zero commutator expectation can make this bound zero even when the linear operators do not commute.

Measurement order can matter: preparing $|+z⟩$, measuring $S_x$, then measuring $S_z$ gives both z outcomes with probability $1/2$ when the intermediate result is ignored. A direct z measurement would give $+\hslash /2$ certainly. Finally the Heisenberg equation of motion $dA/dt=(i/\hslash )[H,A]+\partial A/\partial t$ relates commutation to conservation: an observable with no explicit time dependence and $[H,A]=0$ is conserved. For unbounded self-adjoint operators these statements require the appropriate common domains; a formal commutator calculation alone need not justify simultaneous spectral measurements.

An observable is represented by a self-adjoint operator. For an ideal projective measurement, let $P_a$ be its spectral projector for outcome $a$. The Born rule gives $p(a)=⟨\psi |P_a|\psi ⟩$, or $p(a)=\mathrm{tr}(\rho P_a)$ for a density matrix $\rho$. Conditional on obtaining $a$, the Lüders rule givesFor a degenerate outcome this projects onto the whole associated eigenspace, rather than arbitrarily selecting a vector within it. An immediate repeated ideal measurement returns the same outcome, provided there is no intervening evolution or disturbance.

Commuting finite-dimensional observables admit a common orthonormal eigenbasis; their projectors commute and ideal joint measurements are compatible. Noncommuting observables generally have no complete common eigenbasis, and measurement order can affect the subsequent statistics. Particular common eigenvectors can nevertheless exist, so a nonzero commutator does not mean that every state has a nonzero uncertainty in both quantities.

For a quantitative constraint, apply the Cauchy-Schwarz inequality to $(A-⟨A⟩)|\psi ⟩$ and $(B-⟨B⟩)|\psi ⟩$. The imaginary part of their inner product is $⟨[A,B]⟩/(2i)$, giving the Robertson uncertainty principleThe canonical commutation relation $[x,p]=i\hslash$ gives $\Delta x\Delta p\ge \hslash /2$. Retaining the real part of the same inner product gives the stronger Robertson-Schrödinger uncertainty relation, including a covariance term. These are constraints on the preparation of a quantum state, not simply statements about a poorly calibrated instrument.

A sequential spin one-half example makes the change in statistics explicit. Prepare $|+z⟩$. An immediate $S_z$ measurement is certainly positive. If an $S_x$ measurement is inserted first, its outcomes each have probability $1/2$ and leave states $|\pm x⟩$. Each then yields either $S_z$ result with probability $1/2$, so the final positive fraction is $1/2$. The change reflects projection onto incompatible spin eigenstates, even with ideal measuring apparatus. The theory of measurement in quantum mechanics allows more general state-update rules, but the operator description and the Born rule still constrain their probabilities.

Write $A=g(X)$ and $B=g(Y)$. The two conditional expectation identities supply a $\sigma (Y)$-measurable version of $B$ and a $\sigma (X)$-measurable version of $A$. Work with those versions, which do not change any almost sure equality. Both lie in L2 space, and the Cauchy-Schwarz inequality makes their product an integrable random variable.

Testing the first conditional expectation identity against $B$, and the second against $A$, givesFor completeness, these tests with unbounded multipliers are valid by first using the bounded truncations $B_m=(-m)\vee (B\wedge m)$ and $A_m=(-m)\vee (A\wedge m)$, then passing to the limit with the Cauchy-Schwarz inequality and convergence in L2. Consequently,A nonnegative random variable with zero expected value vanishes almost surely. Therefore $g(X)=g(Y)$ almost surely. This is mutual conditional expectation rigidity in L2; it also follows from the equality case for conditional second moments.

For a counterexample to equality of the original random variables, let $X$ give probability $1/4$ to each of $-2,-1,1,2$, set $Y=-X$, and use the nonconstant function $g(t)=t^2$. Then $g(X)=g(Y)=X^2$ takes both values $1$ and $4$. It is measurable with respect to both generated sigma-algebras, so both required conditional expectation identities hold. Symmetry gives $\mathbb{E}X=\mathbb{E}Y=0$, while neither random variable can be zero. Thus

Assume $X$ is square-integrable. Since $X$ and $Y$ have the same probability distribution, $Y$ is also square-integrable and $\mathbb{E}[Y^2]=\mathbb{E}[X^2]$. The product $XY$ is integrable by the Cauchy-Schwarz inequality.

We use the defining test-function identity for conditional expectation: for bounded $\mathcal{G}$-measurable $H$, $\mathbb{E}[XH]=\mathbb{E}[YH]$. Take $H=Y{1}_{\{|Y|\le m\}}$ and let $m\to \infty$. The dominated convergence theorem, with bounds $|XY|$ and $Y^2$, givesThis is the extension of the conditional expectation pull-out identity to this square-integrable multiplier. ConsequentlyA nonnegative random variable with zero expectation is zero almost surely. HenceGeometrically, this is the equality case in the L2 space orthogonal projection interpretation of conditional expectation.