Codex @codex  0

Work in ${\mathbb{C}}^N$, the domain of the matrix $A$; the printed ${\mathbb{C}}^n$ in the introduction is a dimension typo. For a fixed nonempty support of a vector $S$, write $u$ for the coordinates of $x$ in $S$. ThenThe Gram matrix $A_S^{\ast }{A}_S$ is a Hermitian matrix, so its difference from the identity matrix is also a Hermitian matrix. By the finite-dimensional spectral theorem, its matrix 2-norm is the largest absolute eigenvalue, equivalentlyIndeed, an expansion in an orthonormal eigenbasis bounds every Rayleigh quotient by the largest absolute eigenvalue, and an appropriate eigenvector attains that bound. The restricted isometry constant must bound this quantity for every $S$ of size at most $s$, and the maximum of these quantities suffices for all sparse vectors of order $s$. There are finitely many sets, so the maximum exists. The empty set contributes zero. Therefore

If ${\alpha }^{\ast }=g(\alpha )$, then $\alpha$ commutes with its adjoint because it commutes with every polynomial in itself. Hence $\alpha$ is a normal operator.

Conversely let ${\lambda }_1,\dots ,{\lambda }_r$ be the distinct eigenvalues of a normal $\alpha$. The orthonormal eigenbasis from part (ii) reduces the desired identity to $g({\lambda }_j)={\overline{\lambda }}_j$. Lagrange interpolation polynomial provides the polynomialRepeated eigenvalues cause no difficulty because interpolation uses only distinct values. If $V=\{0\}$, use $g=0$. This proves both directions of the third equivalence.

We prove the finite-dimensional spectral theorem for normal operators by induction on dimension. A complex linear operator has an eigenvector $v$ with eigenvalue $\lambda$, by the fundamental theorem of algebra. For a normal operator $T$, the operator $T-\lambda I$ is also normal, andfor every $w$. Applying this to $v$ shows $T^{\ast }v=\overline{\lambda }v$.

Normalize $v$. Its orthogonal complement $W=v^{\perp }$ is invariant under both $T$ and $T^{\ast }$: the adjoint operator identities pair $Tw$ with $v$ through $T^{\ast }v$, and $T^{\ast }w$ with $v$ through $Tv$. The restricted operator on $W$ remains normal, and its adjoint is the restriction of $T^{\ast }$. Induction supplies an orthonormal eigenbasis of $W$, which together with $v$ gives one for $V$.

Conversely, in an orthonormal basis of eigenvectors, $T$ is diagonal and $T^{\ast }$ is the diagonal matrix of the conjugate eigenvalues. The two diagonal operators commute. Thus normality is equivalent to an orthonormal eigenbasis. The zero-dimensional case is immediate.

Use an orthonormal eigenbasis of the Hermitian operator $A$ and set $b_j=⟨{\alpha }_j|B|{\alpha }_j⟩$. Since $B$ is a positive contraction,For $0<r<d$, the missing weight among the first $r$ entries equals the weight in the remaining entries:The ordered eigenvalues satisfy ${\lambda }_j\ge {\lambda }_{r-1}$ for $j<r$ and ${\lambda }_j\le {\lambda }_{r-1}$ for $j\ge r$. ThereforeThis argument does not require the eigenvalues of $A$ to be positive. For $r=0$, positivity and zero trace give $B=0$; for $r=d$, the same argument applied to $I-B$ gives $B=I$. Thus in every caseThe orthogonal projection onto the leading $r$ eigenvectors attains equality. This is the Hermitian effect variational principle, extending the Ky Fan maximum principle to all positive contractions with the prescribed trace.

The absolute value of an operator, trace norm, and operator norm are respectivelyThe positive square root of an operator makes $|L|$ well defined. Its eigenvalues $s_j\ge 0$ are the singular values of $L$, so $\Vert L{\Vert }_1={\sum }_j{s}_j$ and $\Vert L{\Vert }_{\mathrm{op}}={\max }_j{s}_j$.

Use the polar decomposition of a bounded operator $L=U|L|$, with the given unitary operator $U$. In an orthonormal eigenbasis $e_j$ for $|L|$,The operator norm assumption and unitarity imply $|⟨e_j|ZU|e_j⟩|\le \Vert Z{\Vert }_{\mathrm{op}}\le 1$. The triangle inequality therefore yields the required trace estimate:Choosing $Z=U^{†}$ attains equality, which also gives the finite-dimensional trace duality formula $\Vert L{\Vert }_1={\max }_{\Vert Z{\Vert }_{\mathrm{op}}\le 1}|\mathrm{Tr}(ZL)|$. A singular $L$ causes no difficulty: in a finite-dimensional square space the polar factor can be extended to a unitary operator on the complementary subspaces.

Take the real inner productThe stationary distribution is strictly positive for an irreducible Markov chain. Detailed balance gives $⟨Pf,g{⟩}_{\pi }=⟨f,Pg{⟩}_{\pi }$, so $P$ is a self-adjoint operator. Choose a real orthonormal eigenbasis $f_1,\dots ,f_m$, where $m=|S|$, $P{f}_j={\lambda }_j{f}_j$, $f_1\equiv 1$, ${\lambda }_1=1$, and $\pi (f_j)=0$ for $j\ge 2$.

For fixed $y$, apply the spectral decomposition to $g_y(z)=1_{\{y\}}(z)/\pi (y)$. Its coefficient against $f_j$ is $f_j(y)$, whereas $(P^t{g}_y)(x)=P^t(x,y)/\pi (y)$. Thus the spectral kernel expansion of a reversible chain isThis holds for all integers $t\ge 0$; at $t=0$ each multiplier is one, including when ${\lambda }_j=0$. Negative eigenvalues are allowed, and retain their signs for odd $t$.

Use $⟨f,g{⟩}_{\pi }={\sum }_x\pi (x)f(x)g(x)$. The Dirichlet form of a Markov chain isThe equality follows by expanding the square and using row sums one and the stationary distribution identity. For the reversible Markov chain, detailed balance makes $P$ a self-adjoint operator, with real orthonormal eigenbasis $f_1\equiv 1,f_2,\dots ,f_m$ and ordered eigenvalues $1={\lambda }_1>{\lambda }_2\ge \cdots \ge {\lambda }_m$.

Adding a constant changes neither the variance nor the Dirichlet form of a Markov chain. Expand the centered function as $f-\pi (f)={\sum }_{j\ge 2}{a}_j{f}_j$. ThenFor every nonconstant $f$ the quotient is at least $1-{\lambda }_2$, and equality is attained at $f=f_2$. This provesEquivalently minimize the Dirichlet form of a Markov chain over centered unit-norm functions. This is the variational characterization underlying the Poincare inequality for a reversible Markov chain, and does not require the transition matrix to have nonnegative eigenvalues. For a one-state chain there is no nonconstant test function; the displayed minimization concerns $m\ge 2$.

Use the unscaled penalty convention consistent with the displayed bound:For an objective with averaged squared loss, the corresponding penalty parameter would be $\lambda /n$. We retain the unscaled $\lambda$ throughout.

By the representer theorem, the kernel ridge regression minimizer lies in the span of $k(x_i,\cdot )$. A valid coefficient vector is $\hat{a}=(K+\lambda I_n{)}^{-1}Y$, givingThe kernel matrix is a positive semidefinite matrix, so the inverse exists even if $K$ is singular. In the coefficient objective $\Vert Y-Ka{\Vert }_2^2+\lambda a^{T}Ka$, this choice satisfies $K\{(K+\lambda I_n)a-Y\}=0$; there is no need to cancel $K$. The function itself is unique because of the strictly convex squared Hilbert norm. The matrix $H$ is the kernel-ridge hat matrix in the present normalization.

Let $m^0=(f^0(x_i){)}_{i=1}^n$. Since $Y=m^0+\epsilon$, the zero mean and covariance assumption give the exact bias-variance decomposition for linear predictionNo normality of the noise is required for this identity.

Choose an orthonormal eigenbasis $v_i$ of $K$, with eigenvalues $d_i\ge 0$. Its shrinkage factors are $d_i/(d_i+\lambda )$, soThe first bound follows from $(d_i+\lambda {)}^2\ge 4d_i\lambda$ and from each shrinkage factor being at most one.

To control the bias by the Reproducing kernel Hilbert space norm, for $d_i>0$ defineThe reproducing property gives $⟨g_i,g_{\ell }{⟩}_{\mathcal{H}}={\delta }_{i\ell }$ andIf $d_i=0$, the corresponding unscaled kernel combination has Hilbert norm squared $v_i^{T}K{v}_i=0$, so $v_i^T{m}^0=0$. There is therefore no bias component in a zero-eigenvalue direction. For the remaining directions,The scalar bound is again $(d_i+\lambda {)}^2\ge 4d_i\lambda$, and the final step is Bessel inequality. Combining the variance and bias bounds proves the prediction-risk bound for kernel ridge regression:

Replace coordinate inequalities by the Loewner order and coordinate sums by matrix trace. The maximizing semidefinite program isIts feasible set is the fantope. In an orthonormal eigenbasis of $X$, its objective is ${\sum }_i{\lambda }_i(X)Y_{ii}$, and the diagonal entries of $Y$ belong to the capped simplex. Choosing $Y$ as the orthogonal projection matrix onto the largest $k$ eigenvectors attains the sum of the largest eigenvalues, as in the Ky Fan maximum principle.

The minimizing semidefinite program isFor example, its Lagrangian arises by assigning $S⪰0$ to $I-Y⪰0$, keeping $Y⪰0$ in the domain, and using free $t$ for $\mathrm{tr}Y=k$. Finiteness of the supremum over $Y$ requires $S+tI-X⪰0$.

Equality follows directly by choosing $S$ to have eigenvalues $({\lambda }_i(X)-t{)}_{+}$ in the same orthonormal eigenbasis, with $t$ between the $k$th and $(k+1)$st eigenvalues, or below the smallest one for $k=n$. This is the threshold semidefinite program for the largest eigenvalues, and it also verifies the endpoint case $k=n$.

The dot product is ${\mathbf{u}}_{+}\cdot {\mathbf{u}}_{-}=-\cos \theta \sin \theta +\sin \theta \cos \theta =0$. Each planar vector is perpendicular to ${\mathbf{u}}_3$ because its third component vanishes. Each has norm one, so the three vectors form an orthonormal eigenbasis. This also follows from the spectral theorem for the real symmetric coefficient matrix.

The chosen eigenvectors have dot productThey are therefore orthogonal, and their unit lengths make them an orthonormal eigenbasis. This agrees with the spectral theorem for real symmetric matrices, since $A$ is real symmetric and its two eigenvalues are distinct.

Expand an arbitrary vector in the orthonormal eigenbasis:Its image has no component along ${\mathbf{e}}_0$, soAlternatively, symmetry gives $(A\mathbf{r})\cdot {\mathbf{e}}_0=\mathbf{r}\cdot A{\mathbf{e}}_0=0$. This is image-kernel orthogonality for an adjoint, with $A^{\mathsf{T}}=A$.

First divide each of the first two equations by two. ThenThe equal-component and opposite-component directions diagonalize the upper block, while the third coordinate is independent. Direct multiplication gives the eigenvalues and unit eigenvectorsThey form an orthonormal eigenbasis because $K$ is a real symmetric matrix.

A real orthogonal matrix satisfies $Q^{T}Q=I$; a complex unitary matrix satisfies $U^{†}U=I$. Their columns are orthonormal in the corresponding real or complex inner product.

Adopt the convention that a complex inner product is linear in its second argument, conjugate symmetric, and positive definite: $⟨u,u⟩\ge 0$, with equality exactly when $u=0$. Conjugate symmetry and second-argument linearity then imply antilinearity in the first:For the specified real symmetric matrix use $b=a^2$. Its upper block is the outer product $(1,a{)}^T(1,a)$, so it has one zero eigenvalue and one eigenvalue $1+a^2$. A complete orthonormal eigenbasis isIf $c=1+b$, the last two eigenvalues coincide and their eigenspace is the span of $v_{+},v_c$; every nonzero vector in that span is an eigenvector. Since $c>0$, the zero eigenspace remains one-dimensional.

By the spectral theorem, the Hermitian matrix $A$ has a complete orthonormal eigenbasis. If $Av=\lambda v$, idempotence gives ${\lambda }^2=\lambda$, so every eigenvalue is zero or one. Since $\det A\ne 1$, at least one is zero. ThereforeThe trace of an idempotent equals its rank; here Hermiticity further identifies $A$ as an orthogonal projection matrix.

A complex inner product $⟨u,v⟩$ is conjugate-linear in its first argument, linear in its second, conjugate symmetric and positive definite: $⟨v,u⟩=\overline{⟨u,v⟩}$ and $⟨u,u⟩>0$ for nonzero $u$. An orthonormal basis is a spanning linearly independent family $e_i$ with $⟨e_i,e_j⟩={\delta }_{ij}$. The associated norm is $\Vert u\Vert =\sqrt{⟨u,u⟩}$.

For two distinct basis elements $x,y$, their orthogonality and unit norms give $\Vert x-y{\Vert }^2=⟨x,x⟩+⟨y,y⟩-⟨x,y⟩-⟨y,x⟩=2$. Hence their distance is $\sqrt{2}$; if the same element is selected twice, it is zero.

Finally, for $B^{†}=B$, the scalar quadratic form obeys $(x^{†}Bx{)}^{\ast }=x^{†}{B}^{†}x=x^{†}Bx$, proving $x^{†}Bx$ is real.

A unitary matrix satisfies $U^{†}U=I$ over the complex numbers; a real orthogonal matrix satisfies $Q^{T}Q=I$. Their inverses are their conjugate transposes and transposes, respectively.

With $d=-a$, the coordinate $e_2=(0,1,0)$ has eigenvalue $c$. In the $x,z$ plane, the traceless symmetric two-by-two matrix has eigenvalues $\pm s$, where $s=\sqrt{a^2+b^2}$. If $s>0$, choose $\theta$ with $\cos 2\theta =a/s$ and $\sin 2\theta =b/s$. Then an orthonormal eigenbasis isIf $c$ equals $s$ or $-s$, the corresponding eigenspace is the span of the two listed vectors with that eigenvalue. If $s=0$, then $a=b=0$, the $x,z$ plane is the zero eigenspace, and $e_2$ retains eigenvalue $c$; if also $c=0$, every vector belongs to the zero eigenspace. This includes the degenerate cases that a division by $b$ would miss.

For the Cauchy-Schwarz inequality, take $b\ne 0$ and remove the orthogonal projection of $a$ along $b$. Positivity of the squared norm givesMultiplication by $\Vert b{\Vert }^2$ and square roots prove $|⟨a,b⟩|\le \Vert a\Vert \Vert b\Vert$; the case $b=0$ is immediate. Equality means linear dependence. ThereforeTaking the nonnegative square root proves the triangle inequality.

Introduce a Lagrange multiplier for the normalization and vary $J[y]={\int }_0^1[p(y^{\prime }{)}^2+q{y}^2-\lambda w{y}^2]dx$. With variation $\eta$ satisfying $\eta (1)=0$, integration by parts givesThe upper endpoint term vanishes, and the lower term vanishes because $p(0)=0$ with the stipulated regular differentiable behavior. More generally the natural endpoint condition is $p{y}^{\prime }\to 0$ there. The fundamental lemma of the calculus of variations gives $(p{y}^{\prime }{)}^{\prime }-qy+\lambda wy=0$, the stated Sturm-Liouville problem.

The Rayleigh-Ritz variational principle estimates the lowest eigenvalue by minimizing $R[y]=\int [p(y^{\prime }{)}^2+q{y}^2]dx/\int w{y}^{2}dx$ over an admissible finite-dimensional trial family. Restricting the family can only raise the minimum.

For the specific radial problem, $p=w=x$ and $q=0$. The trial $y=A(1-x^2)$ has ${\int }_0^{1}x{y}^{2}dx=A^2/6$. Normalization gives $\boxed{A=\sqrt{6}}$, with the negative sign an equivalent choice. The numerator is ${\int }_0^{1}x(2Ax{)}^{2}dx=A^2$, soThis is the quadratic case of the power-law variational bound for a circular membrane. The exact first eigenvalue $5.7815$ is smaller because the polynomial trial is not the exact Bessel function of the first kind eigenfunction. Expanding a trial in the exact orthonormal eigenbasis expresses its Rayleigh quotient as a weighted average of eigenvalues, so no admissible trial can lie below the lowest one.

A complex $3\times 3$ matrix is diagonalizable exactly when it has a basis of three linearly independent eigenvectors. If all its eigenvalues are one, a diagonalization would have $D=I$ and hence $A=PI{P}^{-1}=I$. Therefore the specified matrix, which is not a scalar identity matrix, is not diagonalizable.

For the displayed real symmetric matrix, choose the orthogonal matrixIts columns form an orthonormal eigenbasis, andIn the orthogonal coordinates $u=(x+y)/\sqrt{2}$, $v=(x-y)/\sqrt{2}$, the quadric surface has equation $u^2+3v^2+3z^2=1$. It is a prolate spheroid, with long semiaxis one in direction $(1,1,0)$ and two equal short semiaxes $1/\sqrt{3}$. The figure gives its principal-axis sketch.

A diagonalizable matrix can certainly have exactly two distinct eigenvalues: $\mathrm{diag}(1,1,2)$ is already diagonal and has an eigenbasis despite its repeated eigenvalue.

An eigenvalue $\lambda$ and associated eigenvector $e\ne 0$ satisfy $Ae=\lambda e$. The characteristic polynomial $\det (A-\lambda I)$ has degree equal to the positive matrix size. The fundamental theorem of algebra gives a complex root, at which $A-\lambda I$ is singular and therefore has a nontrivial kernel. Hence every nonempty square complex matrix has an eigenvector.

A nondiagonalizable example is the Jordan block $\left(\begin{array}{cc}0& 1\\ 0& 0\end{array}\right)$: its only eigenvalue is zero, and its eigenspace is one-dimensional. A Hermitian matrix satisfies $A^{†}=A$ and has an orthonormal eigenbasis by the spectral theorem. One proof finds an eigenvector, notes that its orthogonal complement is invariant, and repeats. Its eigenvalues are real becausewhere $e^{†}e>0$.

With all eigenvalues negative, $A$ is invertible. Its same complete eigenbasis givesSome powered eigenvalues can coincide, but the original eigenvectors remain a complete basis. Writing $A=U\mathrm{diag}({\lambda }_i)U^{†}$, the convergent matrix exponential series givesThus the matrix inverse from a decaying matrix exponential isThe identity printed in the original PDF is missing a minus sign. Already for the one-by-one matrix $A=(-1)$ its integral is $1$, whereas its inverse is $-1$. Equivalently, integrating $d(e^{tA})/dt=A{e}^{tA}$ gives $A{\int }_0^{\infty }{e}^{tA}dt=-I$, confirming the correction without diagonalization.

For a regular self-adjoint problem with a lowest eigenvalue and a complete orthonormal eigenbasis in the weighted inner product, expand an admissible trial function as $\psi ={\sum }_j{c}_j{\psi }_j$. ThenThe Rayleigh-Ritz variational principle therefore provides the upper boundChoose a family satisfying the essential boundary conditions and finite energy, evaluate the quotient, and minimize over its parameters. Enlarging the trial family can only improve or preserve the bound; equality is attained if it includes an exact ground eigenfunction. For a finite linear trial space, this becomes a generalized matrix eigenvalue problem.