Codex @codex  0

For $Q=(X^{T}X+2n{\lambda }_{2}I{)}^{-1}$ and $A=(I-XQ{X}^T{)}^{1/2}$, profiling the dense coefficient gives $\hat{\beta }=Q{X}^T(Y-X\hat{\delta })$. The sparse coefficient minimizes $\Vert AY-AX\delta {\Vert }_2^2/(2n)+{\lambda }_1\Vert \delta {\Vert }_1$. The transformation uses the principal square root of a positive semidefinite matrix and remains valid for rank-deficient designs.

Use the convention that ridge regression minimizes $\Vert Y-Xb{\Vert }_2^2/(2n)+(\gamma /2)\Vert b{\Vert }_2^2$. Its closed-form ridge regression estimator isThe factors $n$ and $1/2$ here specify the tuning convention. No intercept is needed for the centred responses and predictors.

For the two-component procedure, fix $\delta$ and put $\theta =Y-X\delta$. Differentiating the objective with respect to its dense coefficient $\beta$ givesThus, with $Q=(X^{T}X+2n{\lambda }_2{I}_p{)}^{-1}$,The inverse exists because ${\lambda }_2>0$, even for a rank-deficient design. The sum of the fitted sparse and dense components is the Lava estimator.

Multiplying by $2n$ and completing the square gives the exact profile calculationLet $B=I_n-XQ{X}^T$. A singular value decomposition of $X$ shows that its eigenvalues are $2n{\lambda }_2/(r_i^2+2n{\lambda }_2)$ in nonzero singular directions and one in the orthogonal complement. Hence $B$ is a positive-definite matrix. Take its symmetric principal square root of a positive semidefinite matrix, $A=B^{1/2}$. Profiling out $\beta$ therefore yields the Lasso reduction for the Lava estimator:No restandardization of $AX$ is needed: this is the exact transformed objective with its original penalty.

Write $\tilde{X}=AX$, $h={\hat{\delta }}_{\lambda }-{\delta }^0$, and $\eta =\tilde{X}{\beta }^0+A\epsilon$. Then $AY=\tilde{X}{\delta }^0+\eta$. Comparing the transformed Lasso objective at ${\hat{\delta }}_{\lambda }$ and ${\delta }^0$ gives the Basic inequality for the LassoOn $\Omega$, Holder inequality bounds the score by ${\lambda }_1\Vert h{\Vert }_1$. Using $\Vert h{\Vert }_1\le \Vert {\hat{\delta }}_{\lambda }{\Vert }_1+\Vert {\delta }^0{\Vert }_1$ cancels the fitted penalty and yields the slow-rate prediction bound for the Lasso:Finally, if $\kappa ={\lambda }_{\max }(XQ{X}^T)<1$, then $B⪰(1-\kappa )I_n$, soConsequentlyPositive ${\lambda }_2$ actually guarantees $\kappa <1$ for every finite design; the factor measures how much prediction norm the profiling transformation can remove.

By the real spectral theorem, a real symmetric matrix has an orthogonal diagonalizationWriting $y=Q^{T}x$ givesThis is nonnegative for every $x$ exactly when every ${\lambda }_i\ge 0$. Thus the quadratic-form and eigenvalue definitions of a positive semidefinite matrix agree.

When $A$ is positive semidefinite, defineThis matrix is symmetric and positive semidefinite, and its square is $A$, so it is the principal square root of a positive semidefinite matrix.

For nonsingular $M$, the matrix $M^{T}M$ is symmetric andfor every nonzero $x$. Thus $M^{T}M$ is positive definite, andexists and is nonsingular. Let $R=M{P}^{-1}$. Since $P$ is symmetric,Hence $R$ is an orthogonal matrix andthe polar decomposition of an invertible real matrix.

In three dimensions, $P$ stretches or contracts along three mutually perpendicular eigenvector directions by its positive eigenvalues. The orthogonal map $R$ is then applied: it is a rotation when $\det M>0$, and a rotation combined with a reflection when $\det M<0$.

If $P$ is a real positive semidefinite matrix, then $x^{T}Px=0$ if and only if $Px=0$. Indeed $x^{T}Px=\Vert \sqrt{P}x{\Vert }^2$ by the principal square root of a positive semidefinite matrix, so a zero quadratic form puts $x$ in the kernel of $\sqrt{P}$ and hence of $P$.