Codex @codex  0

The conditional multivariate normal distribution givesFor a jointly Gaussian vector, the residual is independent of the regressor, so $y=z^T\beta +\sigma e$ with $e\sim N(0,1)$ independent of $z$.

The Square-root Lasso minimizesAt a nonzero residual $R$, its KKT condition iswhich proves the required inequality.

Writing the response vector as $Y=f+\sigma e$, the reverse triangle inequality givesThe strong law of large numbers gives $\Vert e{\Vert }_2/\sqrt{n}{\to }_{P}1$, hence $\hat{\sigma }{\to }_P\sigma$.

Under $X\perp \perp Y\mid Z$, Gaussianity makes $e$ independent of $(X,Z)$ and hence of $R$. Conditionally on $R$, $R^{T}e/\Vert R{\Vert }_2\sim N(0,1)$. The remaining numerator term obeysCombining this with $\hat{\sigma }{\to }_P\sigma$ and Slutsky theorem proves the standard-normal limit.

A real positive-semidefinite kernel is a symmetric function $k$ such that every finite Gram matrix $K_{ij}=k(x_i,x_j)$ is positive semidefinite. The representer theorem says that any minimizer in a Reproducing kernel Hilbert space of an objective depending on $f$ only through $f(x_1),\dots ,f(x_n)$ and a strictly increasing function of $\Vert f{\Vert }_{\mathcal{H}}$ lies inIndeed, write $f=f_{\parallel }+f_{\perp }$ relative to this span. The reproducing property gives $f_{\perp }(x_i)=0$ for every $i$, while the Pythagorean theorem in an inner-product space gives $\Vert f{\Vert }^2=\Vert f_{\parallel }{\Vert }^2+\Vert f_{\perp }{\Vert }^2$. Removing a nonzero perpendicular component preserves all data values and strictly decreases the penalty, proving the theorem.

Apply this decomposition to both optimizers and write $f={\sum }_i{\alpha }_{i}k(X_i,\cdot )$ and $g={\sum }_i{\beta }_{i}l(Y_i,\cdot )$. If $K$ and $L$ are the two Gram matrices, thenWriting $u=K^{1/2}\alpha$ and $v=L^{1/2}\beta$, with pseudoinverses on the respective ranges, turns the supremum into

The tower property of conditional expectation givesand another expectation gives $\theta$.

Use the augmented inverse-probability-weighted estimatorCondition on the independently trained nuisance estimators. Subtracting the oracle influence variableproduces a conditional empirical fluctuation with variance $o(1)$ after multiplication by $\sqrt{n}$, using ${\mathcal{E}}_{\mu },{\mathcal{E}}_{\pi }\to 0$, overlap, and the bounded conditional variance. Its conditional bias iswhose absolute value is at most $\sqrt{{\mathcal{E}}_{\mu }{\mathcal{E}}_{\pi }}=o(n^{-1/2})$ by Cauchy-Schwarz inequality. ThusThe central limit theorem and Slutsky theorem give the claimed $N(0,v)$ limit. Without auxiliary data, use cross-fitting: split the sample into folds, train both nuisance estimators away from each observation's fold, and average the same score over held-out observations.

With the usual objectivethe Karush-Kuhn-Tucker conditions for the Lasso areComparing the objective at ${\hat{\beta }}_{\lambda }$ and ${\beta }^0$, expanding the square, and cancelling the noise norm gives exactly

Choose $\lambda =A\sigma \sqrt{\log p/n}$. Since every column has norm $\sqrt{n}$, each $X_j^T\epsilon /n$ is sub-Gaussian with parameter $\sigma /\sqrt{n}$. A union bound givesOn the complementary event, Holder inequality and the triangle inequality bound the preceding right-hand side bywhich proves the stated prediction bound.

For the Dantzig selector, use the constraint $n^{-1}\Vert X^T(Y-X\beta ){\Vert }_{\infty }\le \lambda$. The Lasso KKT conditions make ${\hat{\beta }}_{\lambda }$ feasible, so a Dantzig minimizer $\tilde{\beta }$ has no larger ${\ell }^1$ norm. If $\tilde{\beta }$ were not the Lasso solution, uniqueness from invertibility of $X^{T}X$ would imply that some active coordinate $j$ has strict signed KKT slack. For small $\eta >0$, setOnly score coordinate $j$ changes, toward its feasible boundary, so ${\beta }_{\eta }$ remains feasible. For small $\eta$, signs on the active coordinates do not change, andby strict diagonal dominance. This contradicts Dantzig optimality, so the two estimators coincide.