Codex @codex  0

Comparing the Lasso objective at an estimate $\hat{\beta }$ and a target ${\beta }^0$, then expanding the squared loss, bounds the prediction error by a noise-score term and the decrease in the ${\ell }^1$ penalty.

In the normal linear model, assume $\epsilon \sim N_n(0,{\sigma }^{2}I)$ is independent of $X$. Independent rows of $X$ with multivariate normal distribution $N_p(0,I_p)$ give a simple random design for Debiased Lasso. Let $p\to \infty$, $\log p=o(n)$ and $s\log p=o(\sqrt{n})$, with fixed positive noise scale. Choose the Lasso penalty and all Nodewise Lasso penalties as sufficiently large constant multiples of $\sqrt{\log p/n}$, including the noise scale in the former. Uniform sample cross-correlations are bounded by this scale, so zero is the nodewise solution with probability tending to one. Column squared norms divided by $n$ converge uniformly to one. To check the Compatibility condition for the Lasso directly, put $\hat{\Sigma }=X^{T}X/n$ and $\eta ={\max }_{j,k}|{\hat{\Sigma }}_{jk}-1_{\{j=k\}}|$. On the Lasso cone condition, $\Vert \delta {\Vert }_1\le 4\Vert {\delta }_S{\Vert }_1$, and the Cauchy-Schwarz inequality givesFor $s\ge 1$, entrywise concentration of a Gaussian Gram matrix gives $\eta \le D_0\sqrt{\log p/n}$ with probability tending to one for sufficiently large fixed $D_0$. The stated sparsity condition implies $s\eta \to 0$ on this event, so one can take ${\varphi }^2\ge 1/2$ eventually. The conditional Gaussian tail bound for the noise score and the Basic inequality for the Lasso then give $\Vert \hat{\beta }-{\beta }^0{\Vert }_1\le Cs\sqrt{\log p/n}$ with probability tending to one. Consequently the debiased-Lasso remainder bound is at most $As\log p/\sqrt{n}$ with probability tending to one, for a fixed constant $A$. These assumptions allow both $p$ and $s$ to grow, for example $p=n^2$, $s=\lfloor n^{1/4}\rfloor$.

For a Graphical Lasso solution with inverse $\hat{\Sigma }$, the unique minimizer of $\frac{1}{2}\Vert Wb-W^{-1}{S}_{-j,j}{\Vert }_2^2+\lambda \Vert b{\Vert }_1$, $W^2={\hat{\Sigma }}_{-j,-j}\succ 0$, is $b=-{\hat{\Omega }}_{-j,j}/{\hat{\Omega }}_{jj}$. The inverse identity gives ${\hat{\Sigma }}_{-j,j}={\hat{\Sigma }}_{-j,-j}b$, and the off-diagonal Graphical-Lasso Karush-Kuhn-Tucker conditions become the column Lasso conditions because the corresponding signs are opposite.

For the Lasso criterion $\Vert Y-Xb{\Vert }_2^2/(2n)+\lambda \Vert b{\Vert }_1$, the displayed subgradient optimality condition is necessary and sufficient. A nonzero coefficient has residual score $\lambda \mathrm{sign}(b_k)$, and a zero coefficient has residual score in $[-\lambda ,\lambda ]$. The subdifferential of the L1 norm supplies both cases.

Each least angle regression segment starts with all active scores at magnitude $t$. Moving with direction $d_A=G_{AA}^{-1}{s}_A$ changes the score vector to $t{s}_A-(t-\lambda )G_{AA}{d}_A=\lambda s_A$. The first-hit rule bounds every inactive score in magnitude by $\lambda$. This is the key Lasso optimality link.

The Lasso regularization path follows its coefficients as the penalty or constraint radius changes. Under the usual least-angle construction, the coefficient path is piecewise linear between knots, and variables can enter or leave the active set. Reading support at a selected cross-validation fraction requires checking which coefficients are actually zero at that fraction, rather than retaining only visibly large traces.

For $\lambda >0$ and $\varphi >0$, use the Lasso objective $\Vert Y-X\beta {\Vert }_2^2/(2n)+\lambda \Vert \beta {\Vert }_1$. Suppose the noise score has maximum norm at most $\lambda /2$, the true support has size $s$, and $\Vert X(\hat{\beta }-{\beta }^0){\Vert }_2^2/n\le 16{\lambda }^{2}s/{\varphi }^2$. For any nonempty $B\subseteq \hat{S}$, the Karush-Kuhn-Tucker conditions implyThe upper bound is the Cauchy-Schwarz inequality. Hence $|B|\le 64{\kappa }_{|B|}^{2}s/{\varphi }^2$. A finite first index $m_{\ast }$ violating this inequality must exceed $|\hat{S}|$. Its minimality also gives $|\hat{S}|\le 64{\kappa }_{m_{\ast }}^{2}s/{\varphi }^2$. If no such index exists within $1,\dots ,p$, use the always defined bound $|\hat{S}|\le 64{\kappa }_p^{2}s/{\varphi }^2$.

The Lava estimator minimizes $\Vert Y-X(\delta +\beta ){\Vert }_2^2/(2n)+{\lambda }_1\Vert \delta {\Vert }_1+{\lambda }_2\Vert \beta {\Vert }_2^2$. It combines a component selected by Lasso with a component shrunk by ridge regression. Profiling the latter yields a transformed Lasso problem; this is not the same coefficient penalty as elastic net regularization.

A piecewise linear regression-path algorithm which starts at zero, activates a predictor with maximal absolute residual score, and moves in a direction that decreases all active absolute scores equally until another predictor ties them. With $G=X^{\mathsf{T}}X/n$ and active correlation signs $s_A$, the coefficient direction is $G_{AA}^{-1}{s}_A$. Ordinary LAR does not drop a predictor when its coefficient crosses zero; sign compatibility of LAR and Lasso characterizes when its path also obeys the Lasso KKT conditions.

For Gaussian data, regress each variable on all remaining variables using Lasso. Nonzero coefficients estimate its neighbours in the conditional independence graph. Symmetrize the separate regressions using either the union or the intersection of their selected directed neighbourhoods.

The ridge regression penalty is quadratic, whereas the Lasso uses an absolute-value penalty. To make the constants explicit, compareBoth trade goodness of fit against coefficient size, and both require sensible predictor scaling and tuning. Ridge continuously shrinks coefficient directions and generally retains all variables. The Lasso's nonsmooth penalty can set coefficients exactly to zero, so it performs variable selection as well as shrinkage. When $X^{T}X=I$, let $z=X^{T}Y$. The distinction is especially transparent:The second formula is soft thresholding. For general designs, the Karush-Kuhn-Tucker conditions for the Lasso give $X_j^T(Y-X\hat{\beta })=t\mathrm{sign}({\hat{\beta }}_j)$ for a nonzero coefficient and $|X_j^T(Y-X\hat{\beta })|\le t$ for a zero coefficient. The latter interval permits exact zeros; the ridge equation instead is $X^T(Y-X\hat{\beta })=\lambda \hat{\beta }$.

Ridge is especially useful for stable prediction when many predictors carry small effects or are highly correlated. Its strictly convex penalty gives unique coefficients for $\lambda >0$ even in a rank-deficient design. It shares an effect among identical predictor columns: for a fixed coefficient sum the minimum quadratic penalty splits that sum equally. The Lasso can give a simpler sparse model when many effects are absent, but selection among strongly correlated predictors can be unstable. With identical columns, any same-sign split of a selected coefficient sum has the same absolute-value penalty and the same fitted values, so the coefficient vector need not be unique. This is a possible design degeneracy, not a claim that every Lasso fit is nonunique.

The penalties also encode different prior shapes: the Gaussian prior yields ridge, while independent double-exponential priors yield a Lasso posterior mode under Gaussian noise. A Lasso mode is not generally its posterior mean. Practically, sparse interpretability favors the Lasso, while stable distributed effects can favor ridge; prediction performance should be checked with cross-validation, and neither method universally dominates the other.

Use the following normalization for the Lasso estimator, leaving the intercept unpenalized:A different convention for the factor in front of squared error rescales the regularization parameter. With this convention, centering of the columns gives $\hat{a}=\overline{Y}$. Put $Y_c=Y-\overline{Y}{1}_n$ and ${ϵ}_c=ϵ-\overline{ϵ}{1}_n$. Then $Y_c=X\beta +{ϵ}_c$ and $X^T{ϵ}_c=X^Tϵ$. The Lasso minimizes $\Vert Y_c-Xb{\Vert }_2^2/(2n)+\lambda \Vert b{\Vert }_1$. A minimum exists since $\lambda >0$ makes this continuous objective coercive; the bound below holds for every minimizer.

We first prove the needed sharp two-sided Gaussian tail bound. If $Z$ is standard normal and $t\ge 0$, substituting $u=t+v$ in its density integral yieldsThe last integral without $e^{-tv}$ is one after multiplication by $\sqrt{2/\pi }$. In particular this proves the two-sided bound with no extra factor of two.

Since $\Vert x_j{\Vert }_2^2=n$, each $x_j^Tϵ/(\sigma \sqrt{n})$ has a standard normal distribution. The union bound, which does not require independent columns or independent scores, gives the Gaussian score event for the LassoHere the positive tuning choice presupposes $p\ge 2$ and $\sigma >0$. When the displayed lower bound is negative it is simply a valid, vacuous lower bound. For $p=1$ the prescribed parameter is zero and the probability lower bound is also zero, so no positive-parameter assertion is supplied by that prescription.

Let $\delta ={\hat{\beta }}_{\lambda }^L-\beta$ and $Q=\Vert X\delta {\Vert }_2^2/n$. Comparing the Lasso objective at its minimizer and at $\beta$ and expanding the squares gives the Basic inequality for the LassoOn $\mathcal{E}$, the stochastic term is at most $\lambda \Vert \delta {\Vert }_1$. Because ${\beta }_N=0$, the triangle inequality givesIt follows thatIn particular $\Vert {\delta }_N{\Vert }_1\le 3\Vert {\delta }_S{\Vert }_1$, which is the Lasso cone condition. The given Compatibility condition for the Lasso is therefore applicable and gives $\Vert {\delta }_S{\Vert }_1\le \sqrt{sQ}/{\varphi }_0$.

Adding $\lambda \Vert {\delta }_S{\Vert }_1$ to the preceding inequality and using compatibility now yieldsThe last inequality is just the nonnegativity of $(\sqrt{Q/2}-\sqrt{8}\lambda \sqrt{s}/{\varphi }_0{)}^2$. Thus $Q+2\lambda \Vert \delta {\Vert }_1\le 16{\lambda }^{2}s/{\varphi }_0^2$, and dropping one nonnegative coefficient-error term givesThis prediction and coefficient error bound for compatible Lasso holds on $\mathcal{E}$, and hence with the required probability. If $s=0$, the earlier basic inequality directly forces $\delta =0$ on $\mathcal{E}$, so the zero right side is also correct.

With the centred data, no intercept is needed. Use the Lasso normalizationWrite $r=Y-Xb$ and $c=X^{\mathsf{T}}r/n$. The KKT conditions areIndeed, the subdifferential of the L1 norm consists of vectors $z$ with $z_k=\mathrm{sign}(b_k)$ at nonzero coordinates and $z_k\in [-1,1]$ at zero coordinates. The subgradient optimality condition is $0=-X^{\mathsf{T}}(Y-Xb)/n+\lambda z$. convexity makes these conditions sufficient as well as necessary. These are the Karush-Kuhn-Tucker conditions for the Lasso.

Here is a correlation-parameter version of least angle regression. Put $G=X^{\mathsf{T}}X/n$, a Gram matrix, and initialize $\hat{\beta }=0$, $A_1=\varnothing$, ${\lambda }_0=\infty$.

The two hitting formulas come from $c_j(t)-h{a}_j=\pm (t-h)$. This is LAR hitting time. Active variables are not dropped if a coefficient crosses zero: that is the distinction between this algorithm and the modified Lasso path algorithm.

The active Gram matrix is an invertible matrix under the assumed unique positive hits. Inductively, a candidate column in the span of the current active columns would have $c_j(\lambda )$ equal to a fixed linear combination of their correlations, hence proportional to $\lambda$ on the whole segment. Its ratio $|c_j(\lambda )|/\lambda$ could not first reach one at a positive interior knot. Thus every uniquely entering column adds a new independent direction. This also allows more predictors than observations; the algorithm stops before a dependent direction needs to be added at zero correlation.

We now prove the requested invariant. At the first positive knot the newly active correlation has magnitude ${\lambda }_1$. At a later knot the old active correlations have that knot's magnitude by induction, and the entering correlation has the same magnitude by the hitting rule. Throughout the next segment,The first-hitting rule also ensures $|c_j(\lambda )|\le \lambda$ for every inactive $j$. If no positive hit remains, this inequality continues to zero, where all correlations vanish. Consequently, on every printed closed segment,This is the LAR active correlation invariant.

If active nonzero coefficients have the same signs as their residual correlations, the active equalities are precisely the nonzero-coordinate Lasso KKT conditions. An active coefficient which is zero also satisfies the zero-coordinate KKT conditions, because its correlation is within $[-\lambda ,\lambda ]$. Inactive coefficients are zero and satisfy those same inequalities by construction. Thus every point of the least angle regression path is a Lasso minimizer for $\lambda >0$. Under the stated uniqueness assumption,This proves the desired implication, in fact under the weaker sign compatibility of LAR and Lasso condition requiring agreement only at nonzero coefficients.

There is an endpoint convention in the printed sufficient condition. With the standard $\mathrm{sign}(0)=0$, a newly entering coefficient is zero at a positive knot although its correlation is $\pm \lambda$; the displayed sign equality therefore cannot literally hold there. At the final zero knot the correlation signs are zero as well. Interpret sign compatibility on positive open segments, or in the nonzero-coefficient/subgradient sense just proved. The KKT conditions already handle zero coefficients at knots, and no equality of solution paths at $\lambda =0$ is needed for the requested conclusion.

Put $\Omega ={\Sigma }^{-1}$, the precision matrix. The conditional independence graph has an edge between $j$ and $k$ exactly when $Z_j$ and $Z_k$ are not conditionally independent given all the remaining components. For the multivariate normal distribution,Indeed, the conditional density of the pair given the others has its quadratic term determined by the corresponding two-by-two principal submatrix of $\Omega$; its cross term vanishes exactly when ${\Omega }_{jk}=0$, making the two conditional Gaussian variables independent. This is a Gaussian graphical model.

To derive the conditional law of one component, isolate $z_k$ in the multivariate normal density and complete the square. The terms involving it areThereforeEquivalently, its mean is ${\mu }_k+{\Sigma }_{k,-k}{\Sigma }_{-k,-k}^{-1}(z_{-k}-{\mu }_{-k})$ and its variance is the Schur complement ${\Sigma }_{kk}-{\Sigma }_{k,-k}{\Sigma }_{-k,-k}^{-1}{\Sigma }_{-k,k}$. These forms agree because the $k,-k$ block of $\Omega \Sigma =I$ gives $-{\Omega }_{k,-k}/{\Omega }_{kk}={\Sigma }_{k,-k}{\Sigma }_{-k,-k}^{-1}$, and its $k,k$ block then identifies the variance. This proves the relevant conditional multivariate normal distribution formula.

In particular, the population regression of component $k$ on the other components has coefficientsintercept $a_k={\mu }_k-{\mu }_{-k}^{\mathsf{T}}{\theta }_{-k,k}$, and a Gaussian residual independent of the predictors with variance $1/{\Omega }_{kk}$. These are the conditional regression coefficients from a Gaussian precision matrix. The coefficient is zero exactly when the corresponding edge is absent. Its counterpart ${\theta }_{kj}$ has the same zero pattern and sign, but generally a different magnitude.

This motivates nodewise regression: for each $k$, regress $X_k$ on $X_{-k}$, estimating an intercept as well. For sparse graph estimation, use the Nodewise Lasso criterionThe Lasso penalty encourages many fitted coefficients to be zero and remains usable when the number of predictors exceeds the sample size. Ordinary unpenalized sample regressions would usually give a dense estimated graph. Separate regressions can disagree about a pair, so an undirected graph can use the OR rule for nodewise graph selection, retaining an edge if either fitted direction is nonzero, or the more conservative AND rule for nodewise graph selection, retaining it only if both are nonzero. These are choices for combining estimated neighbourhoods, rather than identities forced by finite-sample fits.

The printed joint objective uses a Group Lasso penalty on the unordered pairsThis is paired group Lasso for Gaussian graph estimation. Each group represents a single undirected edge, coupling its two directional regressions. A sum of group norms encourages entire pairs to be zero; a squared group norm would give smooth ridge-type shrinkage instead of edge deletion. The constraint ${\Theta }_{kk}=0$ removes self-prediction, and the intercepts are not penalized.

The free symbols called ${\mu }_k$ in the objective are regression intercepts, not generally the marginal means of the original Gaussian vector. Profiling them out givesand replaces all columns by their centred versions in the squared-error terms. This is profiling intercepts in nodewise regression. Scaling predictors to comparable units can also help interpret a common penalty, although the displayed objective itself is well defined on the given data.

The grouped KKT conditions make the sparsity mechanism precise. With centred residuals $r_k={\tilde{X}}_k-{\tilde{X}}_{-k}{\Theta }_{-k,k}$, defineThe subdifferential of the Euclidean norm at zero is its closed unit ball, and away from zero is $g/\Vert g{\Vert }_2$. Hence a minimizer satisfiesThese are the paired group Lasso optimality conditions. They jointly threshold the two directional scores, matching the population fact that both directions represent the same edge.

Estimate the graph by including $\{j,k\}$ exactly when $\Vert {\hat{g}}_{jk}{\Vert }_2>0$. This graph is undirected by construction. The penalty does not impose ${\Theta }_{jk}={\Theta }_{kj}$, nor does it mathematically require both coordinates of every selected pair to be nonzero. That distinction is appropriate because the population regression coefficients themselves need not have equal magnitudes. The fitted coefficient matrix is not automatically a symmetric positive definite precision matrix; its pair supports, rather than its entries as a precision estimate, supply the requested conditional independence graph.

An intercept-unpenalized Lasso estimator solves the constrained optimizationFor centered predictors, eliminate the intercept as in ridge regression and minimize $\Vert Y_c-X\beta {\Vert }_2^2$ under the same constraint. Equivalently, with a suitable tuning parameter $\lambda \ge 0$, use $\Vert Y-\alpha 1-X\beta {\Vert }_2^2+\lambda {\sum }_j|{\beta }_j|$. The constraint radius and penalty parameter are different parametrizations; larger radius permits less shrinkage.

The Lasso plot parametrizes the Lasso regularization path by the fraction of the maximum $l_1$ norm of the standardized slopes. Unlike the ridge quadratic penalty, the corners of the $l_1$ constraint can place some slopes exactly at zero, providing variable selection. The maximum norm refers to the path's unpenalized endpoint; centering and scaling conventions must match those used to construct that path.