Codex @codex  0

An intercept-unpenalized Lasso estimator solves the constrained optimizationFor centered predictors, eliminate the intercept as in ridge regression and minimize $\Vert Y_c-X\beta {\Vert }_2^2$ under the same constraint. Equivalently, with a suitable tuning parameter $\lambda \ge 0$, use $\Vert Y-\alpha 1-X\beta {\Vert }_2^2+\lambda {\sum }_j|{\beta }_j|$. The constraint radius and penalty parameter are different parametrizations; larger radius permits less shrinkage.

The Lasso plot parametrizes the Lasso regularization path by the fraction of the maximum $l_1$ norm of the standardized slopes. Unlike the ridge quadratic penalty, the corners of the $l_1$ constraint can place some slopes exactly at zero, providing variable selection. The maximum norm refers to the path's unpenalized endpoint; centering and scaling conventions must match those used to construct that path.

At fraction $0.8$, the Lasso regularization path lies between the vertical lines marked steps 4 and 5. In that interval the first five predictors have nonzero slopes: $X_1,X_2,X_4,X_5$ are positive and $X_3$ is negative. The sixth predictor remains at zero until the later part of the path, beyond this fraction. Therefore the chosen Lasso model includesThe regression intercept is retained. In particular the negative $X_3$ trace has already left zero at fraction $0.8$; a small slope is not the same as a zero slope. The selected fraction came from ten-fold K-fold cross-validation, and is not a numerical value of the ridge penalty in the preceding parts.

Fix the normalizationThe Lasso penalty uses the L1 norm. All arguments below apply to every minimizer; uniqueness of the coefficient vector is not assumed.

The Karush-Kuhn-Tucker conditions are the subgradient optimality conditionThese conditions are necessary and sufficient because the objective is a convex function. They use the subgradient of the absolute value, coordinate by coordinate.

Put $\delta =\hat{\beta }-{\beta }^0$. Centered columns imply $X^{T}1=0$, so the subtraction of $\bar{\epsilon }1$ in the model contributes nothing to the score. The Karush-Kuhn-Tucker conditions becomeMultiply by ${\delta }^T$. Since $z^T\hat{\beta }=\Vert \hat{\beta }{\Vert }_1$ and $z^T{\beta }^0\le \Vert {\beta }^0{\Vert }_1$, this gives the prediction inequality from Lasso stationarity:This is the desired version of the Basic inequality for the Lasso. Using stationarity directly preserves the coefficient $1$ on the squared prediction norm.

For the Gaussian score event for the Lasso, write $X_j$ for column $j$. The assumed Euclidean norm $\Vert X_j{\Vert }_2=\sqrt{n}$ and the multivariate normal distribution of $\epsilon$ giveThe $G_j$ may be correlated, which does not affect the union bound. Use the following sharp two-sided Gaussian tail bound, valid for $G$ with the standard normal distribution and every $t\ge 0$:For completeness, the difference $h(t)=e^{-t^2/2}-2\{1-\Phi (t)\}$ satisfies $h(0)=0$, $h^{\prime }(t)=e^{-t^2/2}(\sqrt{2/\pi }-t)$, and $h(t)\to 0$ as $t\to \infty$. It first increases and then decreases to zero, so it is nonnegative. This bound avoids introducing an unnecessary factor two.

Here $\Phi$ is the standard normal distribution function. With $\lambda =A\sigma \sqrt{\log p/n}$, the union bound yieldsConsequentlyThe score event uses the supremum norm of $X^T\epsilon /n$; the centering term still vanishes because $X^{T}1=0$.

We next work deterministically on $\Omega$. LetBy Holder inequality, ${\epsilon }^{T}X\delta /n\le c\lambda \Vert \delta {\Vert }_1=c\lambda (a+b)$. Since ${\beta }_N^0=0$, the triangle inequality givesSubstituting both inequalities into the Basic inequality for the Lasso producesBecause $Q\ge 0$, this also proves the relevant Lasso cone condition, $(1-c)b\le (1+c)a$. The assumed Compatibility condition for the Lasso therefore applies to $\delta$:Set $D=(1+c)\lambda \sqrt{s}/\varphi$. Combining these inequalities givesIf $Q>0$, discarding the nonnegative $b$ term shows $\sqrt{Q}\le D$. If $Q=0$, compatibility gives $a=0$, and the preceding inequality then gives $b=0$. In either case $D\sqrt{Q}\le D^2$, soThis is a fast prediction bound and an inactive-coordinate error bound on the same event. It is the fast-rate Lasso prediction bound under compatibility.

Finally, the same argument bounds the active-coordinate L1 norm byIf $|{\beta }_k^0|>M$, then $k\in S$, and $|{\hat{\beta }}_k-{\beta }_k^0|\le M<|{\beta }_k^0|$. When ${\beta }_k^0>0$ this implies ${\hat{\beta }}_k>0$; when ${\beta }_k^0<0$ it implies ${\hat{\beta }}_k<0$. Thus the sign function obeysThis is Lasso sign recovery under compatibility. It concerns the sufficiently large nonzero coefficients; it does not assert that every estimated inactive coefficient is zero.

Write $\Omega ={\Sigma }^{-1}$ for a positive-definite matrix. Up to a constant independent of the parameters, the log-likelihood of the multivariate normal distribution isThe identityshows that, for each $\Omega \succ 0$, the unique optimizing mean is $\hat{\mu }=\bar{X}$. Using the matrix trace to rewrite the first quadratic sum, the profile likelihood isTherefore maximum likelihood estimation of the precision matrix, whenever the maximum exists, is exactlyThis is the profile likelihood for a Gaussian precision matrix. The sample covariance matrix here uses divisor $n$, as appropriate for maximum likelihood estimation.

There is an important rank qualification in the printed assumptions. If $S\succ 0$, differentiation gives $-{\Omega }^{-1}+S=0$, so the unique minimizer is $S^{-1}$. Uniqueness follows because the negative log-determinant is a strictly convex function on the positive-definite matrices. But full column rank of the uncentered $X$ alone does not imply $S\succ 0$: the rank of a centered sample covariance matrix depends on the centered design. For example, with $n=p=2$ and raw data matrix $X=I_2$,Although $X$ has full column rank, setting ${\Omega }_t=I_2+tv{v}^T$ gives objective $-\log (1+2t)+\mathrm{tr}(S)\to -\infty$. Thus an unpenalized maximizer need not exist under the literal raw-rank assumption. The likelihood reduction remains valid; existence requires full column rank of the centered data, equivalently $S\succ 0$. For the nonsingular Gaussian model this holds almost surely when $n>p$.

Use the entrywise-penalty convention for the Graphical Lasso:The sum is the entrywise matrix L1 norm. Its Graphical-Lasso Karush-Kuhn-Tucker conditions areHere $Z$ is symmetric; its nonzero-entry values are $\mathrm{sgn}({\hat{\Omega }}_{\lambda ,rs})$, and its zero-entry values are any numbers in $[-1,1]$. These are the subgradient optimality condition for this convex optimization problem. Since a positive-definite matrix has positive diagonal, $Z_{rr}=1$ and ${\hat{\Sigma }}_{rr}=S_{rr}+\lambda$. Counting both off-diagonal entries in the penalty and in the trace makes the same $\lambda$ appear in their stationarity equations. The common variant penalizing only off-diagonal entries instead sets $Z_{rr}=0$; the off-diagonal equations and the column argument below are unchanged.

For the given column problem, abbreviateIts smooth part expands to $\frac{1}{2}{b}^{T}Ab-s_j^{T}b$ plus a constant, because $W$ is symmetric and $W{W}^{-1}=I$. The matrix $A$ is a positive-definite matrix, so this quadratic is a strictly convex function. Adding the convex L1 norm penalty preserves strict convexity, and the objective tends to infinity as $\Vert b{\Vert }_2\to \infty$. The minimizer $b^{\ast }$ exists and is unique. Its Karush-Kuhn-Tucker conditions are

We identify that minimizer directly from the precision matrix. Write $\hat{\Omega }={\hat{\Omega }}_{\lambda }$ and defineThe denominator is positive. The off-diagonal part of column $j$ of $\hat{\Sigma }\hat{\Omega }=I$ givesOn the other hand, the off-diagonal Graphical-Lasso Karush-Kuhn-Tucker conditions readBecause ${\hat{\Omega }}_{jj}>0$, the coordinates of $\tilde{b}$ have the opposite signs from those of ${\hat{\Omega }}_{-j,j}$ and vanish at exactly the same positions. The subgradient of the absolute value therefore satisfiesIt follows thatThese are exactly the Lasso column problem's Karush-Kuhn-Tucker conditions. Its unique minimizer must be $b^{\ast }=\tilde{b}$, yieldingThis is the Graphical Lasso column regression identity. The minus sign is essential: it converts the precision-entry subgradient into the regression-coefficient subgradient. The argument uses the inverse identity directly, so it does not require proving the supplied block inverse formula.

Use the normalizationfor the Lasso. A centered sub-Gaussian random variable with parameter $\sigma$ satisfies $\mathbb{E}{e}^{tW}\le e^{{\sigma }^2{t}^2/2}$ for all $t\in \mathbb{R}$; for a noncentered variable apply this definition to $W-\mathbb{E}W$. With fixed $X$, independence of the errors givesThe exponential Markov bound, optimized over $t$ separately for the two signs, gives $\mathrm{Pr}(|X_j^T\epsilon |/n>u)\le 2e^{-n{u}^2/(2{\sigma }^2)}$. Take $u=\lambda /2=\sigma A\sqrt{\log p/n}$ and apply the union bound over columns:The specified positive tuning parameter requires $p\ge 2$; at $p=1$ its formula is zero. For small $A$ the probability lower bound can be negative and hence uninformative. Although centering makes the centered errors dependent, $X^{T}1=0$ means $X^T(\epsilon -\bar{\epsilon }1)=X^T\epsilon$, so the preceding proof uses the original independent errors, not independence after centering.

The Karush-Kuhn-Tucker conditions, using the subdifferential of the L1 norm, areConsequently, for nonempty $B\subseteq \hat{S}$, on $\Omega$,This establishes the required active-set inequality without any rank condition on $X$.

For $s>0$, a sufficient Compatibility condition for the Lasso isIndeed, for $\delta =\hat{\beta }-{\beta }^0$, comparison of the two Lasso objective values and the score bound on $\Omega$ give the Basic inequality for the LassoIt follows that $\delta$ satisfies the Lasso cone condition and $\Vert {\delta }_S{\Vert }_1\le \sqrt{s{P}_E}/\varphi$. Thus $P_E\le 3\lambda \sqrt{s{P}_E}/\varphi$, including the trivial $P_E=0$ case. This proves the slightly stronger $P_E\le 9{\lambda }^{2}s/{\varphi }^2$, and in particular the requested $16{\lambda }^{2}s/{\varphi }^2$. If $s=0$, the same basic inequality forces $\hat{\beta }=0$ on $\Omega$; no nonempty-support compatibility constant is needed.

For the support-size argument, use the stated prediction bound with constant $16$. Set $b=|B|$. By the Cauchy-Schwarz inequality and the definition of the sparse maximum eigenvalue,Combining with the lower bound and squaring gives $\boxed{b\le 64{\kappa }_b^{2}s/{\varphi }^2}$ for every nonempty $B\subseteq \hat{S}$. If $m_{\ast }<\infty$ and $\hat{s}\ge m_{\ast }$, choose $B$ of size $m_{\ast }$; this contradicts the strict inequality defining $m_{\ast }$. If $m_{\ast }=\infty$, $\hat{s}\le p<\infty$ proves the assertion directly. Hence $\boxed{\hat{s}<m_{\ast }}$ in both cases.

When $1<m_{\ast }\le p$, minimality and monotonicity of the sparse maximum eigenvalues giveWhen $m_{\ast }=1$, the first result gives $\hat{s}=0$, so the same final bound holds. Thus the finite-index conclusion is $\boxed{\hat{s}\le 64{\kappa }_{m_{\ast }}^{2}s/{\varphi }^2(m_{\ast }<\infty )}$.

There is a genuine domain omission in the printed last assertion: the definition of ${\kappa }_m$ only makes sense for $1\le m\le p$, so ${\kappa }_{m_{\ast }}$ is undefined when $m_{\ast }=\infty$. This case can occur: take centered orthogonal columns with $X^{T}X/n=I_p$, $n\ge p+1$, $s=p$, and ${\varphi }^2=1$. Then ${\kappa }_m^2=1$ for every available $m$ and none exceeds $64p$, so $m_{\ast }=\infty$. A universally defined replacement, from $B=\hat{S}$ and monotonicity, isAlternatively, explicitly extend the definition by ${\kappa }_m={\kappa }_p$ for $m>p$, including infinity. Under that added convention the printed final expression is meaningful and follows from this replacement bound. Neither convention nor finiteness should be silently assumed.

Use the convention that ridge regression minimizes $\Vert Y-Xb{\Vert }_2^2/(2n)+(\gamma /2)\Vert b{\Vert }_2^2$. Its closed-form ridge regression estimator isThe factors $n$ and $1/2$ here specify the tuning convention. No intercept is needed for the centred responses and predictors.

For the two-component procedure, fix $\delta$ and put $\theta =Y-X\delta$. Differentiating the objective with respect to its dense coefficient $\beta$ givesThus, with $Q=(X^{T}X+2n{\lambda }_2{I}_p{)}^{-1}$,The inverse exists because ${\lambda }_2>0$, even for a rank-deficient design. The sum of the fitted sparse and dense components is the Lava estimator.

Multiplying by $2n$ and completing the square gives the exact profile calculationLet $B=I_n-XQ{X}^T$. A singular value decomposition of $X$ shows that its eigenvalues are $2n{\lambda }_2/(r_i^2+2n{\lambda }_2)$ in nonzero singular directions and one in the orthogonal complement. Hence $B$ is a positive-definite matrix. Take its symmetric principal square root of a positive semidefinite matrix, $A=B^{1/2}$. Profiling out $\beta$ therefore yields the Lasso reduction for the Lava estimator:No restandardization of $AX$ is needed: this is the exact transformed objective with its original penalty.

Write $\tilde{X}=AX$, $h={\hat{\delta }}_{\lambda }-{\delta }^0$, and $\eta =\tilde{X}{\beta }^0+A\epsilon$. Then $AY=\tilde{X}{\delta }^0+\eta$. Comparing the transformed Lasso objective at ${\hat{\delta }}_{\lambda }$ and ${\delta }^0$ gives the Basic inequality for the LassoOn $\Omega$, Holder inequality bounds the score by ${\lambda }_1\Vert h{\Vert }_1$. Using $\Vert h{\Vert }_1\le \Vert {\hat{\delta }}_{\lambda }{\Vert }_1+\Vert {\delta }^0{\Vert }_1$ cancels the fitted penalty and yields the slow-rate prediction bound for the Lasso:Finally, if $\kappa ={\lambda }_{\max }(XQ{X}^T)<1$, then $B⪰(1-\kappa )I_n$, soConsequentlyPositive ${\lambda }_2$ actually guarantees $\kappa <1$ for every finite design; the factor measures how much prediction norm the profiling transformation can remove.

For a convex function $f$, its subdifferential at $x$ is the set of supporting slopesThe subgradient optimality condition isIndeed, the defining inequality with $g=0$ is exactly the global minimum inequality. For the subdifferential of the L1 norm, the coordinates vary independently:

Let $r=Y-X{\hat{\beta }}_{\lambda }$. The Karush-Kuhn-Tucker conditions for Lasso give a vector $z\in \partial \Vert {\hat{\beta }}_{\lambda }{\Vert }_1$ such that $X^{T}r/n=\lambda z$. If ${\hat{\sigma }}_{\lambda }=\Vert r{\Vert }_2/\sqrt{n}>0$, the unsquared residual norm is differentiable at this fit, with gradientUsing the subdifferential sum rule, choose $\gamma (\lambda )=\lambda /{\hat{\sigma }}_{\lambda }$ to obtainConvexity now proves the tuning relation between Lasso and square-root Lasso:The positive residual assumption is needed both for this gradient and for the quotient defining the tuning parameter.

For the final score calculation, use the usual fixed-design interpretation of the normal linear model: regard $X,Z$ and the tuning parameter $\gamma$ in the regression of $Z$ as fixed, and choose that regression's minimizer using only those inputs. Its nonzero residual $R=R_{\gamma }$ then has the Square-root Lasso optimality conditionExpanding the response residual givesSince $R/\Vert R{\Vert }_2$ is a fixed unit vector and $\epsilon \sim N_n(0,{\sigma }^2{I}_n)$, a linear image of a multivariate normal vector gives $W\sim N(0,{\sigma }^2)$. Finally, Holder inequality and the residual score bound yieldThis is the residual-score decomposition from square-root Lasso. The normality statement also holds conditionally on a design independent of the noise. If $\gamma$ were chosen from the response noise, for example through the preceding $\gamma (\lambda )$ formula, the direction $R$ would require a separate independence argument; the final calculation treats $\gamma$ as fixed.

The moral graph of a Directed acyclic graph is formed by joining any two parents of a common child and then removing the directions from all edges. For D-separation, a path in the underlying undirected graph is active given $S$ when every noncollider on it is outside $S$, and every collider has itself or a descendant in $S$. Disjoint vertex sets $A$ and $B$ are D-separated by $S$ if no such active path connects them.

The adjacency characterization isA one-edge path cannot be blocked by conditioning on other vertices. For a nonadjacent pair, choose the later vertex $v$ in a topological ordering. The earlier vertex $u$ is a nondescendant and is not a parent of $v$, so the parents of $v$ separate the pair by the local Markov property of a directed acyclic graph.

Use faithfulness of a directed acyclic graph to mean that graphical and probabilistic independences agree, for all disjoint sets:This includes the global Markov property of a directed acyclic graph as well as the absence of extra independences. The conditional independence graph of $P$ is the undirected graph in which $u-v$ is absent exactly when $Z_u\perp Z_v\mid Z_{V\setminus \{u,v\}}$.

To identify this graph, condition on every vertex other than $u,v$. Any noncollider in the interior of a path is then conditioned on, so an active path must have only colliders internally. Two adjacent interior vertices cannot both be colliders: the arrow joining them has an arrowhead at only one of its ends. Thus an active path can have no interior vertex, giving a direct edge, or just one, giving $u\to w\leftarrow v$. In the latter case the conditioned common child $w$ activates the path. These possibilities are exactly the edges added or retained by moralization. Faithfulness therefore provesThis is why the moral graph equals the conditional independence graph under faithfulness.

For Gaussian graph estimation, collect the data into an $n\times p$ matrix $D$ and let $S_n=n^{-1}{\sum }_i{x}_i{x}_i^T$. The precision matrix $\Theta ={\Sigma }^{-1}$ has ${\Theta }_{jk}=0$ precisely for absent graph edges, by Gaussian conditional independence. In neighbourhood selection, fit a Lasso regression of each column on the others:Select the nonzero coefficients as neighbours of $j$, then make the graph undirected using either the OR rule, retaining an edge selected in at least one regression, or the AND rule, requiring both. These are instances of Nodewise Lasso.

The Graphical Lasso estimates the whole precision matrix jointly, for example byRead off its nonzero off-diagonal entries as graph edges. This expression uses an off-diagonal penalty; an all-entry penalty is another convention.

To reduce computation in the PC algorithm, start its skeleton search from an estimate of the conditional independence graph rather than the complete graph. The exact graph contains the true DAG skeleton, so oracle tests can remove its extra edges while retaining the true adjacencies. Fewer initial edges and smaller neighbourhoods reduce both candidate pairs and candidate conditioning sets.

For correctness of the orientation stage, preserve valid separating-set records. With an exact conditional independence graph, an initially absent pair $u,v$ has the full complement $V\setminus \{u,v\}$ as a separating set. Record that set, and record the usual discovered sets for edges removed later. Leaving an initially absent pair with an empty default could falsely mark an unshielded noncollider as a collider. This describes PC initialization from a conditional independence graph; with an estimated initial graph, screening errors can propagate into the output.

A vector $g$ is a subgradient of a convex function $f$ at $x$ whenA point $x$ minimizes $f$ exactly when $0\in \partial f(x)$. For absolute value,Coordinate descent repeatedly minimizes the objective over one coordinate while holding the others fixed, cycling through coordinates until convergence.

The Lasso solvesFor the first update, define the partial residual and scoreSince $\Vert X_1{\Vert }_2^2=n$, the coordinate objective differs by a constant fromIts subgradient condition gives the soft thresholding update

For the stated Berhu penalty, the derivative is $\mathrm{sgn}(t)$ when $0<|t|\le \delta$ and $t/\delta$ when $|t|>\delta$. The coordinatewise Karush-Kuhn-Tucker conditions therefore give

The two Lasso problems areandIf $\hat{\beta }$ solves the first problem, the complete solution set of the second isIndeed, the feasible totals are exactly $[-t,t]$, the same as in the first problem.

The Lasso minimizesIts Karush-Kuhn-Tucker conditions areSince the columns of $X$ are centered, $X^T(ϵ-\overline{ϵ}1)=X^Tϵ$. Taking the inner product of the KKT equation with ${\beta }^0-\hat{\beta }$ and usinggives

Put $\delta =\hat{\beta }-{\beta }^0$. On $\Omega$,Using ${\beta }_N^0=0$ andin the basic inequality yieldsIn particular $\delta$ lies in the Lasso cone condition.

The assumed restricted eigenvalue condition and $\Vert {\delta }_S{\Vert }_1\le \sqrt{s}\Vert \delta {\Vert }_2$ giveCanceling one prediction-norm factor and applying the restricted eigenvalue condition again gives

ChooseEvery null coordinate satisfies $|{\hat{\beta }}_j|<\tau$. For $j\in S$,Thus ${\hat{S}}^{\tau }=S$ on $\Omega$.

For a Laplace prior $\varphi (u)\propto e^{-\lambda |u|}$, maximizing the posterior is equivalent to minimizingso the posterior mode is the Lasso. For a Gaussian prior $\varphi (u)\propto e^{-\lambda u^2/2}$, the mode minimizesand is the ridge regression estimator $(X^{T}X+\lambda I{)}^{-1}{X}^{T}Y$. Gaussian conjugacy makes the posterior normal, so its mode and posterior mean coincide at this ridge estimate.

With the convention matching the constants below, the Lasso estimator is any minimizerBecause $X^{T}1=0$, the centered noise has the same score as $\epsilon$: $X^T(\epsilon -\overline{\epsilon }1)=X^T\epsilon$. Each $X_j^T\epsilon /n$ is sub-Gaussian with scale $1/\sqrt{n}$, soFor $\lambda =A\sqrt{\log p/n}$, a union bound over the $p$ columns yieldsCall this event $\mathcal{T}$.