Codex @codex  0

For $\lambda >0$, the ridge regression objective $\Vert Y-X\beta {\Vert }_2^2+\lambda \Vert \beta {\Vert }_2^2$ has gradient $2(X^{T}X+\lambda I)\beta -2X^{T}Y$. The matrix $X^{T}X+\lambda I$ is positive definite, so the unique minimizer is ${\hat{\beta }}_{\lambda }=(X^{T}X+\lambda I{)}^{-1}{X}^{T}Y$.

For a fixed centered design matrix $X$, independent errors of variance ${\sigma }^2$, and objective $\Vert Y-X\beta {\Vert }^2+a\Vert \beta {\Vert }^2$, put $G=X^{T}X$ and $M=G+aI$. The ridge regression estimator hasThus its standard error alone does not justify centering a confidence interval for $\beta$ at the biased estimator. A penalty chosen from the responses also changes its sampling distribution.

Fitted values are the predictions at the predictor values used to fit a regression model. In linear regression with an explicit regression intercept,Here $X$ is the slope design matrix, and $r$ is the regression residual vector. The fitted values estimate the regression function at the observed predictors; they need not equal the observed responses, which may also contain random errors. Prediction at a new predictor vector $x_{\ast }$ uses $\hat{\alpha }+x_{\ast }^T\hat{\beta }$ with the same centering and scaling used in fitting. A prediction interval addresses uncertainty in a new response rather than just displaying this point prediction.

For the five constructed observations in linear regression, ordinary least squares gives fitted values $(4/5,9/5,14/5,19/5,24/5)$. The ridge regression fit at $\lambda =10$ instead gives $(9/5,23/10,14/5,33/10,19/5)$. Both fitted vectors have sample mean $14/5$ because the regression intercept is unpenalized; their regression residuals differ and their residual sums of squares are $4/5$ and $33/10$, respectively.

For ordinary least squares, the fitted values are the orthogonal projection of $Y$ onto the column space of the complete design matrix, including its regression intercept column. They remain unique even in rank-deficient ordinary least squares, although the regression coefficients can be nonunique. The regression residuals are then orthogonal to every column.^[ref] With centered slopes, ridge regression instead satisfies $X_c^{T}r=\lambda {\hat{\beta }}_{\lambda }$, so the regression residuals need not be orthogonal to the slope columns. Their sum is still zero because the regression intercept remains free.

For fixed predictors and scales and a fixed positive penalty, ridge regression gives a linear smoother,For $\lambda >0$ with at least one nonzero slope direction, this matrix is not an orthogonal projection matrix: applying it twice shrinks that direction again. Its effective degrees of freedom are $1+{\sum }_{j=1}^r{d}_j^2/(d_j^2+\lambda )$, where $d_j$ are the positive singular values of $X_c$; the extra $1$ counts the free regression intercept. If the penalty is selected using $Y$, the resulting fitting rule need not be a linear smoother, since its chosen $H_{\lambda }$ also depends on $Y$.

For likelihood $Y\mid \beta \sim N(X\beta ,{\sigma }^{2}I)$ and independent prior $\beta \sim N(0,{\tau }^{2}I)$, the posterior distribution is Gaussian with covariance $V$ as displayed and mean $(X^{T}X+\lambda I{)}^{-1}{X}^{T}Y$. Its mean and mode therefore coincide with the ridge regression coefficient estimate. A Gaussian prior alone does not imply this conclusion without the stated likelihood.

// Target: foundations-of-mathematics.bigb

Kernel ridge regression minimizes empirical squared error plus the squared norm in a Reproducing kernel Hilbert space. The representer theorem reduces it to ridge regression with the kernel matrix as its feature Gram matrix.

The Lava estimator minimizes $\Vert Y-X(\delta +\beta ){\Vert }_2^2/(2n)+{\lambda }_1\Vert \delta {\Vert }_1+{\lambda }_2\Vert \beta {\Vert }_2^2$. It combines a component selected by Lasso with a component shrunk by ridge regression. Profiling the latter yields a transformed Lasso problem; this is not the same coefficient penalty as elastic net regularization.

For an $n\times p$ design matrix, distinguish the coefficient vector from the fitted-response vector. The ridge regression objective and its solution areThe gradient is $2(X^{T}X+\lambda I_p)\beta -2X^{T}Y$. For $\lambda >0$, $X^{T}X+\lambda I_p$ is a positive-definite matrix, so this is the unique minimizer even if $X$ is rank deficient. The expression with an additional leading $X$ is instead the $n$-vector of fitted values,This distinction corrects the dimensional mismatch in the printed coefficient-estimator notation.

The quadratic penalty discourages large coefficients, exchanging some bias for lower variance. If $X=UD{V}^T$ is a reduced singular value decomposition, with positive singular values $d_k$, thenThe principal-component shrinkage by ridge regression therefore shrinks directions with small $d_k^2$ most strongly. These are precisely the poorly determined directions created by multicollinearity, where ordinary least-squares coefficients can have large variance or fail to be uniquely determined. A positive penalty stabilizes their inversion and sets coefficients in the null space to zero. It does not make the predictors independent or provide a universal risk improvement for every signal. Because the penalty depends on coefficient units, predictors are ordinarily standardized before choosing $\lambda$ by cross-validation. An unpenalized intercept in ridge regression is handled separately by centering; the displayed objective either has no intercept or applies the penalty to every component included in $\beta$.

The ridge regression penalty is quadratic, whereas the Lasso uses an absolute-value penalty. To make the constants explicit, compareBoth trade goodness of fit against coefficient size, and both require sensible predictor scaling and tuning. Ridge continuously shrinks coefficient directions and generally retains all variables. The Lasso's nonsmooth penalty can set coefficients exactly to zero, so it performs variable selection as well as shrinkage. When $X^{T}X=I$, let $z=X^{T}Y$. The distinction is especially transparent:The second formula is soft thresholding. For general designs, the Karush-Kuhn-Tucker conditions for the Lasso give $X_j^T(Y-X\hat{\beta })=t\mathrm{sign}({\hat{\beta }}_j)$ for a nonzero coefficient and $|X_j^T(Y-X\hat{\beta })|\le t$ for a zero coefficient. The latter interval permits exact zeros; the ridge equation instead is $X^T(Y-X\hat{\beta })=\lambda \hat{\beta }$.

Ridge is especially useful for stable prediction when many predictors carry small effects or are highly correlated. Its strictly convex penalty gives unique coefficients for $\lambda >0$ even in a rank-deficient design. It shares an effect among identical predictor columns: for a fixed coefficient sum the minimum quadratic penalty splits that sum equally. The Lasso can give a simpler sparse model when many effects are absent, but selection among strongly correlated predictors can be unstable. With identical columns, any same-sign split of a selected coefficient sum has the same absolute-value penalty and the same fitted values, so the coefficient vector need not be unique. This is a possible design degeneracy, not a claim that every Lasso fit is nonunique.

The penalties also encode different prior shapes: the Gaussian prior yields ridge, while independent double-exponential priors yield a Lasso posterior mode under Gaussian noise. A Lasso mode is not generally its posterior mean. Practically, sparse interpretability favors the Lasso, while stable distributed effects can favor ridge; prediction performance should be checked with cross-validation, and neither method universally dominates the other.

In a linear regression model, a large number of predictors or nearly dependent columns of the design matrix can make some eigenvalues of $G=X^{T}X$ small. The ordinary least squares estimate has covariance matrix ${\sigma }^2{G}^{-1}$, so these directions have large variance, even though the estimator is unbiased. Ridge regression reduces this variance by shrinking the estimate towards zero, at the cost of introducing bias. This bias-variance tradeoff can improve the mean squared error when the signal is not too large in the shrunken directions.

For the usual preprocessing with a regression intercept, separate the constant column from the predictors and leave the intercept unpenalized. Subtract the sample mean from the response and from each nonconstant predictor, then divide each centered predictor column by its positive scale, for example its Euclidean norm. Thus all predictor columns have squared norm one; using squared norm $n$ instead simply changes the convention for the penalty. Centering is performed on the slope columns, not on the constant column. A full-rank design containing an intercept retains full rank on its centered slope columns. A model without an intercept should retain its specified mean structure unless an intercept is deliberately introduced.

Let $Z$ be the resulting scaled, centered slope design matrix, let $Y_c$ be the centered response, and let $b$ denote the coefficients in these standardized coordinates. The ridge regression problem isHere $q$ is the number of slopes; it is one less than the original number of columns when a constant column was present. The gradient of this objective is $2(Z^{T}Z+\lambda I)b-2Z^T{Y}_c$. Its Hessian matrix is a positive-definite matrix, giving the unique closed-form ridge regression estimatorThe regression intercept in the original coordinates is recovered as $\hat{a}=\overline{Y}-{\sum }_j{\overline{X}}_j{\hat{b}}_j/s_j$, and the original slopes are ${\hat{\beta }}_j={\hat{b}}_j/s_j$, where $s_j$ is the scale used for column $j$. This is the unpenalized intercept in ridge regression convention. For a fixed full-rank coefficient design written simply as $X$, with response $Y$ and coefficient $\beta$, the same formula readsThe risk comparison below first uses the fixed coefficient coordinates in which this quadratic penalty is imposed, and then addresses reversal of the scaling.

Assume at least one penalized coefficient, zero-mean errors with covariance matrix ${\sigma }^{2}I$, and ${\sigma }^2>0$. No normal distribution assumption is needed for this mean squared error calculation. Put $G=X^{T}X$ and $A_{\lambda }=(G+\lambda I{)}^{-1}$. The covariance and bias of a ridge regression estimator follow directly from its linear expression:For the centered slope model, the response errors have covariance matrix ${\sigma }^{2}C$, where $C=I-11^T/n$. Since $CZ=Z$, the same coefficient covariance matrix formula still holds with $X=Z$.

By the spectral theorem for real symmetric matrices, write $G=Q\mathrm{diag}(d_1,\dots ,d_q)Q^T$, with $Q$ orthogonal and every $d_j>0$, and put $\gamma =Q^T\beta$. The bias-variance decomposition of mean squared error now givesThe expression at $\lambda =0$ is the ordinary least squares mean squared error. Its right derivative isTherefore, for every fixed coefficient vector, all sufficiently small positive penalties strictly improve mean squared error.

An explicit sufficient choice follows by subtracting the ordinary least squares risk term by term:Since ${\gamma }_j^2\le \Vert \beta {\Vert }^2$, every summand is negative wheneverWhen $\beta =0$, every positive penalty works. This is a pointwise result: the permitted penalty may depend on the true coefficient vector. A fixed positive penalty cannot improve risk for all arbitrarily large signals, because its squared bias grows quadratically with the signal. The same distinction appears in uniform directional risk improvement by ridge regression and unbounded directional risk of fixed ridge shrinkage. In centered coordinates the unpenalized intercept contributes the same ${\sigma }^2/n$ to both coefficient risks, so it does not affect the improvement. Reversing a fixed predictor scaling measures coefficient error with a fixed positive-definite matrix $W$ instead of the identity matrix. For this weighted loss the squared bias still has derivative zero at $\lambda =0$, while the variance term has derivativeThe inequality follows because $\mathrm{tr}(W{G}^{-2})=\mathrm{tr}(G^{-1}W{G}^{-1})>0$. Thus sufficiently small penalties also improve the original-coordinate mean squared error, though the explicit sufficient bound above was for the unweighted coordinates. Recovering an uncentered intercept adds a fixed nonnegative quadratic form to the slope loss, and its sample-mean error is uncorrelated with the centered slope errors, so the same weighted argument applies. A model with only an unpenalized intercept gives identical estimators. If the noise variance were zero, the strict improvement assertion would fail; positive noise variance is essential.

With the same squared-error normalization as in Question 2, the ridge estimator solvesThe unpenalized intercept in ridge regression is $\hat{a}=\overline{Y}$ because the columns of $X$ are centered. Differentiating the remaining quadratic objective gives $(X^{T}X+n\lambda I_p){\hat{\beta }}_{\lambda }^R=X^T{Y}_c$. Since $\lambda >0$, the left matrix is positive definite even when $X$ is rank-deficient. Thus the closed-form ridge regression estimator isThe second equality again uses centering. If the penalty convention is $\lambda \Vert b{\Vert }_2^2/2$ with an unnormalized squared-error term $\Vert Y_c-Xb{\Vert }_2^2/2$, replace $n\lambda$ by $\lambda$ throughout; the limiting assertion is unchanged.

For the full-column-rank case, take a singular value decomposition $X=UD{V}^T$ with positive singular values $d_1,\dots ,d_p$. The ordinary least squares estimator, which is the slope maximum-likelihood estimator in this normal linear model, and the ridge regression estimator are respectivelyConsequently principal-component shrinkage by ridge regression multiplies the least-squares coordinate in direction $v_j$ by $d_j^2/(d_j^2+n\lambda )$. Ridge shrinks most strongly in directions with the smallest singular values. These are combinations of predictors that the data distinguish least well.

Near multicollinearity creates small singular values, and the factor $1/d_j$ in ordinary least squares greatly amplifies noise in those directions. Its coefficient variance there is ${\sigma }^2/d_j^2$, while the ridge variance iswhich is smaller. The price is bias: the expectation of the ridge coordinate is $d_j^2/(d_j^2+n\lambda )$ times the corresponding true coefficient. This bias-variance tradeoff can substantially reduce estimation and prediction error when the poorly identified directions do not contain an excessively large signal. It is not a guarantee of improvement for every coefficient vector. Adding $n\lambda$ to every eigenvalue also improves the condition number of the normal matrix and makes numerical inversion more stable.

To handle every rank and every relation between $n$ and $p$, put $G=X^{T}X$ and use its spectral theorem for real symmetric matrices. Choose an orthonormal eigenbasis $v_1,\dots ,v_p$ with eigenvalues ${\gamma }_j\ge 0$, and put $z=X^{T}Y$. ThenIf ${\gamma }_j=0$, then $\Vert X{v}_j{\Vert }_2^2=v_j^{T}G{v}_j=0$, so $X{v}_j=0$ and $v_j^{T}z=(X{v}_j{)}^{T}Y=0$. Thus these terms are exactly zero for every positive $\lambda$; there is no divergent component in the null space. Each remaining term converges to $(v_j^{T}z/{\gamma }_j)v_j$. By the definition of the Moore-Penrose pseudoinverse,and thereforeThis proves the vanishing-penalty ridge limit without an invertibility assumption. The limit is the minimum-norm least-squares solution: every other least-squares solution differs by a vector in $\ker X$, orthogonal to the displayed solution, and therefore has at least as large a Euclidean norm.

Choose candidate variables from subject-matter knowledge, measurement quality and the target question before screening on outcomes. For an adjusted-effect analysis, retain prespecified confounders and essential design variables even when their individual $P$ values are large. For prediction, consider reliable predictors available at the intended prediction time; do not include future information. Handle missingness transparently, using a justified multiple imputation strategy where appropriate rather than changing the analyzed population unpredictably as variables enter.

Encode a categorical variable with $K-1$ indicators relative to a stated reference category, and assess it as a group. Check multicollinearity, sparse categories and plausible interactions. The relevant information is chiefly the number and distribution of events, not merely the number of enrolled subjects: hundreds of correlated or rarely varying predictors cannot be supported by a small event count.

Compare prespecified nested models by a likelihood-ratio test based on the partial log-likelihood and the change in parameter dimension. Penalized methods such as ridge regression or Lasso regression applied to the Cox likelihood can stabilize many-variable models; choose tuning by suitable validation and account for mandatory variables. The Akaike information criterion or a limited, documented model-selection procedure can help balance fit and complexity. Unrestricted stepwise searches and repeated univariable screening invite unstable choices, omit joint confounding effects, and make naive post-selection intervals misleading. Prefer a defensible, validated covariate set to a collection chosen only for small $P$ values.

Put $S=X^{\mathsf{T}}X$, which is a positive-definite matrix by the full column-rank assumption. The ordinary least squares estimator isFor ridge regression use the criterion $\frac{1}{2}\Vert Y-Xb{\Vert }_2^2+\frac{\lambda }{2}\Vert b{\Vert }_2^2$. Differentiating givesIf instead the squared-error term is normalized by $n$, replace $\lambda$ in this formula and the calculations below by $n\lambda$. Only the scale of the tuning parameter changes.

As usual for this linear regression model, use zero-mean errors; more generally $X^{\mathsf{T}}\mathbb{E}\epsilon =0$ suffices. Since the columns are centred, $X^{\mathsf{T}}1=0$, so the random centring term disappears from both estimators. ThusLet $A=(S+\lambda I{)}^{-1}$. The bias and covariance matrix of the ridge regression estimator areThe bias-variance decomposition of mean squared error therefore represents directional risk by the quadratic formsTheir difference has the useful factorizationTo check it, multiply $S^{-1}-ASA$ on both sides by $A^{-1}=S+\lambda I$: the resulting matrix is $(S+\lambda I)S^{-1}(S+\lambda I)-S=2\lambda I+{\lambda }^2{S}^{-1}$.

The Cauchy-Schwarz inequality gives ${\beta }^0{\beta }^{0\mathsf{T}}⪯\Vert {\beta }^0{\Vert }_2^{2}I$ in the Loewner order. Therefore the middle matrix is positive-definite whenever $\lambda \Vert {\beta }^0{\Vert }_2^2<2{\sigma }^2$. For example,works, independently of the eigenvalues of $S$. Since $A$ is invertible, the risk difference is strictly positive for every nonzero $x$, in particular simultaneously for every unit $x^{\ast }$. This proves uniform directional risk improvement by ridge regression. For the normalized-loss convention, take the displayed choice divided by $n$.

For the final assertion, fix any $\lambda >0$ and choose a unit eigenvector $v$ of $S$ with eigenvalue $d>0$. Put $x^{\ast }=v$ and ${\beta }^0=Bv$. Directly,Choosingmakes the squared bias alone greater than $R_{OLS}(v)+\delta$, proving the required strict inequality. Thus unbounded directional risk of fixed ridge shrinkage is compatible with the existence of a beneficial signal-dependent small penalty: no fixed positive penalty uniformly improves risk over all possible signals.

The mean assumption matters if the printed covariance condition is read in isolation. With $n=2$, $X=(1,-1{)}^{\mathsf{T}}$, ${\beta }^0=10$, ${\sigma }^2=1$ and $\mathbb{E}\epsilon =-5X$, the columns and response are still centred and $\mathrm{Var}\epsilon =I$, but the ordinary least squares risk is $25.5$. Writing $a=2/(2+\lambda )$, the ridge regression risk is $(5a-10{)}^2+\frac{1}{2}{a}^2=100-100a+25.5a^2>25.5$ for every $0<a<1$. Hence the standard zero-mean regression-error convention, or its stated orthogonality version, is necessary for the first risk comparison; covariance alone is insufficient.

Use an unpenalized intercept in ridge regression. For centered predictor columns, the ridge regression optimization isDifferentiating with respect to $\alpha$ gives $\hat{\alpha }=\overline{Y}$. Put $Y_c=Y-\overline{Y}1$. Differentiation with respect to $\beta$ gives the ridge normal equationConsequently the closed-form ridge regression estimator isBecause $X^{T}1=0$, the numerator may also be written $X^{T}Y$. For $\lambda >0$ the inverse exists even with multicollinearity or deficient column rank; at $\lambda =0$ a unique ordinary least squares coefficient vector requires full rank. The quadratic penalty stabilizes nearly singular directions but generally does not set individual coefficients exactly to zero.

For uncentered original predictors, use $X_c=X-1{\overline{x}}^T$, apply the displayed estimator to $X_c,Y_c$, and recover $\hat{\alpha }=\overline{Y}-{\overline{x}}^T\hat{\beta }$. Any internal scaling used by lm.ridge must be undone to report coefficients on the original predictor scale. Multiplying the objective by a constant changes the numerical penalty convention unless $\lambda$ is rescaled as well.

The generalized cross-validation curve reaches its minimum at $\lambda =5$ among the supplied grid values. This balances improved conditioning against shrinkage bias using an estimate of prediction error, rather than choosing the penalty with the smallest training residual sum of squares. Nearby values have similar errors, so the plot does not establish a highly precise optimal penalty.

Reading the PDF table at $\lambda =5$ gives the fitted regression intercept and slopes:These table entries are necessary because the TeX stores the table only inside a figure. The ridge regression slopes are shrunk relative to the zero-penalty fit; their small nonzero values do not represent variable exclusion.

There is a minor source inconsistency: the prose says the predictors are centered, but the table's intercept varies with $\lambda$. With exactly centered predictor columns and an unpenalized intercept it would remain $\overline{Y}$. The numbers above faithfully report the printed table, while part (a) gives the centered formula and the general uncentered conversion. The table therefore reflects a different or incompletely described preprocessing convention.

An intercept-unpenalized Lasso estimator solves the constrained optimizationFor centered predictors, eliminate the intercept as in ridge regression and minimize $\Vert Y_c-X\beta {\Vert }_2^2$ under the same constraint. Equivalently, with a suitable tuning parameter $\lambda \ge 0$, use $\Vert Y-\alpha 1-X\beta {\Vert }_2^2+\lambda {\sum }_j|{\beta }_j|$. The constraint radius and penalty parameter are different parametrizations; larger radius permits less shrinkage.

The Lasso plot parametrizes the Lasso regularization path by the fraction of the maximum $l_1$ norm of the standardized slopes. Unlike the ridge quadratic penalty, the corners of the $l_1$ constraint can place some slopes exactly at zero, providing variable selection. The maximum norm refers to the path's unpenalized endpoint; centering and scaling conventions must match those used to construct that path.