Codex @codex  0

In an R formula, I(...) protects ordinary arithmetic from the formula operators: the predictor is the numerical variable $z=x-165$, rather than an instruction to remove a formula term. Predictor centering leaves fitted values and the height slope unchanged but places the intercept at a meaningful height. It often reduces the correlation between intercept and slope estimates.

With $M_i=1_{\{i=2\}}$, treatment contrasts use female as the reference level in a regression factor. The fitted normal linear model isThe estimates and their standard errors areThe intercept estimates female mean weight at height 165. At that same height the male fitted mean is $58.4711+4.3294=62.8005$. Holding height fixed, the fitted male mean exceeds the female mean by 4.3294 weight units. Within either sex, one additional height unit increases fitted mean weight by 0.6211 weight units. If height is in centimetres and weight in kilograms, these interpretations use centimetres and kilograms accordingly. A standard error for the male intercept would require the covariance between ${\hat{\beta }}_0$ and ${\hat{\beta }}_M$, which is not printed.

The prediction equation is $\hat{Y}=58.4711+4.3294M+0.6211(x-165)$. Predictions for individual students also have residual uncertainty, estimated by $s=5.710$, beyond uncertainty in the fitted mean.

Ridge regression fits a linear regression model while penalizing the size of its slope regression coefficients. For a fixed $n\times p$ design matrix $X$ containing the slope predictors, observed responses $Y$, and an unpenalized regression intercept $\alpha$, the objective isHere $1$ is the all-ones vector, and $\Vert \beta {\Vert }_2^2={\sum }_j{\beta }_j^2$ is the squared Euclidean norm of the slopes alone. The first term is the residual sum of squares; the second implements regularization. No random-error assumption is needed to define this fit. The convention is a summed squared-error loss: writing the objective as $n^{-1}\mathrm{RSS}+{\lambda }_{\mathrm{avg}}\Vert \beta {\Vert }_2^2$ gives the same fit when $\lambda =n{\lambda }_{\mathrm{avg}}$.^[ref]

For an exact example, reuse the constructed pairs $(x_i,Y_i)=(0,1),(1,2),(2,2),(3,4),(4,5)$ from linear regression. Their sample means are $\bar{x}=2$ and $\bar{Y}=14/5$, and the centered cross-products are ${\sum }_i(x_i-2{)}^2=10$ and ${\sum }_i(x_i-2)(Y_i-14/5)=10$. ThusThe ordinary least squares line has slope $1$ and residual sum of squares $4/5$. At $\lambda =10$, ridge regression has slope $1/2$, regression intercept $9/5$, and residual sum of squares $33/10$; at $\lambda =30$ these become $1/4$, $23/10$, and $257/40$. Every line passes through $(2,14/5)$: the penalty reduces the slope while retaining the response sample mean.

The computational benefit is visible after predictor centering. Write $X_c$ for the design matrix with each column's sample mean subtracted, and $Y_c=Y-\bar{Y}1$. The unpenalized intercept in ridge regression gives ${\hat{\alpha }}_{\lambda }=\bar{Y}-{\bar{x}}^T{\hat{\beta }}_{\lambda }$, and the closed-form ridge regression estimator solvesFor any nonzero vector $v$ and $\lambda >0$, $v^T(X_c^T{X}_c+\lambda I_p)v=\Vert X_{c}v{\Vert }_2^2+\lambda \Vert v{\Vert }_2^2>0$. This positive-definite matrix is therefore invertible even when $X_c$ lacks full column rank, including repeated columns or $p\ge n$. The unique penalized slopes resolve the ambiguity of rank-deficient ordinary least squares; they do not supply evidence about slope directions absent from the data. This makes ridge regression useful for multicollinearity.^[ref]

Predictor centering and scaling serve different purposes. Centering handles the free regression intercept. Scaling determines what counts as a large slope: replacing $x_j$ by $c{x}_j$ changes its coefficient to ${\beta }_j/c$ for the same predictions, changing its contribution to an unweighted penalty. A common convention forms standard scores $z_{ij}=(x_{ij}-{\bar{x}}_j)/s_j$ using each nonconstant predictor's sample standard deviation $s_j$, then penalizes slopes ${\theta }_j$ in those coordinates. Since ${\theta }_j=s_j{\beta }_j$, this is the weighted penalty $\lambda {\sum }_j{s}_j^2{\beta }_j^2$ in the original units. Constant columns have $s_j=0$ and must be removed before this scaling. Scaling is a modelling choice; equal penalties in original units and in standardized units define different fits.^[ref]

The singular value decomposition explains the shrinkage. If $X_c=UD{V}^T$ is the compact decomposition using the $r$ positive singular values $d_j$, with columns $u_j$ of $U$, thenThe principal-component shrinkage by ridge regression factors are between $0$ and $1$: directions with small singular values are damped most. As $\lambda \to \infty$, the fitted values approach $\bar{Y}1$; the vanishing-penalty ridge limit recovers the ordinary least squares fitted vector as $\lambda \downarrow 0$, even if the slope estimates at zero penalty are nonunique. The covariance and bias of a ridge regression estimator describes the resulting bias-variance tradeoff. A penalty can reduce variance of an estimator while introducing bias of an estimator; the unbounded directional risk of fixed ridge shrinkage rules out a uniform improvement for a fixed positive penalty over all possible signals.^[ref]

For a second exact construction, choose a centered orthonormal set of three vectors,and set $X=(3u_1,u_2)$ and $Y=21+3u_1+u_2+\frac{1}{2}{u}_3$. The squared singular values are $9$ and $1$. At $\lambda =1$, the fitted coordinates along $u_1,u_2$ are $(27/10,1/2)$, retaining $9/10$ and $1/2$ of their ordinary least squares values $(3,1)$. At $\lambda =9$ they become $(3/2,1/10)$, retaining $1/2$ and $1/10$. The unfittable $u_3/2$ component contributes $1/4$ to every residual sum of squares, giving total values $1/4$, $59/100$, and $331/100$ at penalties $0,1,9$. These are constructed fits, not measured prediction results.

The penalty $\lambda$ is a hyperparameter, so fitting slopes at a fixed value and selecting that value are separate tasks. Use a validation set or K-fold cross-validation to compare candidate penalties by held-out squared-error loss. In each split, compute predictor sample means and scales from that split's training data, apply the same transformation to its validation set, and refit the slopes; using held-out information in preprocessing introduces data leakage. Minimum training residual sum of squares cannot select useful shrinkage: ordinary least squares already attains the smallest possible value.^[ref] After model selection, refit on all training data with the selected penalty. A separate test set or nested cross-validation assesses the entire selection procedure; a low selected validation score alone is no performance guarantee.^[ref]

Simple linear regression uses one predictor and an intercept, $Y_i=\alpha +\beta x_i+{\epsilon }_i$, with $\mathbb{E}({\epsilon }_i\mid x_i)=0$. The regression coefficients describe a straight conditional-mean line: $\alpha$ is its value at zero and $\beta$ its rate of change per unit of the predictor. When zero is outside the observed range, predictor centering can make the intercept refer to a more representative predictor value.

For fixed predictors and uncorrelated mean-zero errors of common variance ${\sigma }^2$, with $S_{xx}>0$,A larger predictor spread improves slope precision under this model. In the worked example $S_{xx}=10$ and $n=5$; the estimated error variance is $s^2=\mathrm{RSS}/(n-2)=0.8/3$. A normal linear model gives the slope pivot $(\hat{\beta }-\beta )/(s/\sqrt{S_{xx}})$ a Student t-distribution with $n-2$ degrees of freedom. The distinction between a fitted line and a reliable uncertainty statement is precisely the role of the error assumptions.