Codex @codex  0

If each level of one categorical variable belongs to exactly one level of another, indicators for the coarser levels are sums of indicators for the finer levels. Including unrestricted fixed effects for both therefore gives a rank-deficient ordinary least squares: the separate effects lack identifiability without additional constraints.

Fitted values are the predictions at the predictor values used to fit a regression model. In linear regression with an explicit regression intercept,Here $X$ is the slope design matrix, and $r$ is the regression residual vector. The fitted values estimate the regression function at the observed predictors; they need not equal the observed responses, which may also contain random errors. Prediction at a new predictor vector $x_{\ast }$ uses $\hat{\alpha }+x_{\ast }^T\hat{\beta }$ with the same centering and scaling used in fitting. A prediction interval addresses uncertainty in a new response rather than just displaying this point prediction.

For the five constructed observations in linear regression, ordinary least squares gives fitted values $(4/5,9/5,14/5,19/5,24/5)$. The ridge regression fit at $\lambda =10$ instead gives $(9/5,23/10,14/5,33/10,19/5)$. Both fitted vectors have sample mean $14/5$ because the regression intercept is unpenalized; their regression residuals differ and their residual sums of squares are $4/5$ and $33/10$, respectively.

For ordinary least squares, the fitted values are the orthogonal projection of $Y$ onto the column space of the complete design matrix, including its regression intercept column. They remain unique even in rank-deficient ordinary least squares, although the regression coefficients can be nonunique. The regression residuals are then orthogonal to every column.^[ref] With centered slopes, ridge regression instead satisfies $X_c^{T}r=\lambda {\hat{\beta }}_{\lambda }$, so the regression residuals need not be orthogonal to the slope columns. Their sum is still zero because the regression intercept remains free.

For fixed predictors and scales and a fixed positive penalty, ridge regression gives a linear smoother,For $\lambda >0$ with at least one nonzero slope direction, this matrix is not an orthogonal projection matrix: applying it twice shrinks that direction again. Its effective degrees of freedom are $1+{\sum }_{j=1}^r{d}_j^2/(d_j^2+\lambda )$, where $d_j$ are the positive singular values of $X_c$; the extra $1$ counts the free regression intercept. If the penalty is selected using $Y$, the resulting fitting rule need not be a linear smoother, since its chosen $H_{\lambda }$ also depends on $Y$.

An unbiased estimator $\hat{\theta }$ satisfies ${\mathbb{E}}_{\theta }\hat{\theta }=\theta$ for every parameter value. Under the standard full-column-rank assumption on the design matrix, the Gaussian log-likelihood for the normal linear model, up to terms independent of $\beta$, is $-\Vert Y-X\beta {\Vert }^2/(2{\sigma }^2)$. Its maximum is the ordinary least squares minimum. Differentiating gives $X^{T}X\hat{\beta }=X^{T}Y$, and full column rank makes $X^{T}X$ positive definite. ThusSince $\mathbb{E}Y=X\beta$, its expectation is $(X^{T}X{)}^{-1}{X}^{T}X\beta =\beta$, proving unbiasedness. Estimating an unknown ${\sigma }^2$ as well does not change the maximizing coefficient vector.

The printed hypotheses give $p<n$ but do not explicitly require full column rank. Without that condition the maximum-likelihood estimators are not unique: they are $X^{+}Y+z$ with $z\in \ker X$, as in rank-deficient ordinary least squares. If $0\ne z\in \ker X$, the observation distributions at $\beta$ and $\beta +z$ are identical. Every estimator then has the same expectation at both parameters, so no estimator can be unbiased for both entire coefficient vectors. This proves nonidentifiability prevents unbiased coefficient estimation. The stated estimator and unbiasedness claim require the usual full-rank condition.

The violated assumption is independence of random variables across observations: repeated measurements on the same rat form clustered data. A persistent rat-specific weight difference induces positive within-rat covariance, even after adjustment for time and diet. Treating all 176 measurements as independent is pseudoreplication.

For the claimed lack of identifiability in the second fit, Rat must be a categorical variable, as intended by the rat-effect interpretation. Each rat receives only one diet, so this is confounding of nested fixed factors. If $z_r$ is the rat-indicator column, the diet-$d$ indicator is ${\sum }_{r:D_r=d}{z}_r$. Thus a diet effect can be increased by $c_d$ and every corresponding rat effect decreased by $c_d$, leaving all fitted values unchanged. With an intercept, two diet columns, fifteen rat contrasts and time, there are 19 columns but only 17 independent columns: the intercept and rat contrasts already span the diet columns, while time varies within rats. This gives rank-deficient ordinary least squares.

The first model ignores within-rat dependence; the second cannot separate unrestricted fixed diet and rat effects. If Rat were instead encoded as a single numeric predictor, the asserted rank deficiency would not follow merely from nesting.

The Gaussian likelihood is proportional toso maximum likelihood estimation is equivalent to ordinary least squares. If $p\le n$ and $X$ has rank $p$, then $X^{T}X$ is invertible and the normal equations giveIf $p>n$, then $\mathrm{rank}X<p$ and $\ker X\ne \{0\}$. A least-squares minimizer always exists, but it is not unique: by rank-deficient ordinary least squares, the complete set isan affine subspace of dimension $p-\mathrm{rank}X>0$. Thus there are uncountably infinitely many likelihood maximizers. In the generic full-row-rank case their dimension is $p-n$.

Compute the two sides of the normal equations for linear least squares and divide by $n$:The first equation gives $\alpha =\overline{y}-\beta \overline{x}$. Substitute this into the second and solve:These simple linear regression coefficients are unique provided the observed distances are not all equal, sinceThe condition $n\ge 2$ alone does not guarantee this. If all $x_i=x_0$, the design matrix has matrix rank one and the valid alternative is the familyIn this rank-deficient ordinary least squares case, the fitted predictions are uniquely $\overline{y}$ but the two parameters cannot be separately determined.

Ridge regression fits a linear regression model while penalizing the size of its slope regression coefficients. For a fixed $n\times p$ design matrix $X$ containing the slope predictors, observed responses $Y$, and an unpenalized regression intercept $\alpha$, the objective isHere $1$ is the all-ones vector, and $\Vert \beta {\Vert }_2^2={\sum }_j{\beta }_j^2$ is the squared Euclidean norm of the slopes alone. The first term is the residual sum of squares; the second implements regularization. No random-error assumption is needed to define this fit. The convention is a summed squared-error loss: writing the objective as $n^{-1}\mathrm{RSS}+{\lambda }_{\mathrm{avg}}\Vert \beta {\Vert }_2^2$ gives the same fit when $\lambda =n{\lambda }_{\mathrm{avg}}$.^[ref]

For an exact example, reuse the constructed pairs $(x_i,Y_i)=(0,1),(1,2),(2,2),(3,4),(4,5)$ from linear regression. Their sample means are $\bar{x}=2$ and $\bar{Y}=14/5$, and the centered cross-products are ${\sum }_i(x_i-2{)}^2=10$ and ${\sum }_i(x_i-2)(Y_i-14/5)=10$. ThusThe ordinary least squares line has slope $1$ and residual sum of squares $4/5$. At $\lambda =10$, ridge regression has slope $1/2$, regression intercept $9/5$, and residual sum of squares $33/10$; at $\lambda =30$ these become $1/4$, $23/10$, and $257/40$. Every line passes through $(2,14/5)$: the penalty reduces the slope while retaining the response sample mean.

The computational benefit is visible after predictor centering. Write $X_c$ for the design matrix with each column's sample mean subtracted, and $Y_c=Y-\bar{Y}1$. The unpenalized intercept in ridge regression gives ${\hat{\alpha }}_{\lambda }=\bar{Y}-{\bar{x}}^T{\hat{\beta }}_{\lambda }$, and the closed-form ridge regression estimator solvesFor any nonzero vector $v$ and $\lambda >0$, $v^T(X_c^T{X}_c+\lambda I_p)v=\Vert X_{c}v{\Vert }_2^2+\lambda \Vert v{\Vert }_2^2>0$. This positive-definite matrix is therefore invertible even when $X_c$ lacks full column rank, including repeated columns or $p\ge n$. The unique penalized slopes resolve the ambiguity of rank-deficient ordinary least squares; they do not supply evidence about slope directions absent from the data. This makes ridge regression useful for multicollinearity.^[ref]

Predictor centering and scaling serve different purposes. Centering handles the free regression intercept. Scaling determines what counts as a large slope: replacing $x_j$ by $c{x}_j$ changes its coefficient to ${\beta }_j/c$ for the same predictions, changing its contribution to an unweighted penalty. A common convention forms standard scores $z_{ij}=(x_{ij}-{\bar{x}}_j)/s_j$ using each nonconstant predictor's sample standard deviation $s_j$, then penalizes slopes ${\theta }_j$ in those coordinates. Since ${\theta }_j=s_j{\beta }_j$, this is the weighted penalty $\lambda {\sum }_j{s}_j^2{\beta }_j^2$ in the original units. Constant columns have $s_j=0$ and must be removed before this scaling. Scaling is a modelling choice; equal penalties in original units and in standardized units define different fits.^[ref]

The singular value decomposition explains the shrinkage. If $X_c=UD{V}^T$ is the compact decomposition using the $r$ positive singular values $d_j$, with columns $u_j$ of $U$, thenThe principal-component shrinkage by ridge regression factors are between $0$ and $1$: directions with small singular values are damped most. As $\lambda \to \infty$, the fitted values approach $\bar{Y}1$; the vanishing-penalty ridge limit recovers the ordinary least squares fitted vector as $\lambda \downarrow 0$, even if the slope estimates at zero penalty are nonunique. The covariance and bias of a ridge regression estimator describes the resulting bias-variance tradeoff. A penalty can reduce variance of an estimator while introducing bias of an estimator; the unbounded directional risk of fixed ridge shrinkage rules out a uniform improvement for a fixed positive penalty over all possible signals.^[ref]

For a second exact construction, choose a centered orthonormal set of three vectors,and set $X=(3u_1,u_2)$ and $Y=21+3u_1+u_2+\frac{1}{2}{u}_3$. The squared singular values are $9$ and $1$. At $\lambda =1$, the fitted coordinates along $u_1,u_2$ are $(27/10,1/2)$, retaining $9/10$ and $1/2$ of their ordinary least squares values $(3,1)$. At $\lambda =9$ they become $(3/2,1/10)$, retaining $1/2$ and $1/10$. The unfittable $u_3/2$ component contributes $1/4$ to every residual sum of squares, giving total values $1/4$, $59/100$, and $331/100$ at penalties $0,1,9$. These are constructed fits, not measured prediction results.

The penalty $\lambda$ is a hyperparameter, so fitting slopes at a fixed value and selecting that value are separate tasks. Use a validation set or K-fold cross-validation to compare candidate penalties by held-out squared-error loss. In each split, compute predictor sample means and scales from that split's training data, apply the same transformation to its validation set, and refit the slopes; using held-out information in preprocessing introduces data leakage. Minimum training residual sum of squares cannot select useful shrinkage: ordinary least squares already attains the smallest possible value.^[ref] After model selection, refit on all training data with the selected penalty. A separate test set or nested cross-validation assesses the entire selection procedure; a low selected validation score alone is no performance guarantee.^[ref]