= Fitted values
{title2=$\widehat Y$}
<Fitted values> are the predictions at the predictor values used to fit a <regression model>. In <linear regression> with an explicit <regression intercept>,
$$
\widehat Y=\widehat\alpha\mathbf1+X\widehat\beta,\qquad r=Y-\widehat Y.
$$
Here $X$ is the slope <design matrix>, and $r$ is the <regression residual> vector. The <fitted values> estimate the <regression function> at the observed predictors; they need not equal the observed responses, which may also contain random errors. Prediction at a new predictor <vector> $x_*$ uses $\widehat\alpha+x_*^T\widehat\beta$ with the same centering and scaling used in fitting. A <prediction interval> addresses uncertainty in a new response rather than just displaying this point prediction.
For the five constructed observations in <linear regression>, <ordinary least squares> gives <fitted values> $(4/5,9/5,14/5,19/5,24/5)$. The <ridge regression> fit at $\lambda=10$ instead gives $(9/5,23/10,14/5,33/10,19/5)$. Both fitted vectors have <sample mean> $14/5$ because the <regression intercept> is unpenalized; their <regression residuals> differ and their <residual sums of squares> are $4/5$ and $33/10$, respectively.
For <ordinary least squares>, the <fitted values> are the <orthogonal projection> of $Y$ onto the <column space> of the complete <design matrix>, including its <regression intercept> column. They remain unique even in <rank-deficient ordinary least squares>, although the <regression coefficients> can be nonunique. The <regression residuals> are then <orthogonal> to every column.<Data Mining: 36-462 and 36-662/Regression 1: Different perspectives>[slides 6–10: fitted values, hat matrix, projection and orthogonal residuals]{ref} With centered slopes, <ridge regression> instead satisfies $X_c^Tr=\lambda\widehat\beta_\lambda$, so the <regression residuals> need not be <orthogonal> to the slope columns. Their sum is still zero because the <regression intercept> remains free.
For fixed predictors and scales and a fixed positive penalty, <ridge regression> gives a <linear smoother>,
$$
\widehat Y_\lambda=H_\lambda Y,\qquad H_\lambda=\frac{\mathbf1\mathbf1^T}{n}+X_c(X_c^TX_c+\lambda I_p)^{-1}X_c^T.
$$
For $\lambda>0$ with at least one nonzero slope direction, this <matrix> is not an <orthogonal projection matrix>: applying it twice shrinks that direction again. Its <effective degrees of freedom> are $1+\sum_{j=1}^r d_j^2/(d_j^2+\lambda)$, where $d_j$ are the positive <singular values> of $X_c$; the extra $1$ counts the free <regression intercept>. If the penalty is selected using $Y$, the resulting fitting rule need not be a <linear smoother>, since its chosen $H_\lambda$ also depends on $Y$.
Back to article page
