= Residual sum of squares
{title2=$\operatorname{RSS}$}
{wiki}
= Residual sums of squares
{synonym}
= RSS
{c}
{synonym}
The <residual sum of squares> measures how far the observed responses lie from their <fitted values>. If $r_i=Y_i-\widehat Y_i$ is the $i$th <regression residual>, then
$$
\operatorname{RSS}=\sum_{i=1}^n(Y_i-\widehat Y_i)^2=\|Y-\widehat Y\|_2^2.
$$
It is the <squared Euclidean norm> of the <regression residual> vector. Squaring prevents positive and negative deviations from cancelling and gives large deviations more weight. In the constructed five-point example in <linear regression>, the <regression residuals> are $(1/5,1/5,-4/5,1/5,1/5)$, so the <residual sum of squares> is $4/5$.
<Ordinary least squares> minimizes the <residual sum of squares> over the chosen <linear regression> family; <ridge regression> minimizes the <residual sum of squares> plus a penalty. Consequently the <ridge regression> fit can have a larger <residual sum of squares> than the <ordinary least squares> fit on exactly the same <design matrix>. For the one-predictor construction in <ridge regression>, the <residual sum of squares> rises from $4/5$ to $33/10$ when the penalty changes from $0$ to $10$; that increase is the cost of shrinking the fitted slope.
Dividing the <residual sum of squares> by $n$ gives the <training error> for <squared-error loss>, not an estimate based on fresh responses. Its units are squared response units, and it adds across observations, so raw totals need not be comparable across different response scales or sample sizes. Comparing fits to the same responses is useful, but choosing a <ridge regression> penalty by minimum training <residual sum of squares> favors the unpenalized fit. Assess <prediction error> using a <validation set> or <cross-validation> instead.<Data Mining: 36-462 and 36-662/Model selection and validation 1: Cross-validation>[slides 6–8 and 11–14: fresh-response error, training error and held-out scoring]{ref}
Back to article page
