Regression residual

A regression residual is the observed response minus its fitted value. Residual plots diagnose failures of assumptions such as constant variance, linear mean structure, and independent errors.

If observed responses satisfy $L\le y_i\le U$, their regression residuals $e_i=y_i-{\hat{y}}_i$ necessarily obey the displayed inequalities. Subtracting the corresponding fitted values proves both bounds, without a normal distribution or a fitted-model assumption. On a residual-versus-fitted plot, all points must lie between two parallel lines of slope $-1$. This is also a consistency check when $L,U$ are merely the reported observed minimum and maximum, rather than theoretical response limits. A bounded response does not itself prove that a straight conditional mean is inappropriate.

For a finite sample, let $L={\min }_i{y}_i$, $U={\max }_i{y}_i$, $m={\min }_i{\hat{y}}_i$ and $M={\max }_i{\hat{y}}_i$. The regression residuals are $e_i=y_i-{\hat{y}}_i$. Every residual is at least $L-M$ and at most $U-m$. At an observation attaining $y_i=L$, its residual is at most $L-m$, so $e_{\min }\le L-m$. At an observation attaining $y_i=U$, its residual is at least $U-M$, so $e_{\max }\ge U-M$. These observations prove all four bounds. Merely knowing theoretical response limits, without attainment, would not justify the two inner bounds.

A residual plot displays regression residuals against fitted responses, predictor values or observation order. Curvature can suggest an incorrect regression function; changing spread can suggest unequal error variances; ordered runs can suggest dependence. A plot is a diagnostic, not a proof of independence. Dependence confined to higher moments, as for adjacent products of independent Gaussian variables, may call for lagged squared regression residuals rather than a basic scatter plot. Fitting also induces dependence among regression residuals even under independent original errors.

A systematic curved pattern in regression residuals plotted against fitted values suggests a misspecified conditional mean. Positive residuals at both ends and negative residuals in the middle motivate trying quadratic regression. The diagnostic does not by itself identify the error probability distribution or prove heteroscedasticity.

In an additive mean model, the partial residual for component $j$ subtracts every fitted term except that component. It equals the full regression residual plus the fitted component. Plotting it against that component's predictor shows the conditional pattern after accounting for other fitted effects, and supplies the response for a backfitting update.

In a normal linear model, the $i$th regression residual has variance ${\sigma }^2(1-h_{ii})$, where $h_{ii}$ is a diagonal entry of the hat matrix. Replacing $\sigma$ by the residual standard deviation $s$ gives the standardized residual used in the normal Q-Q plot of R's linear-model diagnostic plots.

In a normal linear model with residual degrees of freedom $\nu \ge 2$, an internally studentized residual satisfies $|{\eta }_i|\le \sqrt{\nu }$ and ${\eta }_i^2/\nu \sim \mathrm{Beta}(1/2,(\nu -1)/2)$, provided its leverage is less than one. Choose an orthonormal basis of the residual space with its first direction proportional to the projection of the $i$th coordinate vector. The Gaussian coordinates in this basis are independent standard normal random variables $Z_1,\dots ,Z_{\nu }$, and ${\eta }_i=\sqrt{\nu }{Z}_1/({\sum }_j{Z}_j^2{)}^{1/2}$. The beta distribution follows from the independent chi-squared distributions of $Z_1^2$ and ${\sum }_{j=2}^{\nu }{Z}_j^2$. The bounded residual is not exactly distributed according to Student's t-distribution; it approaches the standard normal distribution as $\nu \to \infty$.