Ordinary least squares

Ordinary least squares minimizes the unweighted sum of squared residuals between observed and fitted responses.

For a correctly specified full-rank fixed-design Gaussian linear regression with $p<n$, dividing the residual sum of squares by the residual degrees of freedom $n-p$ gives an unbiased estimator of the noise variance. The projection removes the whole deterministic mean, so the numerator's expectation is $(n-p){\sigma }^2$. A misspecified subset model need not have this property.

In a full-rank normal linear model with $n$ observations and $p$ mean coefficients, the residual standard error estimates the common error standard deviation. Its square is the unbiased estimator $\mathrm{RSS}/(n-p)$; the square root itself is not generally unbiased. It measures unexplained response variation, whereas a regression coefficient's standard error measures uncertainty in that coefficient.

Partial regression first projects a response and a selected predictor orthogonally off the remaining design columns. Regressing the residualized response on the residualized predictor yields its coefficient in the full ordinary least squares fit.

If a design matrix $X$ has deficient rank, every ordinary least squares minimizer has the form $X^{+}Y+z$ with $z\in \ker X$, where $X^{+}$ is the Moore-Penrose inverse. Hence the fitted values are unique but the coefficients are not: the minimizers form an affine subspace of dimension $p-\mathrm{rank}X$.

If a design matrix $X$ has a nonzero null vector $z$, the normal linear model has identical observation distributions at $\beta$ and $\beta +z$. Any estimator has the same expectation at those two parameters, and therefore cannot be an unbiased estimator of both coefficient vectors. Full column rank is required for unbiased estimation of the entire unconstrained coefficient vector; identifiable linear combinations can still be estimated without it.

In the model $Y_i=\beta x_i+{\epsilon }_i$ without an intercept, with ${\sum }_i{x}_i^2>0$, the ordinary least-squares estimator isFor independent errors of common variance ${\sigma }^2$, it is unbiased and has variance ${\sigma }^2/{\sum }_i{x}_i^2$.

For independent $Y_i\sim N(\beta x_i,{\sigma }^2)$ with $n\ge 2$ and $\sum x_i^2>0$, the least-squares estimator is $\hat{\beta }=\sum x_i{Y}_i/\sum x_i^2$. The projection of the isotropic Gaussian vector onto the design direction is independent of its orthogonal residual; hence $\hat{\beta }\sqrt{\sum x_i^2}/\sigma$ is standard normal under $\beta =0$ and $\mathrm{SSE}/{\sigma }^2$ is independent chi-squared with $n-1$ degrees of freedom. The displayed statistic therefore has Student's t-distribution with $n-1$ degrees of freedom. A two-sided test rejects at the corresponding upper/lower quantiles. The zero design has no information about the slope; with one observation there is no residual variance degree of freedom.

For a linear regression design matrix $X$ with full column rank, the ordinary least squares coefficient vector is $\hat{\beta }=(X^{T}X{)}^{-1}{X}^{T}Y$. In a normal linear model with error covariance matrix ${\sigma }^{2}I$, it is an unbiased estimator with covariance matrix ${\sigma }^2(X^{T}X{)}^{-1}$. Replacing ${\sigma }^2$ by the residual variance estimate gives the reported coefficient standard errors.

In simple linear regression with a nonconstant predictor,

For a nonconstant predictor, putThe minimized residual sum of squares isEach centered sum can be recovered in constant time from the five raw sums $\sum x_i$, $\sum y_i$, $\sum x_i^2$, $\sum x_i{y}_i$, and $\sum y_i^2$.