= Linear least-squares projection
A <linear least-squares projection> is the <linear least-squares projection>[affine predictor] that minimizes <mean squared error>. For real <random variable>[random variables] $X,Y$ with <second moment>[finite second moments] and $\operatorname{Var}(X)>0$, minimizing $\mathbb E(Y-a-bX)^2$ gives
$$
b=\frac{\operatorname{Cov}(X,Y)}{\operatorname{Var}(X)},\qquad a=\mathbb EY-b\mathbb EX.
$$
The <regression residual>[residual] $R=Y-a-bX$ satisfies $\mathbb ER=0$ and $\mathbb E(RX)=0$. Consequently it is orthogonal, in the inner product $\langle U,V\rangle=\mathbb E(UV)$, to the span of the constant variable and $X$. For any other <linear least-squares projection>[affine predictor] $a'+b'X$, expanding the square yields
$$
\mathbb E(Y-a'-b'X)^2=\mathbb ER^2+\mathbb E[(a-a')+(b-b')X]^2,
$$
because the cross term vanishes. This proves optimality and uniqueness of the predicted <random variable>. The <regression residual>[residual] <variance> is the <Schur complement covariance>. No <normal distribution> is needed. For several predictors, the same orthogonality argument gives the normal equations for <Linear least squares>; a singular <independent variable>[predictor] <covariance> can make coefficients nonunique while leaving the projected prediction unique.
Back to article page