= Solution
Expanding the <ordinary least squares> objective gives
$$
Q(\beta)=Y^TY-2\beta^TX^TY+\beta^TX^TX\beta,
\qquad \nabla Q=2X^T(X\beta-Y),\quad \nabla^2Q=2X^TX.
$$
For a nonzero vector $v$, $v^TX^TXv=\|Xv\|^2>0$, since the <design matrix> has <matrix rank> $p$. The <Hessian matrix> is therefore <positive-definite>, and $Q$ is strictly <convex>. Solving its <normal equations> gives the unique <least-squares estimator>:
$$
\boxed{\widehat\beta=(X^TX)^{-1}X^TY.}
$$
Indeed, expansion about this stationary point and $X^T(Y-X\widehat\beta)=0$ give $Q(\beta)=Q(\widehat\beta)+(\beta-\widehat\beta)^TX^TX(\beta-\widehat\beta)$, which also proves the global minimum directly.
Define the <hat matrix> $H=X(X^TX)^{-1}X^T$. It is a <symmetric matrix> and an <idempotent linear map>, since $H^T=H$ and $H^2=H$, and its image is the column space of $X$. Thus $H$ is the <orthogonal projection> onto the fitted mean space. The <fitted values> are $HY$ and the <regression residual> vector is $(I-H)Y$. Because $I-H$ is also a <symmetric matrix> and an <idempotent linear map>,
$$
\boxed{Q(\widehat\beta)=Y^T(I-H)^T(I-H)Y=Y^T(I-H)Y.}
$$
These minimization and projection identities do not need errors with a <normal distribution>; the <normal linear model> becomes relevant to the distributional calculations.
Back to article page
