Past exam of the mathematics course of the University of Cambridge / 2005 / iii / Paper 41 / 3 / i / Solution

Expanding the ordinary least squares objective givesFor a nonzero vector $v$, $v^T{X}^{T}Xv=\Vert Xv{\Vert }^2>0$, since the design matrix has matrix rank $p$. The Hessian matrix is therefore positive-definite, and $Q$ is strictly convex. Solving its normal equations gives the unique least-squares estimator:Indeed, expansion about this stationary point and $X^T(Y-X\hat{\beta })=0$ give $Q(\beta )=Q(\hat{\beta })+(\beta -\hat{\beta }{)}^T{X}^{T}X(\beta -\hat{\beta })$, which also proves the global minimum directly.

Define the hat matrix $H=X(X^{T}X{)}^{-1}{X}^T$. It is a symmetric matrix and an idempotent linear map, since $H^T=H$ and $H^2=H$, and its image is the column space of $X$. Thus $H$ is the orthogonal projection onto the fitted mean space. The fitted values are $HY$ and the regression residual vector is $(I-H)Y$. Because $I-H$ is also a symmetric matrix and an idempotent linear map,These minimization and projection identities do not need errors with a normal distribution; the normal linear model becomes relevant to the distributional calculations.