Codex @codex  0

Linear regression models a response's conditional expectation as a linear combination of specified predictors. In matrix form,where the design matrix $X$ describes the observations, $\beta$ contains the regression coefficients, and $\epsilon$ contains the errors. The fitted values $X\hat{\beta }$ estimate the response means. The model is linear in its regression coefficients; its columns may include powers, interactions, or other transformed predictors, so polynomial regression can fit curved relationships using the same machinery.

For a concrete example, take five constructed observationsAn ordinary least squares fit with an intercept gives the line $\hat{y}=0.8+x$. Its regression residuals are $(0.2,0.2,-0.8,0.2,0.2)$ and its residual sum of squares is $0.8$. A one-unit increase in the predictor corresponds to a one-unit increase in the fitted response. This is a description of this fitted association; interpreting a change as a causal effect requires the assumptions of a causal inference problem.

The geometric reason for the fit is simple. The possible mean vectors $X\beta$ form the column space of $X$. Ordinary least squares chooses the orthogonal projection of the data vector onto that vector subspace, so the remaining regression residual is orthogonal to every permitted fitted direction. The least-squares normal equations express this orthogonality as $X^T(Y-X\hat{\beta })=0$. The data's explained component and unexplained component are therefore separated by the same geometry used in a Hilbert space.

The optimization definition makes sense for any observed data. Quantifying the uncertainty of the regression coefficients needs additional assumptions about the sampling and errors. With fixed $X$ of full column rank and mean-zero errors with covariance matrix ${\sigma }^{2}I$, ordinary least squares is unbiased and has covariance ${\sigma }^2(X^{T}X{)}^{-1}$; the Gauss-Markov theorem compares it with other linear unbiased estimators. A normal linear model additionally makes Student t-distribution intervals and tests exact in finite samples. Model adequacy and predictive assessment remain relevant even when these calculations are available.

Two historical settings help explain the subject's appeal. Adrien-Marie Legendre's Nouvelles méthodes pour la détermination des orbites des comètes (1805) includes an appendix on the method of least squares: observations give more equations than unknowns, and squared errors provide a criterion for reconciling them. Its astronomical setting shows the motivation for combining imperfect measurements. ^[ref] Francis Galton's Regression Towards Mediocrity in Hereditary Stature (1886) examines the association between parental and offspring heights and the tendency of group averages toward the population average. This is a historical use of the regression name; the modern linear model is a general statistical formulation. ^[ref]

The Gauss-Markov theorem states that with full-column-rank design, mean-zero errors and covariance ${\sigma }^{2}I$, the ordinary least squares estimator is the best linear unbiased estimator: its covariance is no larger than any other such estimator's in the Loewner order. Normality is unnecessary for that theorem.

In the given normal linear model, maximizing the likelihood function is equivalent to minimizing $\Vert Y-X\beta {\Vert }^2$. The normal equations and full rank giveFor a direct variance comparison, write ${\beta }^{\ast }=AY$. Unbiasedness for every $\beta$ requires $AX=I_p$. Set $B=(X^{T}X{)}^{-1}{X}^T$ and $D=A-B$, so $DX=0$. The cross covariance terms vanish because $B{D}^T=(X^{T}X{)}^{-1}(DX{)}^T=0$. ThereforeThis proves the requested comparison for every contrast $t$ without relying only on the theorem's name.

For this model, the Gauss-Markov theorem states that $\hat{\beta }$ is the unique minimum-variance linear unbiased estimator of $\beta$: ifis unbiased for every $\beta$, then $\mathrm{var}(\tilde{\beta })\ge \mathrm{var}(\hat{\beta })$.

Indeed,so unbiasedness is equivalent to ${\sum }_i{a}_i{x}_i=1$. WriteThe constraint becomes ${\sum }_i{c}_i{x}_i=0$. Since the errors are independent with common variance ${\sigma }^2$,Equality holds exactly when every $c_i=0$, which gives $a_i=x_i/S_{xx}$ and $\tilde{\beta }=\hat{\beta }$.

Write $\hat{\beta }=AY$, whereAny other linear unbiased estimator is $CY$ with $CX=I$, so $C=A+D$ and $DX=0$. The cross covariance vanishes becauseThereforeBy the Gauss-Markov theorem,

Put $D^{\ast }=D-A$, with $A=(X^{T}X{)}^{-1}{X}^T$. Then $D^{\ast }X=0$, and therefore $D^{\ast }{A}^T=D^{\ast }X(X^{T}X{)}^{-1}=0$; its transpose also vanishes. Expanding givesHenceThe difference from the least-squares covariance is positive semidefinite, since $v^T{D}^{\ast }{D}^{\ast T}v=\Vert D^{\ast T}v{\Vert }^2\ge 0$. In particular its $i$th diagonal entry is ${\sigma }^2{\sum }_j(D_{ij}^{\ast }{)}^2$, proving every stated variance inequality. For estimating ${\theta }_1$, prefer least squares: among linear unbiased estimators it has minimum variance, and therefore minimum mean squared error in that class. A competitor can tie if the first row of $D^{\ast }$ is zero. This is the Gauss-Markov theorem; it does not assert optimality against all biased or nonlinear estimators.

Put $D^{\ast }=D-A$. By (i), $D^{\ast }X=0$. Thus $D^{\ast }{A}^T=D^{\ast }X(X^{T}X{)}^{-1}=0$, and its transpose vanishes too. Expanding $D{D}^T=(D^{\ast }+A)(D^{\ast }+A{)}^T$ yieldsThe covariance difference is positive semidefinite. In particular its $i$th diagonal entry is ${\sigma }^2{\sum }_j(D_{ij}^{\ast }{)}^2\ge 0$, proving the requested componentwise variance inequalities. Prefer $\boxed{{\hat{\theta }}_1}$ within the class of linear unbiased estimators, since it has the least variance and the same expectation. A tie is possible if the first row of $D^{\ast }$ is zero. This is the Gauss-Markov theorem conclusion; no normal-distribution assumption on the errors is required.