Past exam of the mathematics course of the University of Oxford / 2023 / Prelims / Mathematics III / 9 / a

Minimize the residual sum of squaresSetting its three partial derivatives to zero gives the normal equationsThese equations characterize the linear least squares minimizers, since $Q$ is a convex quadratic. A unique coefficient vector additionally requires full column rank of the design matrix.

The centered regressors make the first normal equation read $n{\hat{\beta }}_0={\sum }_i{Y}_i$. DefineThe remaining equations are $A{\hat{\beta }}_1+B{\hat{\beta }}_2=u$ and $B{\hat{\beta }}_1+C{\hat{\beta }}_2=v$. Provided $D=AC-B^2>0$, matrix inversion givesBy the Cauchy-Schwarz inequality, $D\ge 0$. If $D=0$, the two regressor columns are linearly dependent, and their separate slopes are not uniquely identifiable. One still has well-defined linear least squares fitted values, but cannot use the displayed divisions by $D$.

If the two regressors are strongly associated, the data may provide little support for varying one while holding the other fixed. Multicollinearity makes the separate slope estimates unstable, and exact linear dependence makes them unidentifiable. A conditional slope may also be an extrapolation beyond combinations actually observed.

Furthermore, a slope in a linear regression model is not automatically a causal effect. A causal interpretation needs appropriate design or assumptions excluding confounding; independence of the errors across observations and zero mean alone do not establish those assumptions. The coefficients describe the specified linear mean relation, but changing a regressor by intervention can have a different interpretation.