Past exam of the mathematics course of the University of Cambridge / 2018 / iii / Paper 218 / 6

For the residual-versus-fitted plot, compute fitted values ${\hat{y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{x}_{i1}+{\hat{\beta }}_2{x}_{i2}$ and regression residuals $e_i=y_i-{\hat{y}}_i$; the plotted point is $({\hat{y}}_i,e_i)$.

For the normal Q-Q plot, let $H=X(X^{T}X{)}^{-1}{X}^T$ be the hat matrix for the full three-column design matrix and let $s^2=\mathrm{RSS}/(n-3)$. Form the standardized regression residualsand order them as $r_{(1)}\le \cdots \le r_{(n)}$. R plots $r_{(j)}$ against ${\Phi }^{-1}(p_j)$, where $p_j$ are its plotting positions. For $n>10$, $p_j=(j-1/2)/n$; $\Phi$ is the standard normal distribution function. This standardization is specified in R's diagnostic-plot documentation.

The original PDF shows a residual cloud around zero without compelling systematic curvature or a clear funnel, and a Q-Q pattern approximately following the reference line. Observations 81 and 83 are conspicuous positive tail points, with a smaller lower-tail departure. The plots show no decisive violation of linearity, constant variance or normality, but the extreme residuals merit inspection. They do not test independence across observations and cannot establish that all model assumptions hold.

The two predictors have correlation coefficient $\rho =0.9813846$, so there is severe multicollinearity. Their centered slope Gram matrix isThus either partial slope has variance ${\sigma }^2/[n(1-{\rho }^2)]$. Relative to a univariate regression with the same error variance, the two-predictor variance inflation factor and standard-error factor areThe small eigenvalue $n(1-\rho )$ makes the contrast between the two slope effects particularly poorly estimated. The univariate regressions test marginal association with the response; the bivariate tests ask whether one predictor adds an effect while holding its nearly identical companion fixed. Their much larger standard errors therefore make the partial tests insignificant without contradicting the marginal significance. The two fits also estimate slightly different residual variances, explaining why the reported standard-error ratio is not exactly 5.21.

The command performs a nested-model F-test for $H_0:{\beta }_1=0$ against $H_1:{\beta }_1\ne 0$, retaining the intercept and $x_2$ coefficient in both models. Write $RS{S}_2$ and $RS{S}_3$ for the residual sum of squares in the reduced and full fits. Under $H_0$ and the independent normal-error assumptions,By Cochran's theorem, the numerator and denominator are independent scaled chi-squares with one and $n-3$ degrees of freedom. Equivalently, for this single-coefficient restriction, $F=t_{{\beta }_1}^2$, so its p-value equals that of the two-sided coefficient Student's t-test in model3.

Using the printed $t$ value gives $F=0.800^2=0.640$ and $p\approx 0.43$. The printed estimate and standard error instead giveThis discrepancy is present in the PDF and is too large to be explained by rounding at the shown precision; the underlying ANOVA output is omitted, so an exact unique statistic cannot be recovered. The univariate correlation and $t$ value imply $n\approx 2+2.575^2(1-0.2517297^2)/0.2517297^2\approx 100.006$, consistent with $n=100$ after rounding. With denominator degrees of freedom 97, the two versions give p-values approximately $0.4257$ and $0.4298$, respectively. Either way the partial effect is not significant at 5%.

Let $\lambda$ denote the effective penalty in the Gaussian squared-error objective. With the loss normalization used in the glmnet documentation, alpha zero gives ridge regression:The intercept is fitted separately and unpenalized. The constant column supplied by model.matrix is not an additional penalized slope; glmnet excludes zero-variance predictor columns. Centering and the stated predictor norms make the two slope columns already standardized under the divisor-$n$ convention.

Let $X_s$ consist of those two columns, $z=y-\bar{y}1$, and $G=X_s^T{X}_s$. Minimizing over the intercept gives ${\hat{\beta }}_0=\bar{y}$, and the closed-form ridge regression estimator under this normalization isThus the right-hand limit in the plotted log-$\lambda$ coordinate is $\lambda \to \infty$, givingThe left-hand limit is $\lambda \to 0$, giving the unpenalized full-model slopesThe plotted curves are these two slopes. If the objective is instead written as unnormalized RSS plus a quadratic penalty, its penalty coefficient is $n\lambda$, not $\lambda$.

There is also a software response-scale convention to distinguish from this mathematical parameterization. The Gaussian fitting engine divides a supplied penalty by the response standard deviation $s_y=\sqrt{n^{-1}{\sum }_i(y_i-\bar{y}{)}^2}$ while fitting a standardized response, then rescales the coefficients. For pure ridge with these already standardized predictors, this corresponds to effective $\lambda ={\lambda }_{\mathrm{API}}/s_y$ in the displayed objective. Such a fixed positive rescaling of the horizontal parameter for the observed dataset does not change either asymptote. The fixed-penalty variance formulas below refer to $\lambda$ in the displayed mathematical objective.

Condition on the fixed design matrix and a prespecified $\lambda$. Put $M=G+n\lambda I_2$. The covariance and bias of a ridge regression estimator follow from its being a linear transformation of the response:Here centering the response introduces no correction in this expression because $X_s^{T}1=0$. For the equal-norm two-predictor design, either diagonal entry simplifies toHowever,The variances quantify sampling variability, but alone do not give valid confidence intervals for the unshrunk slopes. Centering a normal interval at the ridge estimate ignores its bias, which depends on the unknown slopes and need not be small relative to its standard error. Intervals require an appropriate bias correction, a bound on that bias, or another justified inferential procedure. A data-selected penalty introduces additional selection dependence not covered by the fixed-$\lambda$ formula. Similarly, holding a software penalty fixed while converting it using an estimated response scale does not hold the effective penalty fixed over repeated samples; the displayed covariance is not an unconditional variance formula for that nonlinear procedure.

One suitable criterion among the three unpenalized normal linear models is the Akaike information criterion. With $d$ estimated mean coefficients and the variance also estimated, minimizing AIC is equivalent to minimizingThe values of $d$ are $2,2,3$ for model1, model2 and model3. This compares fit and complexity rather than choosing solely by individual coefficient significance. The Bayesian information criterion or a common-fold predictive comparison would also be reasonable, depending on the selection objective.

For the ridge regression tuning parameter, use K-fold cross-validation with squared prediction loss:Use the same folds for every candidate, estimate any centering or scaling within each training fold, choose a minimizing $\lambda$, and refit on all observations. This estimates mean squared prediction error while avoiding data leakage. The printed summaries do not supply all residual sums of squares or validation losses, so they do not determine which model or penalty wins.