Past exam of the mathematics course of the University of Cambridge / 2022 / iii / Paper 218 / 6

The prior isThe software samples from the Bayesian posterior; the displayed "Estimate" values are posterior means computed from the Markov-chain Monte Carlo draws.

With unit error variance, the Gaussian likelihood has log-likelihoodFull column rank gives $\hat{\beta }=(X^{T}X{)}^{-1}{X}^{T}Y$ and the least-squares identityMultiplying the likelihood by ${\prod }_j\varphi ({\beta }_j)$ and absorbing all terms independent of $\beta$ into $C$ gives

For a Laplace prior $\varphi (u)\propto e^{-\lambda |u|}$, maximizing the posterior is equivalent to minimizingso the posterior mode is the Lasso. For a Gaussian prior $\varphi (u)\propto e^{-\lambda u^2/2}$, the mode minimizesand is the ridge regression estimator $(X^{T}X+\lambda I{)}^{-1}{X}^{T}Y$. Gaussian conjugacy makes the posterior normal, so its mode and posterior mean coincide at this ridge estimate.

The nearly identical predictor columns create severe multicollinearity. Their sum is well identified, as shown by the excellent fitted response, but their difference is weakly identified, allowing ordinary least squares to choose huge opposite coefficients with huge standard errors. The independent $N(0,1)$ priors impose the ridge penalty from part c, shrinking that unstable difference toward zero and producing the stable estimates near $(1,1)$.

The reported 95% credible interval for ${\beta }_1$ isConditional on the model, prior, and observed data, its posterior probability is 0.95. A frequentist 95% confidence interval instead has 95% coverage under repeated sampling before the data are observed; it does not assign a sampling probability to the fixed parameter after observing this dataset.