Past exam of the mathematics course of the University of Cambridge / 2008 / iii / Paper 45 / 2 / b / Solution

Use the Gaussian sampling model $Y\mid \beta \sim N(X\beta ,{\sigma }^2{I}_n)$ with known ${\sigma }^2>0$, and the independent prior $\beta \sim N(0,{\tau }^2{I}_p)$ with ${\tau }^2>0$. A Gaussian prior alone is not enough to imply the ridge result without this likelihood assumption. The posterior distribution is proportional toMaximizing it is equivalent to minimizing the ridge objective with $\lambda ={\sigma }^2/{\tau }^2$. More explicitly, complete the square. The posterior precision, covariance matrix and mean areConsequently the Gaussian posterior representation of ridge regression isA tighter prior, smaller ${\tau }^2$, implies stronger shrinkage; larger observation-noise variance at fixed prior scale also increases the penalty. The posterior covariance is $V$, not the frequentist sampling covariance of the ridge estimator, which is ${\sigma }^2(X^{T}X+\lambda I{)}^{-1}{X}^{T}X(X^{T}X+\lambda I{)}^{-1}$. If an intercept is to remain unpenalized, give only the slopes this proper Gaussian prior and center the design and responses.