Past exam of the mathematics course of the University of Cambridge / 2021 / iii / Paper 218 / 1

Put $s=x^{T}x>0$ and $z=x^{T}Y$. With objective $\Vert Y-x\beta {\Vert }_2^2+\lambda {\beta }^2$, ridge regression givesFor the duplicated design, the objective depends on ${\beta }_1+{\beta }_2$ through the loss and symmetry makes the minimum-penalty decomposition equal:Duplicating a predictor therefore halves its effective ridge penalty.

The one-variable constraint is ${\beta }^2\le t$, an interval whose endpoints expand as $\sqrt{t}$. The duplicated constraint is the disk ${\beta }_1^2+{\beta }_2^2\le t$; the loss contours are parallel strips perpendicular to $(1,1)$. Their first contact with the disk lies on ${\beta }_1={\beta }_2$. Equivalently, the fitted total coefficient is the least-squares coefficient clipped to $[-\sqrt{2t},\sqrt{2t}]$, compared with $[-\sqrt{t},\sqrt{t}]$ for one copy.

The two Lasso problems areandIf $\hat{\beta }$ solves the first problem, the complete solution set of the second isIndeed, the feasible totals are exactly $[-t,t]$, the same as in the first problem.

The duplicated feasible set is the diamond $|{\beta }_1|+|{\beta }_2|\le t$. A loss contour first touches an entire line segment of the diamond whenever the optimal total has the sign of a sloping face; all points on that segment give the same fit. As $t$ passes the unconstrained optimum, the solution set becomes the intersection of the interior diamond with the line ${\beta }_1+{\beta }_2=z/s$.

For prediction at covariate value $u$, the intercept contributes variance ${\sigma }^2/n$ in every case because $x$ is centered. The one-copy ridge shrinkage is $a_1=s/(s+\lambda )$, while the duplicated-design total has $a_2=2s/(2s+\lambda )$. ThusandSince $a_2>a_1$, duplication reduces ridge bias and increases variance.

For constrained Lasso, let $Z=z/s\sim N({\beta }_0,{\sigma }^2/s)$ and $C_t(Z)=\max (-t,\min (Z,t))$. Both designs have the identical fitted total $C_t(Z)$, so both haveDuplicating the predictor has no effect on Lasso predictions, despite making the coefficient vector nonunique.