Past exam of the mathematics course of the University of Cambridge / 2021 / iii / Paper 205 / 1

WriteIts score isThe Karush-Kuhn-Tucker conditions for L1-penalized logistic regression are therefore

The scalar logistic loss $u\mapsto -u+\log (1+e^u)$ is strictly convex because its second derivative is $e^u/(1+e^u{)}^2>0$. If $\hat{\beta }$ and $\tilde{\beta }$ are minimizers but $X\hat{\beta }\ne X\tilde{\beta }$, strict convexity of the loss as a function of the fitted vector and convexity of the L1 norm make the objective at their midpoint strictly smaller than the common minimum. This contradiction proves thatfor every pair of solutions.

Part b shows that all solutions have the same fitted vector, hence the same score and the same set $E$. The KKT conditions show that every nonzero coordinate of any solution belongs to $E$, since a nonzero coefficient forces the corresponding score to have absolute value $\lambda$. Thus every solution is supported on $E$ and satisfiesIf $\mathrm{rank}(X_E)=|E|$, the linear map $u\mapsto X_{E}u$ is injective. Consequently ${\beta }_E$ and hence $\beta$ are unique.