Elastic net regularization

Elastic-net regression minimizes squared error plus both an ${\ell }^1$ coefficient penalty and a squared ${\ell }^2$ penalty. It combines exact sparsity with strict convexity when the quadratic penalty is positive.

Extend $J$ by infinity outside ${\ell }^1$. For a bounded linear operator $K:{\ell }^2\to {\ell }^2$, the objective $\Vert Ku-f{\Vert }_2^2/2+\alpha J(u)$ is coercive, weakly lower semicontinuous and strictly convex for $\alpha >0$, so it has a unique minimizer. The quadratic penalty ensures existence and uniqueness even if $K$ has a null space. Its limiting exact solution minimizes $J$ in the data fiber, which need not be the minimum-norm least-squares solution.

For fixed $\alpha >0$ and nonzero bounded $K$, the reconstruction vanishes when $\Vert K^{\ast }f{\Vert }_{\infty }\le \alpha$, which includes a neighborhood of zero. If it were linear it would vanish everywhere. Choose a coordinate $e_j$ with $K{e}_j\ne 0$ and data $f=tK{e}_j$ with $t\Vert K{e}_j{\Vert }^2>\alpha$; the optimality inequality at zero fails, so its minimizer is nonzero. Thus the map is nonlinear. The qualification $K\ne 0$ is essential: $K=0$ gives the zero linear map.

Assume an exact solution of finite penalty exists and select the unique $J$-minimizing exact solution $u^{†}$. Minimality gives $\Vert K{u}_{\alpha }^{\delta }-f^{\delta }{\Vert }^2/2+\alpha J(u_{\alpha }^{\delta })\le {\delta }^2/2+\alpha J(u^{†})$. If $\alpha \to 0$ and ${\delta }^2/\alpha \to 0$, boundedness, weak sequential compactness of bounded sequences in a reflexive Banach space and lower semicontinuity identify every weak limit with $u^{†}$. The separate lower semicontinuity of $\Vert u{\Vert }_1$ and $\Vert u{\Vert }_2^2$, together with the upper bound on $J$, forces the Hilbert norms to converge. The Radon-Riesz theorem then gives norm convergence.

The quadratic penalty encourages strongly correlated predictors to receive similar coefficients, so an elastic net tends to retain or discard correlated predictors as a group more readily than the Lasso.