 Regularization

Regularization constrains or penalizes fitting to control the complexity or instability of a learned rule. A penalty formulation iswhere $L_n$ is empirical loss function and $J$ measures a chosen notion of complexity. Ridge regression penalizes squared coefficient size, the Lasso penalizes the L1 norm, and a roughness penalty penalizes derivatives of a fitted function. Early stopping can regularize an iterative fitting procedure without an explicit added penalty.

For a scalar observation $Y=\theta +\epsilon$ with centered noise of variance ${\sigma }^2$, minimizing $(Y-a{)}^2+\lambda a^2$ gives $a=Y/(1+\lambda )$. Its mean squared error is $({\lambda }^2{\theta }^2+{\sigma }^2)/(1+\lambda {)}^2$: shrinkage reduces variance while introducing bias. This illustrates why regularization can improve prediction, but does not improve risk at every parameter value for every tuning choice. Cross-validation or a validation set can choose strength while a reserved test set assesses the fitted procedure.