Basic inequality for a penalized least-squares estimator ($\Vert h{\Vert }_2^2/n\le 2⟨z,h⟩/n+\lambda J({\theta }^{\ast })-\lambda J(\hat{\theta })$)

For the identity design and $y={\theta }^{\ast }+z$, comparing the minimizing objective at $\hat{\theta }$ with its value at ${\theta }^{\ast }$ and expanding the squared Euclidean norm proves the displayed inequality, with $h=\hat{\theta }-{\theta }^{\ast }$. This deterministic inequality does not assume any probability distribution for the noise. A bound on the noise inner product turns it into a statistical error estimate.