Representer theorem

Let $\mathcal{H}$ be a Reproducing kernel Hilbert space with positive-semidefinite kernel $k$. Suppose an objective has the formwhere $\Omega$ is strictly increasing. If a minimizer exists, every minimizer lies in the linear span of the sections $k(x_i,\cdot )$:To prove this, decompose $f=f_{\parallel }+f_{\perp }$ into its orthogonal projection onto that finite-dimensional linear span and its orthogonal remainder. The reproducing property gives $f_{\perp }(x_i)=0$, so removing the remainder preserves the data-dependent term. The Pythagorean theorem gives $\Vert f{\Vert }_{\mathcal{H}}^2=\Vert f_{\parallel }{\Vert }_{\mathcal{H}}^2+\Vert f_{\perp }{\Vert }_{\mathcal{H}}^2$, so strict increase of $\Omega$ excludes a nonzero remainder at a minimizer.

For merely nondecreasing $\Omega$, projecting an existing minimizer still gives a minimizer in the linear span, but other minimizers may remain outside it. The theorem is a finite-representation result; existence of a minimizer needs separate hypotheses.