= Solution
A real <positive-definite kernel> is a symmetric <function> $k:\mathcal X\times\mathcal X\to\mathbb R$ such that, for every finite collection $x_1,\ldots,x_n$ and every $a\in\mathbb R^n$,
$$
\sum_{i,l=1}^na_i a_l k(x_i,x_l)\ge0.
$$
Thus every <kernel matrix> is a <positive semidefinite matrix>; “positive definite” here does not require strict positivity. A <Reproducing kernel Hilbert space> is a <Hilbert space> $\mathcal H$ of <functions> on $\mathcal X$ in which every <point evaluation> is a <continuous linear functional>. Its <reproducing property> is $f(x)=\langle f,k_x\rangle_{\mathcal H}$ with $k_x=k(\cdot,x)\in\mathcal H$, and $k(x,y)=\langle k_y,k_x\rangle_{\mathcal H}$. For a sum of <positive-definite kernels>, the <quadratic form> of its <kernel matrix> is a sum of nonnegative <quadratic forms>. Symmetry also survives summation. Consequently $\boxed{k=\sum_jk_j\text{ is a positive-definite kernel}.}$
For the <representer theorem>, put $V=\operatorname{span}\{k_{x_1},\ldots,k_{x_n}\}$. This is a closed <finite-dimensional vector space>. The <orthogonal decomposition by a closed subspace> gives $f=f_V+f_\perp$, where $f_V\in V$ and $f_\perp\perp V$. The <reproducing property> implies $f_\perp(x_i)=0$ for every observed input, and therefore
$$
Q_1(f)=Q_1(f_V)+\lambda\|f_\perp\|_{\mathcal H}^2.
$$
Since $\lambda>0$, any <minimizer> must lie in $V$. For $f_\alpha=\sum_i\alpha_i k_{x_i}$, the <reproducing property> gives the evaluation <vector> $K\alpha$ and $\|f_\alpha\|_{\mathcal H}^2=\alpha^TK\alpha$. Hence $Q_1(f_\alpha)=M(\alpha)$. If $\widehat f$ minimizes $Q_1$, its representation yields a minimizing $\widehat\alpha$. Conversely, if $\widehat\alpha$ minimizes $M$, then, for any $f\in\mathcal H$, representing $f_V=f_\alpha$ gives $Q_1(f)\ge M(\alpha)\ge M(\widehat\alpha)$. Thus
$$
\boxed{\widehat f=\sum_i\widehat\alpha_i k(\cdot,x_i),\qquad \widehat\alpha\in\operatorname*{argmin}_{\alpha\in\mathbb R^n}M(\alpha).}
$$
This is an equivalence of <minimizers>, not an existence or uniqueness assertion: the <loss function> is arbitrary. A singular <kernel matrix> also allows nonunique coefficients, since differences in $\ker K$ represent the zero <function>. Strict <convexity> of the loss was not used.
Now use the stated minimum-norm description of a <sum of reproducing-kernel Hilbert spaces>. Let $\widehat f=\sum_j\widehat f_j$. If the minimizing tuple did not attain the minimum total squared <norm> among decompositions of $\widehat f$, replacing it by that unique minimum-norm tuple would preserve the <loss function> and strictly lower the penalty. This contradicts optimality. Therefore
$$
Q_2(\widehat f_1,\ldots,\widehat f_p)=Q_1(\widehat f).
$$
For an arbitrary $f\in\mathcal H$, take its minimum-norm decomposition $(f_1,\ldots,f_p)$. Then $Q_1(\widehat f)=Q_2(\widehat f_1,\ldots,\widehat f_p)\le Q_2(f_1,\ldots,f_p)=Q_1(f)$, proving that $\widehat f$ minimizes $Q_1$.
Choose $\widehat\alpha$ from the <representer theorem> for this minimizing sum, and define $g_j=\sum_i\widehat\alpha_i k_j(\cdot,x_i)$. Write $K_j$ for the component <kernel matrices>. Then
$$
\sum_jg_j=\widehat f,\qquad
\sum_j\|g_j\|_{\mathcal H_j}^2=\sum_j\widehat\alpha^TK_j\widehat\alpha=\widehat\alpha^TK\widehat\alpha=\|\widehat f\|_{\mathcal H}^2.
$$
By uniqueness of the minimum-norm decomposition, $g_j=\widehat f_j$ for every $j$. This proves the <common representer coefficients for a sum of kernels> conclusion:
$$
\boxed{\widehat f_j(\cdot)=\sum_{i=1}^n\widehat\alpha_i k_j(\cdot,x_i)\quad\text{for every }j,\quad\widehat\alpha\in\operatorname*{argmin}M.}
$$
The same coefficient <vector> works simultaneously. Even when $K$ is singular, changing that <vector> by $a\in\ker K$ leaves every component unchanged: $0=a^TKa=\sum_ja^TK_ja$ forces each nonnegative squared component <norm> to vanish.
Back to article page
