= Solution
With the convention that the residual term is a sum rather than an average, the <penalised least squares> criterion is
$$
\boxed{J_\lambda(g)=\sum_{i=1}^n\{y_i-g(x_i)\}^2+\lambda\int_{x_1}^{x_n}\{g''(x)\}^2\,dx,\qquad\lambda>0.}
$$
Minimize over functions whose first <derivative> is absolutely continuous and whose second <derivative> is square-integrable, that is, the usual <Sobolev space> $H^2[x_1,x_n]$. The <cubic smoothing spline> $\widehat g_\lambda$ is its minimizer, with natural <boundary conditions> and linear continuation outside the data range. It is a <natural cubic spline> with possible knots at the observed $x_i$, but usually does not interpolate the noisy $y_i$.
To see why the minimizer is a natural <spline>, for any $g$ take the <natural cubic spline interpolant> $s$ through its values $g(x_i)$ and write $h=g-s$. Then $h(x_i)=0$. Integrating $\int s''h''$ by parts on every knot interval twice gives zero: the terms involving $h$ vanish at knots, the terms involving $h'$ cancel because $s''$ is continuous there, and the outer terms vanish by the natural <boundary conditions>. Hence
$$
\int(g'')^2=\int(s'')^2+\int(h'')^2\ge\int(s'')^2,
$$
while their <residual sums of squares> agree. Replacing $g$ by $s$ never increases the criterion. This proves the <minimum roughness property of the natural cubic spline interpolant> needed here.
For fitted values $u_i=s(x_i)$, write $\int(s'')^2=u^TKu$, where $K$ is positive semidefinite and has the sampled affine functions as its two-dimensional nullspace. The finite-dimensional minimization becomes $\|y-u\|^2+\lambda u^TKu$. Its strictly positive definite Hessian is $2(I+\lambda K)$, so it has the unique solution
$$
\boxed{\widehat u=A_\lambda y,\qquad A_\lambda=(I+\lambda K)^{-1}.}
$$
This is the <roughness-matrix formula for a natural cubic smoothing spline>. Larger $\lambda$ penalizes curvature more strongly. As $\lambda\downarrow0$ the fit approaches interpolation; as $\lambda\to\infty$ it approaches the ordinary least-squares affine line.
Back to article page
