= Solution
For $\lambda>0$, minimize the <smoothing spline> objective over the <Sobolev space> of functions with square-integrable second derivative on the observed interval:
$$
J(m)=\sum_i\{Y_i-m(X_i)\}^2+\lambda\int_{X_{\min}}^{X_{\max}}\{m''(x)\}^2\,dx.
$$
The <roughness penalty> measures curvature; its null space consists of <affine functions>. To see why the minimizer is a <natural cubic spline>, let $g$ be the natural spline interpolating the values of any competitor $m$ at its distinct observation locations, and put $h=m-g$. <Integration by parts> twice on each knot interval gives $\int g''h''=0$: $g''''=0$ on each interval, $h$ vanishes at the knots, $g''$ is continuous, and $g''$ vanishes at the outer endpoints. Hence
$$
\int(m'')^2=\int(g'')^2+\int(h'')^2\ge\int(g'')^2,
$$
while both functions have the same residual sum of squares. This proves the reduction to a finite-dimensional spline space.
For distinct $X_i$ and $n\ge2$, use its $n$-dimensional <basis>, write $B_{ij}=b_j(X_i)$, and define the <positive semidefinite matrix>
$$
\Omega_{jk}=\int b_j''(x)b_k''(x)\,dx.
$$
The <penalized least squares> problem is $\|Y-B\beta\|^2+\lambda\beta^T\Omega\beta$, so
$$
\boxed{\widehat m_\lambda(x)=b(x)^T(B^TB+\lambda\Omega)^{-1}B^TY.}
$$
Use linear continuation outside the observation interval. The fitted-value matrix is $S_\lambda=B(B^TB+\lambda\Omega)^{-1}B^T$; its trace gives the <effective degrees of freedom>. As $\lambda$ decreases to zero, the natural spline approaches interpolation, with trace $n$; as it grows without bound, the fit approaches <ordinary least squares> on $1,x$, with trace two. For $\lambda=0$ the criterion alone does not determine values between observations; the natural interpolant is the limiting convention. Repeated observation locations are combined by using distinct knots and their multiplicities as weights, rather than pretending the spline design has $n$ independent interpolation values.
Back to article page
