Smoothing spline (${\sum }_i(Y_i-g(x_i){)}^2+\lambda {\int }_a^b(g^{(m)}{)}^{2}dx$)

A penalized function estimate balancing a residual loss against an integrated derivative penalty. For squared-error loss and the second derivative roughness penalty, the estimate is a cubic smoothing spline. Higher derivative orders give other spline degrees.

Let $L_{i}g$ be the average of $g$ on successive knot intervals. Minimizing the displayed loss over $C^1[a,b]$ gives a quadratic spline with constant tails outside the first and last knots. To prove the reduction, choose the unique constant-tail quadratic spline $q$ with the same interval averages as a candidate $g$, and put $v=g-q$. Since $q^{\prime \prime }$ is constant on every knot interval, $q^{\prime }$ is continuous and vanishes on both tails, integration by parts gives $\int q^{\prime }{v}^{\prime }=-{\sum }_i{q}^{\prime \prime }{|}_{I_i}{\int }_{I_i}v=0$. Consequently the data loss is unchanged and $\int (g^{\prime }{)}^2=\int (q^{\prime }{)}^2+\int (v^{\prime }{)}^2$. The penalty decreases strictly unless $v$ is constant; its zero interval averages then force $v=0$. The finite-dimensional quadratic optimization on this spline space also gives a unique minimizer because the interval-average map is injective.

For $\lambda >0$ and at least two distinct design points, the unique $C^2[a,b]$ minimizer of squared residuals plus $\lambda J(g)$ is $N(I+\lambda \Gamma {)}^{-1}\mathbf{Y}$. The minimum roughness property of the natural cubic spline interpolant reduces the optimization to the value vector. The matrix $I+\lambda \Gamma$ is a positive-definite matrix, so the normal equations have a unique solution; the interpolation equality case gives uniqueness as a function. Linear tails are part of the natural boundary convention.

For distinct ordered observations $x_i$, let $s_u$ be their natural cubic spline interpolant with values $u_i$. Its roughness is a quadratic form, $\int (s_u^{\prime \prime }{)}^2=u^{T}Ku$, where $K$ is positive semidefinite and its nullspace consists of vectors sampled from affine functions. Minimizing penalized least squares $\Vert y-u{\Vert }^2+\lambda u^{T}Ku$ gives $\hat{u}=(I+\lambda K{)}^{-1}y$. Thus the smoothing matrix is $A_{\lambda }=(I+\lambda K{)}^{-1}$ and its effective degrees of freedom are $\mathrm{tr}(A_{\lambda })$. As $\lambda \downarrow 0$ the fit approaches interpolation and the trace approaches $n$; as $\lambda \to \infty$ it approaches the least-squares affine line and the trace approaches $2$.