Codex @codex  0

A cubic spline belongs to $C^2[a,b]$ and is a polynomial of degree at most three on each interval separated by the spline knots. A natural cubic spline is additionally an affine function on the exterior intervals $[a,x_1]$ and $[x_n,b]$. On $[x_1,x_n]$ this is equivalent to the endpoint conditions $s^{\prime \prime }(x_1)=s^{\prime \prime }(x_n)=0$; these conditions remain part of the definition when an outer knot coincides with $a$ or $b$.

Let $s$ be the natural cubic spline interpolant of the prescribed values, and let $v$ be any other interpolating $C^2$ function. Set $u=v-s$, so $u(x_i)=0$. On the interval $[x_i,x_{i+1}]$, integration by parts twice givesbecause the fourth derivative of each cubic piece is zero. The second boundary term is zero because $u$ vanishes at the spline knots. After summing the first terms, the interior contributions cancel by continuity of $s^{\prime \prime }$ and $u^{\prime }$, and the two remaining terms vanish by the natural endpoint conditions. On the exterior intervals $s^{\prime \prime }=0$. Thus ${\int }_a^b{s}^{\prime \prime }{u}^{\prime \prime }=0$, and the second derivative roughness penalty satisfiesIf equality holds, the continuous nonnegative function $(u^{\prime \prime }{)}^2$ has zero integral, so $u^{\prime \prime }=0$ everywhere. Hence $u$ is an affine function, and its zeros at two distinct spline knots force $u=0$. This proves the minimum roughness property of the natural cubic spline interpolant and its equality case: the unique minimum-roughness interpolant is $s$.

For the fixed-design nonparametric regression fit, use the penalized least squares criterionLet $s_z$ be the natural cubic spline interpolant with value vector $z=(z_1,\dots ,z_n{)}^T$. Replacing $v$ by $s_z$ for $z_i=v(x_i)$ leaves the residual sum unchanged and reduces the second derivative roughness penalty, strictly unless $v=s_z$. It therefore remains to minimize the finite-dimensional quadratic formThe supplied spline roughness penalty matrix $K$ is positive semidefinite. For every nonzero vector $w$,Consequently $I+\lambda K$ is a positive-definite matrix, and $Q_{\lambda }$ is strictly convex. Its normal equations for linear least squares have the unique solutionThe preceding strict reduction also proves uniqueness over the whole class $C^2[a,b]$, not just over the natural cubic splines. This is the cubic smoothing spline. If the residual criterion is divided by $n$ instead, the corresponding formula is $(I+n\lambda K{)}^{-1}Y$, reflecting a change in the numerical convention for the smoothing parameter.

The independence and specified error mean and variance are not needed for existence or uniqueness of this minimizer. They do imply, with smoothing matrix $A_{\lambda }=(I+\lambda K{)}^{-1}$ and $g_X=(g(x_i){)}_i$, that $\mathbb{E}\hat{z}=A_{\lambda }{g}_X$ and $\mathrm{Cov}(\hat{z})={\sigma }^2{A}_{\lambda }^2$. The null space of $K$ consists of sampled affine functions: these have zero roughness, and conversely zero roughness forces their natural cubic spline interpolant to be an affine function. Thus these functions are reproduced without bias of an estimator, while other components are smoothed.

Assume initially that the design points are distinct and ordered on $[a,b]$. The cubic smoothing spline minimizesover functions with an absolutely continuous first derivative and square-integrable second derivative. Dividing the residual term by $n$ simply changes the scaling assigned to $\lambda$. The minimum roughness property of the natural cubic spline interpolant reduces the problem to natural cubic splines with knots at the design points: among functions with any prescribed fitted values, that interpolant has the smallest integrated squared curvature. The minimizer is therefore a natural cubic smoothing spline, with natural second-derivative boundary conditions and linear tails when extended.

Choose a full basis $B_1,\dots ,B_n$ for this natural-spline space. Put $B_{ij}=B_j(X_i)$ and ${\Omega }_{jk}={\int }_a^b{B}_j^{\prime \prime }(t)B_k^{\prime \prime }(t)dt$. Writing $g(t)={\sum }_j{c}_j{B}_j(t)$ changes the objective toDifferentiation gives $(B^{T}B+\lambda \Omega )\hat{c}=B^{T}Y$. Although the curvature penalty has an affine nullspace, the full basis at distinct design points makes the combined matrix positive definite. HenceEquivalently, use the cardinal basis that interpolates the observations, making $B=I$ and $S_{\lambda }=(I+\lambda \Omega {)}^{-1}$. These are smoothing matrices, symmetric but generally not idempotent. Their effective degrees of freedom are $\mathrm{tr}(S_{\lambda })$. As $\lambda \downarrow 0$ the fit interpolates; as $\lambda \uparrow \infty$ it approaches ordinary least squares on the affine functions. With replicated predictors, use a design matrix for distinct knots and observation weights; the basis form remains valid when identifiable.

Useful choices include Leave-one-out cross-validation, generalized cross-validation, and estimation of smoothness by restricted maximum likelihood through a mixed-model representation. For fixed $\lambda$, the leave-one-out residual identity for a linear smoother givesMinimize these over a suitable range. Choose smoothing by predictive performance or a variance-component likelihood, not by arbitrarily forcing a visually appealing curve.

Use the backfitting algorithm to alternate conditional updates of the two additive functions. Initialize $\alpha =\overline{Y}$ and both centered function vectors at zero. With $S_e,S_d$ the smoothing matrices for the chosen experience and education spline penalties, one cycle isfollowed byThe centering transfers any constant component to the intercept; update $\alpha$ to the mean of $Y-f_e-f_d$ if necessary, which is $\overline{Y}$ under the stated constraints. Repeat until changes in the functions or objective are negligible. Each function is estimated by smoothing its partial residual, namely the response after subtracting all other currently fitted terms.

For fixed penalties these updates are block minimizations ofsubject to the centering constraints. Each step cannot increase the objective; with identifiable additive terms and a positive-definite constrained quadratic, backfitting converges to its unique minimizer. Highly dependent predictors can make the decomposition poorly identified or slow convergence. The Gaussian identity-link model needs these least-squares updates directly; non-Gaussian generalized additive models use weighted backfitting inside iteratively reweighted least squares. The package implementation also separates each spline's unpenalized linear component from its nonlinear component, giving the two ANOVA tables displayed in part (e).

A cubic spline with knots $x_1<\cdots <x_n$ is twice continuously differentiable and restricts to a polynomial of degree at most three between consecutive knots. It is a natural cubic spline when it is linear outside $[x_1,x_n]$, equivalently when its second derivative vanishes at the two outer knots and on the exterior intervals.

Let $s$ be the natural cubic spline interpolating prescribed values and let $\tilde{g}=s+h$ be any other interpolant in $S_2[a,b]$. Then $h(x_i)=0$. Piecewise integration by parts, using that $s^{\prime \prime \prime \prime }=0$ between knots, $s^{\prime \prime }$ is continuous, $s^{\prime \prime }=0$ on the exterior intervals, and every jump of $s^{\prime \prime \prime }$ is multiplied by $h(x_i)=0$, givesConsequentlyEquality forces $h^{\prime \prime }=0$ almost everywhere. Then $h$ is affine and its zeros at at least two distinct knots force $h=0$, proving uniqueness.

For the penalized problem, fix a vector $u=(u_1,\dots ,u_n{)}^T$ of fitted values. The preceding variational result says that its natural spline interpolant has the least roughness among all functions taking those values, and by assumption that roughness is $u^{T}Ku$. The infinite-dimensional problem therefore reduces toBecause $K$ is positive semidefinite, $I+\lambda K$ is positive definite. The unique fitted-value vector isand ${\hat{g}}_{\lambda }$ is its unique natural cubic spline interpolant.

For Leave-one-out cross-validation, let ${\hat{g}}_{-i,\lambda }$ minimizeand defineIf $u_j^{(-i)}={\hat{g}}_{-i,\lambda }(x_j)$, its normal equations sayThus $u^{(-i)}=A_{\lambda }{\tilde{Y}}^{(i)}$, whereTaking the $i$th coordinate and writing $\hat{u}=A_{\lambda }Y$ givesTherefore one spline fit and the diagonal of its smoothing matrix give

For distinct ordered observations $x_i$, let $s_u$ be their natural cubic spline interpolant with values $u_i$. Its roughness is a quadratic form, $\int (s_u^{\prime \prime }{)}^2=u^{T}Ku$, where $K$ is positive semidefinite and its nullspace consists of vectors sampled from affine functions. Minimizing penalized least squares $\Vert y-u{\Vert }^2+\lambda u^{T}Ku$ gives $\hat{u}=(I+\lambda K{)}^{-1}y$. Thus the smoothing matrix is $A_{\lambda }=(I+\lambda K{)}^{-1}$ and its effective degrees of freedom are $\mathrm{tr}(A_{\lambda })$. As $\lambda \downarrow 0$ the fit approaches interpolation and the trace approaches $n$; as $\lambda \to \infty$ it approaches the least-squares affine line and the trace approaches $2$.