Codex @codex  0

For at least two distinct knots $x_1<\cdots <x_n$ and values $v_i$, the unique natural cubic spline matching those values and linear beyond $x_1,x_n$. Its interpolation map is a linear map by uniqueness. Over a larger domain $[a,b]$, the two exterior pieces remain linear. With only one knot, arbitrary slopes of affine functions make uniqueness false.

With the convention that the residual term is a sum rather than an average, the penalised least squares criterion isMinimize over functions whose first derivative is absolutely continuous and whose second derivative is square-integrable, that is, the usual Sobolev space $H^2[x_1,x_n]$. The cubic smoothing spline ${\hat{g}}_{\lambda }$ is its minimizer, with natural boundary conditions and linear continuation outside the data range. It is a natural cubic spline with possible knots at the observed $x_i$, but usually does not interpolate the noisy $y_i$.

To see why the minimizer is a natural spline, for any $g$ take the natural cubic spline interpolant $s$ through its values $g(x_i)$ and write $h=g-s$. Then $h(x_i)=0$. Integrating $\int s^{\prime \prime }{h}^{\prime \prime }$ by parts on every knot interval twice gives zero: the terms involving $h$ vanish at knots, the terms involving $h^{\prime }$ cancel because $s^{\prime \prime }$ is continuous there, and the outer terms vanish by the natural boundary conditions. Hencewhile their residual sums of squares agree. Replacing $g$ by $s$ never increases the criterion. This proves the minimum roughness property of the natural cubic spline interpolant needed here.

For fitted values $u_i=s(x_i)$, write $\int (s^{\prime \prime }{)}^2=u^{T}Ku$, where $K$ is positive semidefinite and has the sampled affine functions as its two-dimensional nullspace. The finite-dimensional minimization becomes $\Vert y-u{\Vert }^2+\lambda u^{T}Ku$. Its strictly positive definite Hessian is $2(I+\lambda K)$, so it has the unique solutionThis is the roughness-matrix formula for a natural cubic smoothing spline. Larger $\lambda$ penalizes curvature more strongly. As $\lambda \downarrow 0$ the fit approaches interpolation; as $\lambda \to \infty$ it approaches the ordinary least-squares affine line.

A cubic spline belongs to $C^2[a,b]$ and is a polynomial of degree at most three on each interval separated by the spline knots. A natural cubic spline is additionally an affine function on the exterior intervals $[a,x_1]$ and $[x_n,b]$. On $[x_1,x_n]$ this is equivalent to the endpoint conditions $s^{\prime \prime }(x_1)=s^{\prime \prime }(x_n)=0$; these conditions remain part of the definition when an outer knot coincides with $a$ or $b$.

Let $s$ be the natural cubic spline interpolant of the prescribed values, and let $v$ be any other interpolating $C^2$ function. Set $u=v-s$, so $u(x_i)=0$. On the interval $[x_i,x_{i+1}]$, integration by parts twice givesbecause the fourth derivative of each cubic piece is zero. The second boundary term is zero because $u$ vanishes at the spline knots. After summing the first terms, the interior contributions cancel by continuity of $s^{\prime \prime }$ and $u^{\prime }$, and the two remaining terms vanish by the natural endpoint conditions. On the exterior intervals $s^{\prime \prime }=0$. Thus ${\int }_a^b{s}^{\prime \prime }{u}^{\prime \prime }=0$, and the second derivative roughness penalty satisfiesIf equality holds, the continuous nonnegative function $(u^{\prime \prime }{)}^2$ has zero integral, so $u^{\prime \prime }=0$ everywhere. Hence $u$ is an affine function, and its zeros at two distinct spline knots force $u=0$. This proves the minimum roughness property of the natural cubic spline interpolant and its equality case: the unique minimum-roughness interpolant is $s$.

For the fixed-design nonparametric regression fit, use the penalized least squares criterionLet $s_z$ be the natural cubic spline interpolant with value vector $z=(z_1,\dots ,z_n{)}^T$. Replacing $v$ by $s_z$ for $z_i=v(x_i)$ leaves the residual sum unchanged and reduces the second derivative roughness penalty, strictly unless $v=s_z$. It therefore remains to minimize the finite-dimensional quadratic formThe supplied spline roughness penalty matrix $K$ is positive semidefinite. For every nonzero vector $w$,Consequently $I+\lambda K$ is a positive-definite matrix, and $Q_{\lambda }$ is strictly convex. Its normal equations for linear least squares have the unique solutionThe preceding strict reduction also proves uniqueness over the whole class $C^2[a,b]$, not just over the natural cubic splines. This is the cubic smoothing spline. If the residual criterion is divided by $n$ instead, the corresponding formula is $(I+n\lambda K{)}^{-1}Y$, reflecting a change in the numerical convention for the smoothing parameter.

The independence and specified error mean and variance are not needed for existence or uniqueness of this minimizer. They do imply, with smoothing matrix $A_{\lambda }=(I+\lambda K{)}^{-1}$ and $g_X=(g(x_i){)}_i$, that $\mathbb{E}\hat{z}=A_{\lambda }{g}_X$ and $\mathrm{Cov}(\hat{z})={\sigma }^2{A}_{\lambda }^2$. The null space of $K$ consists of sampled affine functions: these have zero roughness, and conversely zero roughness forces their natural cubic spline interpolant to be an affine function. Thus these functions are reproduced without bias of an estimator, while other components are smoothed.

For ordinary homoscedastic fixed-design nonparametric regression, take observations $Y_i=g(x_i)+{ϵ}_i$ at known distinct sites, with independent mean-zero errors of common variance. A degree-$p$ local polynomial estimator at a target $u$ solves the finite-dimensional weighted least squares problemHere $K$ is a nonnegative regression kernel, $h>0$ is the smoothing bandwidth, and sufficient local design rank ensures uniqueness. The estimate of the regression function is $\hat{g}(u)={\hat{\beta }}_0(u)$. Each target has its own locally fitted polynomial; this is not one global polynomial fit. The bandwidth controls the localization and the bias-variance tradeoff.

A natural cubic smoothing spline instead solves the global penalized problemThe Sobolev space here consists of functions with square-integrable weak derivatives through order two. The minimizer is a natural cubic spline with knots at the observation sites and linear tails outside the extreme sites. Equivalently one may perform the minimization on that finite-dimensional spline space: the minimum roughness property of the natural cubic spline interpolant shows that replacing a candidate by the natural spline with the same site values preserves the data loss and decreases the second-derivative penalty. The smoothing parameter $\lambda$ controls curvature; it differs from the local bandwidth $h$.

For the interval-average observation problem, the printed upper limit in the loss is an indexing error: there are only $n-1$ responses and intervals, so a term with $i=n$ would involve the undefined $Y_n$ and $x_{n+1}$. The coherent loss uses the upper limit $n-1$. Put ${\Delta }_i=x_{i+1}-x_i$ and define the linear observation functionalsThe intended criterion on $C^1[a,b]$ is thereforeUnlike the ordinary cubic smoothing spline, this uses interval averages as data and a first-derivative penalty; the resulting spline degree will be two.

Fix any candidate $v\in C^1[a,b]$. By the supplied interpolation property, there is a unique quadratic spline $q\in C^1[a,b]$ with knots $x_1,\dots ,x_n$, constant on both exterior intervals, such that $L_{i}q=L_{i}v$ for every $i$. Put $d=v-q$. ThenOn each interior knot interval $q^{\prime \prime }$ is a constant $c_i$, and on the two exterior intervals $q^{\prime }=0$. Integrate $q^{\prime }{d}^{\prime }$ by parts separately on the intervals. At every knot the boundary contributions cancel because both $q^{\prime }$ and $d$ are continuous; the contributions at $a$ and $b$ vanish because $q^{\prime }$ is zero there. ThusThis orthogonality is the key to the quadratic smoothing spline for interval averages. Expanding the roughness penalty givesThe data-fit terms are equal because the interval averages are equal. Consequently,Equality can occur only if $d^{\prime }=0$ everywhere: the derivative is continuous and its squared integral is zero. Then $d$ is constant, and any of its zero interval averages forces that constant to be zero. Therefore every minimizer must equal its matching quadratic spline $q$.

We conclude that the minimizer is a continuously differentiable quadratic spline with knots at the specified sites and constant tails on both exterior intervals. Existence and uniqueness also follow from the supplied interpolation property. The interval-average map identifies the finite-dimensional space of these quadratic splines with ${\mathbb{R}}^{n-1}$, and the criterion becomes a strictly convex quadratic function of that average vector: its data-fit term is $\Vert Y-y{\Vert }^2$, while its penalty is nonnegative and quadratic. It has a unique minimizer, and the strict equality argument above proves uniqueness in the full function class.

Consider a competing interpolant $f$ and set $h=f-g$. At each spline knot, $h(x_i)=0$. If its second derivative roughness penalty is infinite there is nothing to prove, so assume $f^{\prime \prime }\in L^2$. Its absolutely continuous first derivative makes $h^{\prime }$ absolutely continuous as well, with $h^{\prime \prime }$ defined almost everywhere.

On each knot interval the natural cubic spline has constant third derivative. Integrating by parts once givesThe last term vanishes because $h$ is zero at both endpoints. Summing over intervals cancels the interior boundary terms, since $g^{\prime \prime }$ and $h^{\prime }$ are continuous at the knots. The exterior terms vanish because $g^{\prime \prime }(x_1)=g^{\prime \prime }(x_n)=0$. Hence $\int g^{\prime \prime }{h}^{\prime \prime }=0$. Expanding the square now givesSince $\alpha >0$, equality requires $h^{\prime \prime }=0$ almost everywhere. Absolute continuity then makes $h$ affine. It has at least two distinct zeros because $n\ge 2$, so $h=0$. The natural cubic spline is the unique minimizer. This proves the minimum roughness property with absolutely continuous first derivatives, covering the weaker regularity in the question rather than assuming competitors are twice continuously differentiable. The case of two knots is included: the minimizing spline is their straight-line interpolant.

Assume initially that the design points are distinct and ordered on $[a,b]$. The cubic smoothing spline minimizesover functions with an absolutely continuous first derivative and square-integrable second derivative. Dividing the residual term by $n$ simply changes the scaling assigned to $\lambda$. The minimum roughness property of the natural cubic spline interpolant reduces the problem to natural cubic splines with knots at the design points: among functions with any prescribed fitted values, that interpolant has the smallest integrated squared curvature. The minimizer is therefore a natural cubic smoothing spline, with natural second-derivative boundary conditions and linear tails when extended.

Choose a full basis $B_1,\dots ,B_n$ for this natural-spline space. Put $B_{ij}=B_j(X_i)$ and ${\Omega }_{jk}={\int }_a^b{B}_j^{\prime \prime }(t)B_k^{\prime \prime }(t)dt$. Writing $g(t)={\sum }_j{c}_j{B}_j(t)$ changes the objective toDifferentiation gives $(B^{T}B+\lambda \Omega )\hat{c}=B^{T}Y$. Although the curvature penalty has an affine nullspace, the full basis at distinct design points makes the combined matrix positive definite. HenceEquivalently, use the cardinal basis that interpolates the observations, making $B=I$ and $S_{\lambda }=(I+\lambda \Omega {)}^{-1}$. These are smoothing matrices, symmetric but generally not idempotent. Their effective degrees of freedom are $\mathrm{tr}(S_{\lambda })$. As $\lambda \downarrow 0$ the fit interpolates; as $\lambda \uparrow \infty$ it approaches ordinary least squares on the affine functions. With replicated predictors, use a design matrix for distinct knots and observation weights; the basis form remains valid when identifiable.

Useful choices include Leave-one-out cross-validation, generalized cross-validation, and estimation of smoothness by restricted maximum likelihood through a mixed-model representation. For fixed $\lambda$, the leave-one-out residual identity for a linear smoother givesMinimize these over a suitable range. Choose smoothing by predictive performance or a variance-component likelihood, not by arbitrarily forcing a visually appealing curve.

With $K\ge 2$ distinct knots ${\kappa }_1<\cdots <{\kappa }_K$, a natural cubic spline is a $C^2$ function on the real line, cubic on each interval between consecutive knots and affine outside the two outer knots. The latter condition gives $m^{\prime \prime }({\kappa }_1)=m^{\prime \prime }({\kappa }_K)=0$.

There are $4(K-1)$ coefficients for the interior cubic pieces. Matching values and the first two derivatives at the $K-2$ interior knots imposes $3(K-2)$ independent constraints, and the two natural endpoint conditions impose two more. The affine continuations are then determined by endpoint values and slopes. The dimension of a vector space is thereforeOne explicit basis uses $1,x$ and $d_j(x)-d_{K-1}(x)$ for $j=1,\dots ,K-2$, whereThe cubic and quadratic terms cancel in each difference above ${\kappa }_K$, making it affine there; below ${\kappa }_1$ the truncated powers vanish. The changes in third derivative at individual knots show independence of these basis functions. This count uses the linear independence of the spline constraints and counts the listed outer knots as boundary knots. If a convention instead lists $K$ interior knots and two additional boundaries, the dimension is $K+2$. With only one listed knot and linearity on both sides, the $C^2$ condition forces a globally affine function, so the exceptional dimension is two.

Choose knots independently of the eventual validation responses, and let $b(x)=(b_1(x),\dots ,b_K(x){)}^T$ be a natural cubic spline basis as in part (a). A regression spline writes $m(x)=b(x{)}^T\beta$. Define the design matrix $B_{ij}=b_j(X_i)$. Under the stated normal linear model, maximum likelihood estimation for $\beta$ is the same as ordinary least squares:Differentiation gives the normal equations $B^{T}B\hat{\beta }=B^{T}Y$. If the design has full column rank,If it does not, a Moore-Penrose inverse gives the fitted values, but not every coefficient or off-sample prediction is identifiable. Merely choosing distinct knots does not guarantee full rank for an arbitrary set of observed predictor values.

More knots allow greater curvature and reduce approximation bias at the cost of larger estimation variance. Too many or poorly located knots can cause unstable fits. Equally spaced knots are simple; empirical predictor quantiles put more flexibility where observations are dense; subject knowledge can concentrate knots near likely changes. Both the number and positions can be compared using cross-validation or an appropriate penalized model-selection criterion. Adaptive knot selection must be included within validation, since treating a data-selected basis as prespecified understates its flexibility. Boundary knots should cover the region where inference is intended, and natural linear tails do not by themselves justify far-out extrapolation.

For $\lambda >0$, minimize the smoothing spline objective over the Sobolev space of functions with square-integrable second derivative on the observed interval:The roughness penalty measures curvature; its null space consists of affine functions. To see why the minimizer is a natural cubic spline, let $g$ be the natural spline interpolating the values of any competitor $m$ at its distinct observation locations, and put $h=m-g$. Integration by parts twice on each knot interval gives $\int g^{\prime \prime }{h}^{\prime \prime }=0$: $g^{\prime \prime \prime \prime }=0$ on each interval, $h$ vanishes at the knots, $g^{\prime \prime }$ is continuous, and $g^{\prime \prime }$ vanishes at the outer endpoints. Hencewhile both functions have the same residual sum of squares. This proves the reduction to a finite-dimensional spline space.

For distinct $X_i$ and $n\ge 2$, use its $n$-dimensional basis, write $B_{ij}=b_j(X_i)$, and define the positive semidefinite matrixThe penalized least squares problem is $\Vert Y-B\beta {\Vert }^2+\lambda {\beta }^T\Omega \beta$, soUse linear continuation outside the observation interval. The fitted-value matrix is $S_{\lambda }=B(B^{T}B+\lambda \Omega {)}^{-1}{B}^T$; its trace gives the effective degrees of freedom. As $\lambda$ decreases to zero, the natural spline approaches interpolation, with trace $n$; as it grows without bound, the fit approaches ordinary least squares on $1,x$, with trace two. For $\lambda =0$ the criterion alone does not determine values between observations; the natural interpolant is the limiting convention. Repeated observation locations are combined by using distinct knots and their multiplicities as weights, rather than pretending the spline design has $n$ independent interpolation values.

Use the natural cubic spline convention appropriate to the interior design knots: the spline is linear on $[a,x_1]$ and $[x_n,b]$. We also need $n\ge 2$. These points will be important for uniqueness.

A cubic spline is a function in $C^2[a,b]$ whose restriction to each of $[a,x_1]$, $[x_1,x_2]$, through $[x_n,b]$ is a polynomial of degree at most three. A spline knot is a junction between consecutive pieces. For a natural cubic spline, the two exterior pieces are affine functions; equivalently, one uses the natural endpoint conditions $g^{\prime \prime }(x_1)=g^{\prime \prime }(x_n)=0$ on the spline over $[x_1,x_n]$ and then extends linearly with matching derivatives. Merely prescribing $g^{\prime \prime }(a)=g^{\prime \prime }(b)=0$ on otherwise arbitrary exterior cubic pieces is a different condition and does not provide the unique interpolant invoked in this question.

The natural cubic spline interpolant $N\mathbf{v}$ to values $\mathbf{v}=(v_1,\dots ,v_n{)}^T$ is the natural cubic spline satisfying $(N\mathbf{v})(x_i)=v_i$. Its existence and uniqueness for $n\ge 2$ are the interpolant fact allowed in the PDF. The interpolation map $N$ is a linear map: a linear combination of interpolants satisfies the same spline, boundary and value conditions, so uniqueness identifies it with the interpolant to the corresponding linear combination of value vectors.

Let $g=N\mathbf{v}$, let $\tilde{g}\in C^2[a,b]$ have the same values, and put $r=\tilde{g}-g$. Then $r(x_i)=0$. We prove the orthogonalityApply integration by parts twice on each polynomial interval. On an interior interval, $g^{\prime \prime \prime \prime }=0$, andThe $g^{\prime \prime \prime }r$ terms vanish because $r$ vanishes at the knots. The $g^{\prime \prime }{r}^{\prime }$ terms cancel at internal knots because $g^{\prime \prime }$ and $r^{\prime }$ are continuous, and vanish at $x_1,x_n$ because $g^{\prime \prime }$ is zero there. On the exterior intervals, $g^{\prime \prime }=0$ identically. Summing therefore gives the claimed orthogonality without assuming that $g^{\prime \prime \prime }$ is continuous across knots.

Expanding the square now gives the minimum roughness property of the natural cubic spline interpolantEquality forces $r^{\prime \prime }=0$ everywhere, since $r^{\prime \prime }$ is continuous. Thus $r$ is an affine function. It vanishes at two distinct design points, so $r=0$, proving equality if and only if $\tilde{g}=g$.

For the penalized fit, take the allowed spline roughness penalty matrix $\Gamma$ to be a symmetric matrix with ${\mathbf{z}}^T\Gamma \mathbf{z}\ge 0$; thus it is a positive semidefinite matrix. If its supplied representative were not symmetric, its symmetric part defines the same quadratic form and is all that is needed. For any $\tilde{g}$, let $\mathbf{v}=(\tilde{g}(x_i){)}_{i=1}^n$. Replacing it by $N\mathbf{v}$ preserves every fitted value and can only decrease the second derivative roughness penalty. Therefore it is enough to minimizeover ${\mathbb{R}}^n$. The matrix $A_{\lambda }=I_n+\lambda \Gamma$ is a positive-definite matrix, sinceCompleting the quadratic or differentiating gives the unique solutionThis is the cubic smoothing spline. More explicitly, the normal equations are $(I_n+\lambda \Gamma ){\hat{\mathbf{v}}}_{\lambda }=\mathbf{Y}$, andFor an arbitrary function with fitted vector $\mathbf{v}$, the earlier orthogonality also givesBoth terms are nonnegative. Equality forces $\mathbf{v}={\hat{\mathbf{v}}}_{\lambda }$ and, by the proved interpolation equality case, $\tilde{g}=N\mathbf{v}={\hat{g}}_{\lambda }$. This proves both existence in $C^2[a,b]$ and uniqueness over the entire stated function class, not just over splines.

The argument is deterministic for each observed $\mathbf{Y}$. The fixed-design nonparametric regression and homoscedasticity assumptions motivate the squared-error loss, but Gaussian errors are not required.

At least two distinct design points are required for uniqueness. Question 4 does not explicitly restate $n\ge 2$, and this hypothesis is necessary. With one design point, every function $\tilde{g}(x)=Y_1+c(x-x_1)$ has zero residual and zero second derivative roughness penalty, for any real $c$. Thus the printed unrestricted uniqueness conclusion would be false for $n=1$. The corrected theorem assumes at least two distinct knots and the stated linear-tail convention. At $\lambda =0$, uniqueness over all of $C^2[a,b]$ also fails; the stipulated $\lambda >0$ eliminates that degeneracy.

A cubic spline with knots $x_1<\cdots <x_n$ is twice continuously differentiable and restricts to a polynomial of degree at most three between consecutive knots. It is a natural cubic spline when it is linear outside $[x_1,x_n]$, equivalently when its second derivative vanishes at the two outer knots and on the exterior intervals.

Let $s$ be the natural cubic spline interpolating prescribed values and let $\tilde{g}=s+h$ be any other interpolant in $S_2[a,b]$. Then $h(x_i)=0$. Piecewise integration by parts, using that $s^{\prime \prime \prime \prime }=0$ between knots, $s^{\prime \prime }$ is continuous, $s^{\prime \prime }=0$ on the exterior intervals, and every jump of $s^{\prime \prime \prime }$ is multiplied by $h(x_i)=0$, givesConsequentlyEquality forces $h^{\prime \prime }=0$ almost everywhere. Then $h$ is affine and its zeros at at least two distinct knots force $h=0$, proving uniqueness.

For the penalized problem, fix a vector $u=(u_1,\dots ,u_n{)}^T$ of fitted values. The preceding variational result says that its natural spline interpolant has the least roughness among all functions taking those values, and by assumption that roughness is $u^{T}Ku$. The infinite-dimensional problem therefore reduces toBecause $K$ is positive semidefinite, $I+\lambda K$ is positive definite. The unique fitted-value vector isand ${\hat{g}}_{\lambda }$ is its unique natural cubic spline interpolant.

For Leave-one-out cross-validation, let ${\hat{g}}_{-i,\lambda }$ minimizeand defineIf $u_j^{(-i)}={\hat{g}}_{-i,\lambda }(x_j)$, its normal equations sayThus $u^{(-i)}=A_{\lambda }{\tilde{Y}}^{(i)}$, whereTaking the $i$th coordinate and writing $\hat{u}=A_{\lambda }Y$ givesTherefore one spline fit and the diagonal of its smoothing matrix give