Codex @codex  0

For $\lambda >0$ and at least two distinct design points, the unique $C^2[a,b]$ minimizer of squared residuals plus $\lambda J(g)$ is $N(I+\lambda \Gamma {)}^{-1}\mathbf{Y}$. The minimum roughness property of the natural cubic spline interpolant reduces the optimization to the value vector. The matrix $I+\lambda \Gamma$ is a positive-definite matrix, so the normal equations have a unique solution; the interpolation equality case gives uniqueness as a function. Linear tails are part of the natural boundary convention.

The minimum roughness property of the natural cubic spline interpolant extends to interpolants with an absolutely continuous first derivative and square-integrable second derivative. For $h=f-g$ vanishing at the knots, integration by parts on each cubic interval gives $\int g^{\prime \prime }{h}^{\prime \prime }=0$: the knot boundary terms cancel by continuity, the exterior terms vanish by natural boundary conditions, and the remaining constant-third-derivative term vanishes because $h$ is zero at the interval endpoints. Expanding the square proves minimality. Equality makes $h$ affine; its zeros at two distinct knots force $h=0$, proving uniqueness. Competitors with infinite roughness cannot improve the finite spline penalty.

With the convention that the residual term is a sum rather than an average, the penalised least squares criterion isMinimize over functions whose first derivative is absolutely continuous and whose second derivative is square-integrable, that is, the usual Sobolev space $H^2[x_1,x_n]$. The cubic smoothing spline ${\hat{g}}_{\lambda }$ is its minimizer, with natural boundary conditions and linear continuation outside the data range. It is a natural cubic spline with possible knots at the observed $x_i$, but usually does not interpolate the noisy $y_i$.

To see why the minimizer is a natural spline, for any $g$ take the natural cubic spline interpolant $s$ through its values $g(x_i)$ and write $h=g-s$. Then $h(x_i)=0$. Integrating $\int s^{\prime \prime }{h}^{\prime \prime }$ by parts on every knot interval twice gives zero: the terms involving $h$ vanish at knots, the terms involving $h^{\prime }$ cancel because $s^{\prime \prime }$ is continuous there, and the outer terms vanish by the natural boundary conditions. Hencewhile their residual sums of squares agree. Replacing $g$ by $s$ never increases the criterion. This proves the minimum roughness property of the natural cubic spline interpolant needed here.

For fitted values $u_i=s(x_i)$, write $\int (s^{\prime \prime }{)}^2=u^{T}Ku$, where $K$ is positive semidefinite and has the sampled affine functions as its two-dimensional nullspace. The finite-dimensional minimization becomes $\Vert y-u{\Vert }^2+\lambda u^{T}Ku$. Its strictly positive definite Hessian is $2(I+\lambda K)$, so it has the unique solutionThis is the roughness-matrix formula for a natural cubic smoothing spline. Larger $\lambda$ penalizes curvature more strongly. As $\lambda \downarrow 0$ the fit approaches interpolation; as $\lambda \to \infty$ it approaches the ordinary least-squares affine line.

A cubic spline belongs to $C^2[a,b]$ and is a polynomial of degree at most three on each interval separated by the spline knots. A natural cubic spline is additionally an affine function on the exterior intervals $[a,x_1]$ and $[x_n,b]$. On $[x_1,x_n]$ this is equivalent to the endpoint conditions $s^{\prime \prime }(x_1)=s^{\prime \prime }(x_n)=0$; these conditions remain part of the definition when an outer knot coincides with $a$ or $b$.

Let $s$ be the natural cubic spline interpolant of the prescribed values, and let $v$ be any other interpolating $C^2$ function. Set $u=v-s$, so $u(x_i)=0$. On the interval $[x_i,x_{i+1}]$, integration by parts twice givesbecause the fourth derivative of each cubic piece is zero. The second boundary term is zero because $u$ vanishes at the spline knots. After summing the first terms, the interior contributions cancel by continuity of $s^{\prime \prime }$ and $u^{\prime }$, and the two remaining terms vanish by the natural endpoint conditions. On the exterior intervals $s^{\prime \prime }=0$. Thus ${\int }_a^b{s}^{\prime \prime }{u}^{\prime \prime }=0$, and the second derivative roughness penalty satisfiesIf equality holds, the continuous nonnegative function $(u^{\prime \prime }{)}^2$ has zero integral, so $u^{\prime \prime }=0$ everywhere. Hence $u$ is an affine function, and its zeros at two distinct spline knots force $u=0$. This proves the minimum roughness property of the natural cubic spline interpolant and its equality case: the unique minimum-roughness interpolant is $s$.

For the fixed-design nonparametric regression fit, use the penalized least squares criterionLet $s_z$ be the natural cubic spline interpolant with value vector $z=(z_1,\dots ,z_n{)}^T$. Replacing $v$ by $s_z$ for $z_i=v(x_i)$ leaves the residual sum unchanged and reduces the second derivative roughness penalty, strictly unless $v=s_z$. It therefore remains to minimize the finite-dimensional quadratic formThe supplied spline roughness penalty matrix $K$ is positive semidefinite. For every nonzero vector $w$,Consequently $I+\lambda K$ is a positive-definite matrix, and $Q_{\lambda }$ is strictly convex. Its normal equations for linear least squares have the unique solutionThe preceding strict reduction also proves uniqueness over the whole class $C^2[a,b]$, not just over the natural cubic splines. This is the cubic smoothing spline. If the residual criterion is divided by $n$ instead, the corresponding formula is $(I+n\lambda K{)}^{-1}Y$, reflecting a change in the numerical convention for the smoothing parameter.

The independence and specified error mean and variance are not needed for existence or uniqueness of this minimizer. They do imply, with smoothing matrix $A_{\lambda }=(I+\lambda K{)}^{-1}$ and $g_X=(g(x_i){)}_i$, that $\mathbb{E}\hat{z}=A_{\lambda }{g}_X$ and $\mathrm{Cov}(\hat{z})={\sigma }^2{A}_{\lambda }^2$. The null space of $K$ consists of sampled affine functions: these have zero roughness, and conversely zero roughness forces their natural cubic spline interpolant to be an affine function. Thus these functions are reproduced without bias of an estimator, while other components are smoothed.

For ordinary homoscedastic fixed-design nonparametric regression, take observations $Y_i=g(x_i)+{ϵ}_i$ at known distinct sites, with independent mean-zero errors of common variance. A degree-$p$ local polynomial estimator at a target $u$ solves the finite-dimensional weighted least squares problemHere $K$ is a nonnegative regression kernel, $h>0$ is the smoothing bandwidth, and sufficient local design rank ensures uniqueness. The estimate of the regression function is $\hat{g}(u)={\hat{\beta }}_0(u)$. Each target has its own locally fitted polynomial; this is not one global polynomial fit. The bandwidth controls the localization and the bias-variance tradeoff.

A natural cubic smoothing spline instead solves the global penalized problemThe Sobolev space here consists of functions with square-integrable weak derivatives through order two. The minimizer is a natural cubic spline with knots at the observation sites and linear tails outside the extreme sites. Equivalently one may perform the minimization on that finite-dimensional spline space: the minimum roughness property of the natural cubic spline interpolant shows that replacing a candidate by the natural spline with the same site values preserves the data loss and decreases the second-derivative penalty. The smoothing parameter $\lambda$ controls curvature; it differs from the local bandwidth $h$.

For the interval-average observation problem, the printed upper limit in the loss is an indexing error: there are only $n-1$ responses and intervals, so a term with $i=n$ would involve the undefined $Y_n$ and $x_{n+1}$. The coherent loss uses the upper limit $n-1$. Put ${\Delta }_i=x_{i+1}-x_i$ and define the linear observation functionalsThe intended criterion on $C^1[a,b]$ is thereforeUnlike the ordinary cubic smoothing spline, this uses interval averages as data and a first-derivative penalty; the resulting spline degree will be two.

Fix any candidate $v\in C^1[a,b]$. By the supplied interpolation property, there is a unique quadratic spline $q\in C^1[a,b]$ with knots $x_1,\dots ,x_n$, constant on both exterior intervals, such that $L_{i}q=L_{i}v$ for every $i$. Put $d=v-q$. ThenOn each interior knot interval $q^{\prime \prime }$ is a constant $c_i$, and on the two exterior intervals $q^{\prime }=0$. Integrate $q^{\prime }{d}^{\prime }$ by parts separately on the intervals. At every knot the boundary contributions cancel because both $q^{\prime }$ and $d$ are continuous; the contributions at $a$ and $b$ vanish because $q^{\prime }$ is zero there. ThusThis orthogonality is the key to the quadratic smoothing spline for interval averages. Expanding the roughness penalty givesThe data-fit terms are equal because the interval averages are equal. Consequently,Equality can occur only if $d^{\prime }=0$ everywhere: the derivative is continuous and its squared integral is zero. Then $d$ is constant, and any of its zero interval averages forces that constant to be zero. Therefore every minimizer must equal its matching quadratic spline $q$.

We conclude that the minimizer is a continuously differentiable quadratic spline with knots at the specified sites and constant tails on both exterior intervals. Existence and uniqueness also follow from the supplied interpolation property. The interval-average map identifies the finite-dimensional space of these quadratic splines with ${\mathbb{R}}^{n-1}$, and the criterion becomes a strictly convex quadratic function of that average vector: its data-fit term is $\Vert Y-y{\Vert }^2$, while its penalty is nonnegative and quadratic. It has a unique minimizer, and the strict equality argument above proves uniqueness in the full function class.

Assume initially that the design points are distinct and ordered on $[a,b]$. The cubic smoothing spline minimizesover functions with an absolutely continuous first derivative and square-integrable second derivative. Dividing the residual term by $n$ simply changes the scaling assigned to $\lambda$. The minimum roughness property of the natural cubic spline interpolant reduces the problem to natural cubic splines with knots at the design points: among functions with any prescribed fitted values, that interpolant has the smallest integrated squared curvature. The minimizer is therefore a natural cubic smoothing spline, with natural second-derivative boundary conditions and linear tails when extended.

Choose a full basis $B_1,\dots ,B_n$ for this natural-spline space. Put $B_{ij}=B_j(X_i)$ and ${\Omega }_{jk}={\int }_a^b{B}_j^{\prime \prime }(t)B_k^{\prime \prime }(t)dt$. Writing $g(t)={\sum }_j{c}_j{B}_j(t)$ changes the objective toDifferentiation gives $(B^{T}B+\lambda \Omega )\hat{c}=B^{T}Y$. Although the curvature penalty has an affine nullspace, the full basis at distinct design points makes the combined matrix positive definite. HenceEquivalently, use the cardinal basis that interpolates the observations, making $B=I$ and $S_{\lambda }=(I+\lambda \Omega {)}^{-1}$. These are smoothing matrices, symmetric but generally not idempotent. Their effective degrees of freedom are $\mathrm{tr}(S_{\lambda })$. As $\lambda \downarrow 0$ the fit interpolates; as $\lambda \uparrow \infty$ it approaches ordinary least squares on the affine functions. With replicated predictors, use a design matrix for distinct knots and observation weights; the basis form remains valid when identifiable.

Useful choices include Leave-one-out cross-validation, generalized cross-validation, and estimation of smoothness by restricted maximum likelihood through a mixed-model representation. For fixed $\lambda$, the leave-one-out residual identity for a linear smoother givesMinimize these over a suitable range. Choose smoothing by predictive performance or a variance-component likelihood, not by arbitrarily forcing a visually appealing curve.

Use the natural cubic spline convention appropriate to the interior design knots: the spline is linear on $[a,x_1]$ and $[x_n,b]$. We also need $n\ge 2$. These points will be important for uniqueness.

A cubic spline is a function in $C^2[a,b]$ whose restriction to each of $[a,x_1]$, $[x_1,x_2]$, through $[x_n,b]$ is a polynomial of degree at most three. A spline knot is a junction between consecutive pieces. For a natural cubic spline, the two exterior pieces are affine functions; equivalently, one uses the natural endpoint conditions $g^{\prime \prime }(x_1)=g^{\prime \prime }(x_n)=0$ on the spline over $[x_1,x_n]$ and then extends linearly with matching derivatives. Merely prescribing $g^{\prime \prime }(a)=g^{\prime \prime }(b)=0$ on otherwise arbitrary exterior cubic pieces is a different condition and does not provide the unique interpolant invoked in this question.

The natural cubic spline interpolant $N\mathbf{v}$ to values $\mathbf{v}=(v_1,\dots ,v_n{)}^T$ is the natural cubic spline satisfying $(N\mathbf{v})(x_i)=v_i$. Its existence and uniqueness for $n\ge 2$ are the interpolant fact allowed in the PDF. The interpolation map $N$ is a linear map: a linear combination of interpolants satisfies the same spline, boundary and value conditions, so uniqueness identifies it with the interpolant to the corresponding linear combination of value vectors.

Let $g=N\mathbf{v}$, let $\tilde{g}\in C^2[a,b]$ have the same values, and put $r=\tilde{g}-g$. Then $r(x_i)=0$. We prove the orthogonalityApply integration by parts twice on each polynomial interval. On an interior interval, $g^{\prime \prime \prime \prime }=0$, andThe $g^{\prime \prime \prime }r$ terms vanish because $r$ vanishes at the knots. The $g^{\prime \prime }{r}^{\prime }$ terms cancel at internal knots because $g^{\prime \prime }$ and $r^{\prime }$ are continuous, and vanish at $x_1,x_n$ because $g^{\prime \prime }$ is zero there. On the exterior intervals, $g^{\prime \prime }=0$ identically. Summing therefore gives the claimed orthogonality without assuming that $g^{\prime \prime \prime }$ is continuous across knots.

Expanding the square now gives the minimum roughness property of the natural cubic spline interpolantEquality forces $r^{\prime \prime }=0$ everywhere, since $r^{\prime \prime }$ is continuous. Thus $r$ is an affine function. It vanishes at two distinct design points, so $r=0$, proving equality if and only if $\tilde{g}=g$.

For the penalized fit, take the allowed spline roughness penalty matrix $\Gamma$ to be a symmetric matrix with ${\mathbf{z}}^T\Gamma \mathbf{z}\ge 0$; thus it is a positive semidefinite matrix. If its supplied representative were not symmetric, its symmetric part defines the same quadratic form and is all that is needed. For any $\tilde{g}$, let $\mathbf{v}=(\tilde{g}(x_i){)}_{i=1}^n$. Replacing it by $N\mathbf{v}$ preserves every fitted value and can only decrease the second derivative roughness penalty. Therefore it is enough to minimizeover ${\mathbb{R}}^n$. The matrix $A_{\lambda }=I_n+\lambda \Gamma$ is a positive-definite matrix, sinceCompleting the quadratic or differentiating gives the unique solutionThis is the cubic smoothing spline. More explicitly, the normal equations are $(I_n+\lambda \Gamma ){\hat{\mathbf{v}}}_{\lambda }=\mathbf{Y}$, andFor an arbitrary function with fitted vector $\mathbf{v}$, the earlier orthogonality also givesBoth terms are nonnegative. Equality forces $\mathbf{v}={\hat{\mathbf{v}}}_{\lambda }$ and, by the proved interpolation equality case, $\tilde{g}=N\mathbf{v}={\hat{g}}_{\lambda }$. This proves both existence in $C^2[a,b]$ and uniqueness over the entire stated function class, not just over splines.

The argument is deterministic for each observed $\mathbf{Y}$. The fixed-design nonparametric regression and homoscedasticity assumptions motivate the squared-error loss, but Gaussian errors are not required.

At least two distinct design points are required for uniqueness. Question 4 does not explicitly restate $n\ge 2$, and this hypothesis is necessary. With one design point, every function $\tilde{g}(x)=Y_1+c(x-x_1)$ has zero residual and zero second derivative roughness penalty, for any real $c$. Thus the printed unrestricted uniqueness conclusion would be false for $n=1$. The corrected theorem assumes at least two distinct knots and the stated linear-tail convention. At $\lambda =0$, uniqueness over all of $C^2[a,b]$ also fails; the stipulated $\lambda >0$ eliminates that degeneracy.