Past exam of the mathematics course of the University of Cambridge / 2007 / iii / Paper 43 / 5

A Gaussian generalised additive model with the identity link specifiesThe $f_j$ are unknown smooth functions estimated from the data, rather than necessarily straight lines. The model is additive in their effects, so it does not automatically contain interactions between covariates. To identify the intercept separately from the smooths, impose a constraint such as ${\sum }_i{f}_j(x_{ij})=0$ for each $j$. Linear terms can also be included alongside smooth terms. More generally a specified link function can make $g(E(Y_i))$ additive; for the normal identity-link model the mean itself has the displayed form. Smoothness is typically controlled through a basis and a roughness penalty, preventing an unrestricted smooth from interpolating all the noise.

A natural cubic spline interpolant $s$ satisfies $s(x_i)=y_i$, is twice continuously differentiable, and is a polynomial of degree at most three on each interval $[x_i,x_{i+1}]$. Its natural boundary conditions areIf defined outside the data interval, it continues linearly on both exterior intervals. Thus function values and the first two derivatives match at the knots, but the third derivative may jump.

For an explicit construction, let $h_i=x_{i+1}-x_i>0$ and $M_i=s^{\prime \prime }(x_i)$. Set $M_1=M_n=0$ and solveOn $[x_i,x_{i+1}]$, putThis formula interpolates the values and matches second derivatives; the tridiagonal equations enforce first-derivative matching. The homogeneous system has only the zero solution: in a row with maximal nonzero $|M_i|$, its diagonal coefficient strictly exceeds the sum of its neighbour coefficients, making cancellation impossible. Thus the natural interpolant exists uniquely.

With the convention that the residual term is a sum rather than an average, the penalised least squares criterion isMinimize over functions whose first derivative is absolutely continuous and whose second derivative is square-integrable, that is, the usual Sobolev space $H^2[x_1,x_n]$. The cubic smoothing spline ${\hat{g}}_{\lambda }$ is its minimizer, with natural boundary conditions and linear continuation outside the data range. It is a natural cubic spline with possible knots at the observed $x_i$, but usually does not interpolate the noisy $y_i$.

To see why the minimizer is a natural spline, for any $g$ take the natural cubic spline interpolant $s$ through its values $g(x_i)$ and write $h=g-s$. Then $h(x_i)=0$. Integrating $\int s^{\prime \prime }{h}^{\prime \prime }$ by parts on every knot interval twice gives zero: the terms involving $h$ vanish at knots, the terms involving $h^{\prime }$ cancel because $s^{\prime \prime }$ is continuous there, and the outer terms vanish by the natural boundary conditions. Hencewhile their residual sums of squares agree. Replacing $g$ by $s$ never increases the criterion. This proves the minimum roughness property of the natural cubic spline interpolant needed here.

For fitted values $u_i=s(x_i)$, write $\int (s^{\prime \prime }{)}^2=u^{T}Ku$, where $K$ is positive semidefinite and has the sampled affine functions as its two-dimensional nullspace. The finite-dimensional minimization becomes $\Vert y-u{\Vert }^2+\lambda u^{T}Ku$. Its strictly positive definite Hessian is $2(I+\lambda K)$, so it has the unique solutionThis is the roughness-matrix formula for a natural cubic smoothing spline. Larger $\lambda$ penalizes curvature more strongly. As $\lambda \downarrow 0$ the fit approaches interpolation; as $\lambda \to \infty$ it approaches the ordinary least-squares affine line.

Let $A=A_{\lambda }$ be the influence matrix, so $\hat{y}=Ay$. Its trace is the effective degrees of freedom, accounting for shrinkage of the fitted smooth rather than just counting basis coefficients. The generalised cross-validation score isFor a fixed smoothing parameter, the exact leave-one-out residual of this penalized linear fit is $(y_i-{\hat{y}}_i)/(1-A_{ii})$, obtained by removing its observation's contribution to the normal equations. Thus Leave-one-out cross-validation uses the average of their squared values. GCV replaces the individual leverages $A_{ii}$ by their average $\mathrm{tr}(A)/n$. This avoids separate refits and gives a criterion balancing the small training residuals of a flexible fit against its larger complexity.

Evaluate the score across positive $\lambda$, or optimize it numerically, and choose $\hat{\lambda }$ minimizing GCV. Then refit or retain the spline at this value. In the full natural smoothing-spline family the trace decreases from $n$ towards $2$ as smoothing increases. The selected parameter is intended to give good prediction, rather than necessarily to pass through every point.

library(mgcv) loads the fitting package; gam fits a Gaussian generalised additive model with the printed identity link. The smooth specification s(x, bs="cr") uses a penalized cubic regression spline. This is a finite-dimensional smooth basis, rather than a full interpolating spline with a free value at all $401$ observations. The fitted model isHere $f(x)={\sum }_{r=1}^q{\theta }_r{B}_r(x)$ in the centred cubic regression-spline basis, with its curvature penalized through $\lambda \int (f^{\prime \prime }{)}^2$. The intercept is estimated separately. In the displayed fit the smooth test space has rank $9$, consistent with nine smooth coefficients after the centring constraint, while penalization lowers its effective dimension.

The intercept estimate is $-0.13241$, with standard error $0.05426$, $t=-2.44$ and $p=0.0151$. Because the smooth is centred over the observations, the intercept is the overall fitted average, not necessarily the fitted response at $x=0$. Its test compares this average with zero. The significance-code legend maps progressively smaller $p$-values to stars; stars are an annotation, not an effect-size measure.

The smooth has $8.237$ effective degrees of freedom, displaying substantial nonlinearity: a centred purely linear term would use one. Est.rank=9 reports the rank used for its approximate significance calculation. The $p$-value below $2\times 10^{-16}$ is very strong evidence that the mean depends on $x$, against a zero smooth term. It does not mean the fitted curve is the exact generating function, or that lack of fit has been tested. The total effective dimension including the intercept is $1+8.237=9.237$.

The adjusted coefficient of determination $R^2$ of $0.983$ and deviance explained of $98.4\%$ indicate that nearly all variation about the response mean is explained. For this Gaussian model, deviance explained is $1-\mathrm{RSS}/\mathrm{SST}$. The scale estimate $0.96737$ estimates residual variance, corresponding to a residual standard deviation about $0.984$. The GCV score $0.99018$ is the prediction-oriented criterion used for smoothing selection, not a probability. It is consistent with the reported scale and effective dimension: $0.96737\times 401/(401-9.237)\simeq 0.99018$.

In the PDF's plot, the solid curve is the estimated centred smooth $f(x)$, and the dashed curves show pointwise uncertainty around that smooth, rather than prediction intervals for new noisy responses. The label s(x,8.24) gives its rounded effective degrees of freedom. The rug marks the observed covariate locations. Reading the endpoints givesThe smooth is increasing with an almost flat middle and one inflection, resembling a cubic. A simpler model to investigate is ordinary Gaussian polynomial regressionIt uses four mean coefficients rather than about nine effective ones. The apparent near-odd shape also makes ${\beta }_0+{\beta }_3{x}^3$ a plausible further reduction, but the linear and quadratic terms should be checked rather than discarded solely from the picture. The figure suggests this alternative; it does not supply coefficients or establish the true simulation model.