Past exam of the mathematics course of the University of Cambridge / 2008 / iii / Paper 45 / 1

Write $f_j=(f_j(x_{1j}),\dots ,f_j(x_{nj}){)}^T$ for the training values of each component in the additive regression model. Separate the regression intercept from the smooth components by imposing $1^T{f}_j=0$ for every $j$. Without these constraints, adding a constant to a component and subtracting it from the intercept gives exactly the same mean function. A population alternative is $\mathbb{E}{f}_j(X_j)=0$ when those expectations exist. Additional identifiability of additive regression components requirements exclude redundant component directions, as discussed below.

The backfitting algorithm proceeds as follows:

Because every component remains centered, the least-squares update of the intercept remains $\overline{y}$ throughout. The essential update isFor penalized least-squares smoothers these updates are coordinate descent on the additive fitting objective. For a completely arbitrary smoother, convergence is an additional property to establish, not a consequence of merely specifying the iteration. Centering removes the constant ambiguity but does not by itself remove concurvity.

There are two distinct identifiability issues. Without centering, constants can move between the regression intercept and the additive components. After centering, the decomposition still fails to be unique if there are nonzero allowed centered perturbations $h_j$ satisfyingThen replacing $f_j$ by $f_j+h_j$ leaves every fitted response unchanged. This nonlinear analogue of multicollinearity is concurvity. For example, if two predictor columns are identical, a centered function $h$ can be added to the effect of the first and subtracted from the effect of the second. Conceptually, the data observe their combined effect but cannot tell which predictor deserves the credit. A unique fitted sum can therefore coexist with nonunique individual effects. Smoothness penalties can distinguish some such decompositions, so a representation ambiguity is not automatically an ambiguity of every penalized estimator.

For a precise criterion for the backfitting algorithm with linear smoothers, put $C=I-11^T/n$, $y_c=Cy$, and $T_j=C{S}_{j}C$. On the centered component space its fixed-point equations are the linear backfitting equationsA consistent system has more than one solution exactly when there is a nonzero centered tuple satisfying $h_j+T_j{\sum }_{k\ne j}{h}_k=0$ for all $j$. In the two-component case, elimination givesThusA common centered direction $h$ reproduced exactly by both smoothers, $T_{1}h=T_{2}h=h$, gives the homogeneous perturbation $(h,-h)$ and is a concrete instance. For example, two identical predictor columns with smoothers that reproduce the same centered linear trend have this property. Approximate concurvity yields an eigenvalue near one, causing poor conditioning and slow or unstable component estimation, but it does not constitute exact nonuniqueness. Failure of an iteration to converge is also distinct from the existence of multiple fixed points.

To isolate the bias-variance tradeoff of the smoothing step, fix the three neighbor locations and write $Y_{(k)}=m_k+{ϵ}_{(k)}$, with independent zero-mean errors of common variance ${\sigma }^2$. At a target $x$, the local estimate isThe bias and variance of a three-neighbour weighted smoother areConsequently the variance is minimized by equal weights, $w=1/3$, when it equals ${\sigma }^2/3$. It decreases as $w$ increases from zero to $1/3$, and increases thereafter toward ${\sigma }^2$ as $w\to 1$. In particular, variance is not increasing over the entire interval $0<w<1$.

Giving more weight to the closest location reduces the influence of more distant responses and often reduces smoothing bias. If the closest location is the target itself, $m_1=m(x)$, put $\Delta =(m_2+m_3)/2-m(x)$. Then the squared bias is $(1-w{)}^2{\Delta }^2$, andThus a locally flat signal favors equal averaging, while stronger local smoothing bias favors greater nearest-point weight. At a general target the bias is affine in $w$, and its absolute value need not decrease: farther neighbors can cancel one another's approximation errors. Even without that cancellation, a smoothness-based worst-case bias bound decreases with $w$ because the first neighbor is closest. The exact tradeoff is therefore lower variance near balanced weights versus potentially lower bias near the closest-point fit. These formulas describe the elementary smoother; fitted partial residuals, centering and repeated backfitting can introduce dependence, so the variance of the final additive fit requires its full fitting operator rather than this single-step calculation.