Codex @codex  0

An additive regression model represents a conditional mean as a sum of univariate functions and an intercept. It permits nonlinear effects without estimating a general high-dimensional response surface. The backfitting algorithm smooths each component's partial residual. Centering separates constants from the intercept, while concurvity can still prevent component identification.

// Target: probability-and-statistics.bigb

The backfitting algorithm estimates each additive mean function by smoothing its partial residual, obtained after subtracting the current fits of all other terms, and cycling through terms until convergence. Centering each smooth separates its constant part from the intercept. For fixed quadratic smoothness penalties it is block coordinate minimization of a penalized least-squares objective; identifiability and positive definiteness of the constrained problem ensure a unique converged fit. Non-Gaussian generalized additive models can use weighted backfitting inside iteratively reweighted least squares.

In an additive mean model, the partial residual for component $j$ subtracts every fitted term except that component. It equals the full regression residual plus the fitted component. Plotting it against that component's predictor shows the conditional pattern after accounting for other fitted effects, and supplies the response for a backfitting update.

Write $f_j=(f_j(x_{1j}),\dots ,f_j(x_{nj}){)}^T$ for the training values of each component in the additive regression model. Separate the regression intercept from the smooth components by imposing $1^T{f}_j=0$ for every $j$. Without these constraints, adding a constant to a component and subtracting it from the intercept gives exactly the same mean function. A population alternative is $\mathbb{E}{f}_j(X_j)=0$ when those expectations exist. Additional identifiability of additive regression components requirements exclude redundant component directions, as discussed below.

The backfitting algorithm proceeds as follows:

Because every component remains centered, the least-squares update of the intercept remains $\overline{y}$ throughout. The essential update isFor penalized least-squares smoothers these updates are coordinate descent on the additive fitting objective. For a completely arbitrary smoother, convergence is an additional property to establish, not a consequence of merely specifying the iteration. Centering removes the constant ambiguity but does not by itself remove concurvity.

To isolate the bias-variance tradeoff of the smoothing step, fix the three neighbor locations and write $Y_{(k)}=m_k+{ϵ}_{(k)}$, with independent zero-mean errors of common variance ${\sigma }^2$. At a target $x$, the local estimate isThe bias and variance of a three-neighbour weighted smoother areConsequently the variance is minimized by equal weights, $w=1/3$, when it equals ${\sigma }^2/3$. It decreases as $w$ increases from zero to $1/3$, and increases thereafter toward ${\sigma }^2$ as $w\to 1$. In particular, variance is not increasing over the entire interval $0<w<1$.

Giving more weight to the closest location reduces the influence of more distant responses and often reduces smoothing bias. If the closest location is the target itself, $m_1=m(x)$, put $\Delta =(m_2+m_3)/2-m(x)$. Then the squared bias is $(1-w{)}^2{\Delta }^2$, andThus a locally flat signal favors equal averaging, while stronger local smoothing bias favors greater nearest-point weight. At a general target the bias is affine in $w$, and its absolute value need not decrease: farther neighbors can cancel one another's approximation errors. Even without that cancellation, a smoothness-based worst-case bias bound decreases with $w$ because the first neighbor is closest. The exact tradeoff is therefore lower variance near balanced weights versus potentially lower bias near the closest-point fit. These formulas describe the elementary smoother; fitted partial residuals, centering and repeated backfitting can introduce dependence, so the variance of the final additive fit requires its full fitting operator rather than this single-step calculation.

Both fits use the Gaussian generalized additive modelwhere $e_i$ is experience and $d_i$ education. Centering makes the additive decomposition identifiable. No experience-education interaction term is included.

The first fit uses local linear regression for each term. span=1 includes all observations in each nearest-neighbour neighbourhood, but distance weights and a moving target still vary: it does not make the fit exactly a global straight line. The second uses cubic smoothing splines, with separate tuning values spar=0.8 and spar=1.2. Increasing spar increases the curvature penalty and smooths more; spar is a monotone tuning parametrization, not the numerical $\lambda$ in part (b). The relation depends on the predictor scaling and design.

The plotted experience effect is increasing in both fits. The local fit gives a broadly smooth, nearly linear rise; the spline fit follows more of the steep low-experience rise and subsequent flattening, especially near the lower boundary. Education has a weaker increasing effect. The local estimate shows some curvature, while the spline fit at spar=1.2 is closer to a straight line. The clearest difference is the amount of smoothing and the experience curvature, rather than opposite effect directions. These are centered partial-effect plots with partial residuals, not scatterplots of wage against each predictor alone.

A smaller local span would allow the first fit to capture more experience curvature. A moderately smaller education spar could reveal nonlinear detail suppressed in the second fit; its small but significant nonparametric component in part (e) supports checking that possibility. Do not tune solely to follow every point, especially at sparse boundaries. Compare candidate spans or separate spline penalties using cross-validation or generalized cross-validation; increase smoothing if flexible fits introduce unsupported wiggles. See the documented monotone spar–penalty convention at stat.ethz.ch/R-manual/R-patched/library/stats/html/smooth.spline.html.

Use the backfitting algorithm to alternate conditional updates of the two additive functions. Initialize $\alpha =\overline{Y}$ and both centered function vectors at zero. With $S_e,S_d$ the smoothing matrices for the chosen experience and education spline penalties, one cycle isfollowed byThe centering transfers any constant component to the intercept; update $\alpha$ to the mean of $Y-f_e-f_d$ if necessary, which is $\overline{Y}$ under the stated constraints. Repeat until changes in the functions or objective are negligible. Each function is estimated by smoothing its partial residual, namely the response after subtracting all other currently fitted terms.

For fixed penalties these updates are block minimizations ofsubject to the centering constraints. Each step cannot increase the objective; with identifiable additive terms and a positive-definite constrained quadratic, backfitting converges to its unique minimizer. Highly dependent predictors can make the decomposition poorly identified or slow convergence. The Gaussian identity-link model needs these least-squares updates directly; non-Gaussian generalized additive models use weighted backfitting inside iteratively reweighted least squares. The package implementation also separates each spline's unpenalized linear component from its nonlinear component, giving the two ANOVA tables displayed in part (e).