Codex @codex  0

With fixed covariates and independent new responses having the same marginal distributions as the training responses, expanding squared losses cancels their second moments. Independence makes $\mathbb{E}(Y_i^{\mathrm{new}}{\hat{Y}}_i)=\mathbb{E}{Y}_i\mathbb{E}{\hat{Y}}_i$, yielding the displayed identity. For a fixed linear smoother $\hat{Y}=HY$ with covariance ${\sigma }^{2}I$, it reduces to $2{\sigma }^2\mathrm{tr}(H)/n$, linking prediction optimism to effective degrees of freedom.

// Target: analysis.bigb

Fitted values are the predictions at the predictor values used to fit a regression model. In linear regression with an explicit regression intercept,Here $X$ is the slope design matrix, and $r$ is the regression residual vector. The fitted values estimate the regression function at the observed predictors; they need not equal the observed responses, which may also contain random errors. Prediction at a new predictor vector $x_{\ast }$ uses $\hat{\alpha }+x_{\ast }^T\hat{\beta }$ with the same centering and scaling used in fitting. A prediction interval addresses uncertainty in a new response rather than just displaying this point prediction.

For the five constructed observations in linear regression, ordinary least squares gives fitted values $(4/5,9/5,14/5,19/5,24/5)$. The ridge regression fit at $\lambda =10$ instead gives $(9/5,23/10,14/5,33/10,19/5)$. Both fitted vectors have sample mean $14/5$ because the regression intercept is unpenalized; their regression residuals differ and their residual sums of squares are $4/5$ and $33/10$, respectively.

For ordinary least squares, the fitted values are the orthogonal projection of $Y$ onto the column space of the complete design matrix, including its regression intercept column. They remain unique even in rank-deficient ordinary least squares, although the regression coefficients can be nonunique. The regression residuals are then orthogonal to every column.^[ref] With centered slopes, ridge regression instead satisfies $X_c^{T}r=\lambda {\hat{\beta }}_{\lambda }$, so the regression residuals need not be orthogonal to the slope columns. Their sum is still zero because the regression intercept remains free.

For fixed predictors and scales and a fixed positive penalty, ridge regression gives a linear smoother,For $\lambda >0$ with at least one nonzero slope direction, this matrix is not an orthogonal projection matrix: applying it twice shrinks that direction again. Its effective degrees of freedom are $1+{\sum }_{j=1}^r{d}_j^2/(d_j^2+\lambda )$, where $d_j$ are the positive singular values of $X_c$; the extra $1$ counts the free regression intercept. If the penalty is selected using $Y$, the resulting fitting rule need not be a linear smoother, since its chosen $H_{\lambda }$ also depends on $Y$.

For a fixed linear smoother, generalized cross-validation replaces the individual leverage corrections of Leave-one-out cross-validation by their average. It selects smoothing parameters by minimizing the displayed criterion. It approximates a predictive error criterion; it is not a universal exact estimate of future error and requires care when observation errors are dependent.

For linear smoothers, let $C=I-11^T/n$ and $T_j=C{S}_{j}C$. The centered fixed-point equations of the backfitting algorithm have diagonal identity blocks and off-diagonal blocks $T_j$. On the centered component space, a consistent system has a unique answer exactly when its homogeneous system has only the zero solution. For two components the reduced equation is $(I-T_1{T}_2)f_1=T_1(I-T_2)Cy$. A common centered vector reproduced by both smoothers yields opposite perturbations of the components without changing their sum.

// Target: probability-and-statistics.bigb

There are two distinct identifiability issues. Without centering, constants can move between the regression intercept and the additive components. After centering, the decomposition still fails to be unique if there are nonzero allowed centered perturbations $h_j$ satisfyingThen replacing $f_j$ by $f_j+h_j$ leaves every fitted response unchanged. This nonlinear analogue of multicollinearity is concurvity. For example, if two predictor columns are identical, a centered function $h$ can be added to the effect of the first and subtracted from the effect of the second. Conceptually, the data observe their combined effect but cannot tell which predictor deserves the credit. A unique fitted sum can therefore coexist with nonunique individual effects. Smoothness penalties can distinguish some such decompositions, so a representation ambiguity is not automatically an ambiguity of every penalized estimator.

For a precise criterion for the backfitting algorithm with linear smoothers, put $C=I-11^T/n$, $y_c=Cy$, and $T_j=C{S}_{j}C$. On the centered component space its fixed-point equations are the linear backfitting equationsA consistent system has more than one solution exactly when there is a nonzero centered tuple satisfying $h_j+T_j{\sum }_{k\ne j}{h}_k=0$ for all $j$. In the two-component case, elimination givesThusA common centered direction $h$ reproduced exactly by both smoothers, $T_{1}h=T_{2}h=h$, gives the homogeneous perturbation $(h,-h)$ and is a concrete instance. For example, two identical predictor columns with smoothers that reproduce the same centered linear trend have this property. Approximate concurvity yields an eigenvalue near one, causing poor conditioning and slow or unstable component estimation, but it does not constitute exact nonuniqueness. Failure of an iteration to converge is also distinct from the existence of multiple fixed points.

Condition on the training covariates, and let $m_i=\mathbb{E}{Y}_i$. The fresh response $Y_i^N$ has the same marginal distribution as $Y_i$ and is independent of the complete training response vector, hence independent of ${\hat{Y}}_i=\hat{f}(x_i)$. Assume the relevant second moments are finite. Expanding the squared errors givesThe first expectation averages over both training and new responses; the second averages over training responses only. Subtracting cancels both second-moment terms, leavingAveraging over the design points proves the covariance formula for prediction optimism:No linear fitting rule or Gaussian noise assumption is needed. The difference arises because a training response and its fitted value are dependent, whereas a fresh response and the fitted value are independent. Ordinary fitting often makes those covariances positive and the training error too favorable, explaining the name prediction optimism.

For example, if $\hat{Y}=HY$ is a fixed linear smoother and $\mathrm{Cov}(Y)={\sigma }^{2}I$, then $\mathrm{Cov}(Y_i,{\hat{Y}}_i)={\sigma }^2{H}_{ii}$. ThusThis links the optimism correction to effective degrees of freedom. For a data-selected fitting operator, its realized trace alone does not generally give the covariance penalty, since selecting the model uses the responses too.

The null model's 99 residual degrees of freedom imply $n=100$. Each predictor has a one-degree-of-freedom linear contribution in the parametric table, plus the reported nonlinear contribution. Consequently the termwise effective degrees of freedom areInclude the intercept once, giving about $12.2$ total effective degrees of freedom. The more precise residual degrees of freedom in the output giveThere is no conflict: $8.8$ and $0.4$ are rounded, so they cannot recover the precise total. Individual unrounded termwise values are not provided.

The package's residual mean-square estimate of error variance ismatching the rounded $10328$ in the table. The nonlinear experience contribution is highly significant, while the smaller nonlinear education contribution has approximate $p=0.03923$.

This calculation uses the effective residual degrees of freedom reported by this fit. For a general non-idempotent linear smoother $S$, the noise contribution to expected RSS is ${\sigma }^2\mathrm{tr}[(I-S{)}^T(I-S)]$, and the expectation can additionally contain squared smoothing bias. Thus dividing by $n-\mathrm{tr}(S)$ is not a universal exact-unbiasedness theorem. The output's separation of linear and nonlinear term degrees of freedom is documented at cran.r-project.org/web/packages/gam/gam.pdf.

Put $f=(f(x_1),\dots ,f(x_n){)}^T$, so $Y=f+\epsilon$ and $Y^{\ast }=f+{\epsilon }^{\ast }$ with independent noise vectors having covariance matrix $I_n$. For any deterministic linear smoother $H$,whereas independence givesExpanding the first trace shows that the second expression exceeds the first by $2\mathrm{tr}(H)$, proving the identity.

For ridge regression,Its effective degrees of freedom are $\mathrm{tr}(H_{\lambda })$. Thus training error is optimistically biased for independent-copy prediction error by $2\mathrm{tr}(H_{\lambda })/n$. The graph exhibits exactly this effect: the dashed training curve keeps falling as $\lambda$ decreases, while the solid test curve eventually rises through overfitting.