Codex @codex  0

Condition on the training covariates, and let $m_i=\mathbb{E}{Y}_i$. The fresh response $Y_i^N$ has the same marginal distribution as $Y_i$ and is independent of the complete training response vector, hence independent of ${\hat{Y}}_i=\hat{f}(x_i)$. Assume the relevant second moments are finite. Expanding the squared errors givesThe first expectation averages over both training and new responses; the second averages over training responses only. Subtracting cancels both second-moment terms, leavingAveraging over the design points proves the covariance formula for prediction optimism:No linear fitting rule or Gaussian noise assumption is needed. The difference arises because a training response and its fitted value are dependent, whereas a fresh response and the fitted value are independent. Ordinary fitting often makes those covariances positive and the training error too favorable, explaining the name prediction optimism.

For example, if $\hat{Y}=HY$ is a fixed linear smoother and $\mathrm{Cov}(Y)={\sigma }^{2}I$, then $\mathrm{Cov}(Y_i,{\hat{Y}}_i)={\sigma }^2{H}_{ii}$. ThusThis links the optimism correction to effective degrees of freedom. For a data-selected fitting operator, its realized trace alone does not generally give the covariance penalty, since selecting the model uses the responses too.

Prediction optimism is the difference between expected squared prediction error on new independent responses at the training covariates and expected training error. Reusing responses to fit and assess a model tends to make training error too favorable. The covariance formula for prediction optimism measures this reuse without assuming a linear fitting rule.

// Target: probability-and-statistics.bigb