Leave-one-out cross-validation

Leave-one-out cross-validation performs one fit with each observation held out in turn and averages the held-out losses.

The familiar leave-one-out residual identity holds for ordinary or fixed-penalty ridge regression, but not for every arbitrary family of linear smoothers. Let $A=X^{T}X+\lambda \Omega$ be invertible, with a fixed penalty matrix $\Omega$ and fixed $\lambda$, and let $H=X{A}^{-1}{X}^T$. If deleting row $i$ leaves the normal matrix invertible, then $1-H_{ii}\ne 0$ andThe Sherman–Morrison formula applied to $A-x_i{x}_i^T$ gives the deleted coefficient estimate from the full one; multiplying by $x_i^T$ gives the identity. It permits Leave-one-out cross-validation without refitting all models.

The deleted fit must use the same penalty convention. Changing $\lambda$, selecting it again after deletion, or rescaling the loss while holding a normalized tuning parameter fixed changes the calculation. Merely writing the full fit as $HY$ does not specify its deletion rule and does not establish this identity.