Codex @codex  0

Choose $\lambda$ to minimize an estimate of out-of-sample mean squared prediction error, commonly K-fold cross-validation or a separate validation set. As $\lambda$ increases, the coefficient vector is shrunk toward zero. This generally increases bias of an estimator but decreases variance of an estimator; the minimizing value balances the two contributions in the bias-variance tradeoff. The independent test set in the question can assess the final choice, but repeatedly selecting $\lambda$ on that same set would cause data leakage.

Write $x\otimes y$ for the rank-one operator $h\mapsto ⟨h,y⟩x$. The estimator is the kernel representation of the empirical covariance operatorSince $\mathbb{E}X=0$,Thus the estimator has the fixed bias of an estimator ${\mu }_0\otimes {\mu }_0$ for $C_X$.

The fourth-moment assumption makes $(X_i-{\mu }_0)\otimes (X_i-{\mu }_0)$ square-integrable in the Hilbert space of Hilbert-Schmidt operators. The weak law of large numbers therefore givesConsequently consistency for $C_X$ holds exactly when ${\mu }_0=0$. More precisely, if $Y=(X-{\mu }_0)\otimes (X-{\mu }_0)$, thenThe Hilbert-space central limit theorem also yieldswhere $G$ is a centered Gaussian random element in the Hilbert-Schmidt operator space with covariance determined by $Y$. Relative to $C_X$, the same fluctuation is displaced by $\sqrt{n}({\mu }_0\otimes {\mu }_0)$ and hence does not have a finite centered limit when ${\mu }_0\ne 0$.

Among all depth-one regression tree splits, the best split separates the second observation from the first and third by cutting advert1 between $0.89$ and $1.13$. The test point with advert1 equal to zero reaches the leaf containing responses $60$ and $53$, so the output is their arithmetic mean,This decision stump makes a piecewise-constant prediction far outside the observed predictor range and cannot extrapolate the spending trend towards the origin. Its shallow structure and leaf averaging keep its variance of an estimator modest, while that extrapolation failure can produce substantial bias of an estimator.

The $L$-nearest-neighbour classifier finds the $L$ training predictors nearest to $x$ and returns the majority class among their labels. Increasing $L$ averages more labels and reduces variance of an estimator, but uses observations farther from $x$ and therefore increases bias of an estimator; decreasing $L$ reverses this bias-variance tradeoff.