Codex @codex  0

For an $n\times p$ design matrix, distinguish the coefficient vector from the fitted-response vector. The ridge regression objective and its solution areThe gradient is $2(X^{T}X+\lambda I_p)\beta -2X^{T}Y$. For $\lambda >0$, $X^{T}X+\lambda I_p$ is a positive-definite matrix, so this is the unique minimizer even if $X$ is rank deficient. The expression with an additional leading $X$ is instead the $n$-vector of fitted values,This distinction corrects the dimensional mismatch in the printed coefficient-estimator notation.

The quadratic penalty discourages large coefficients, exchanging some bias for lower variance. If $X=UD{V}^T$ is a reduced singular value decomposition, with positive singular values $d_k$, thenThe principal-component shrinkage by ridge regression therefore shrinks directions with small $d_k^2$ most strongly. These are precisely the poorly determined directions created by multicollinearity, where ordinary least-squares coefficients can have large variance or fail to be uniquely determined. A positive penalty stabilizes their inversion and sets coefficients in the null space to zero. It does not make the predictors independent or provide a universal risk improvement for every signal. Because the penalty depends on coefficient units, predictors are ordinarily standardized before choosing $\lambda$ by cross-validation. An unpenalized intercept in ridge regression is handled separately by centering; the displayed objective either has no intercept or applies the penalty to every component included in $\beta$.

The ridge regression penalty is quadratic, whereas the Lasso uses an absolute-value penalty. To make the constants explicit, compareBoth trade goodness of fit against coefficient size, and both require sensible predictor scaling and tuning. Ridge continuously shrinks coefficient directions and generally retains all variables. The Lasso's nonsmooth penalty can set coefficients exactly to zero, so it performs variable selection as well as shrinkage. When $X^{T}X=I$, let $z=X^{T}Y$. The distinction is especially transparent:The second formula is soft thresholding. For general designs, the Karush-Kuhn-Tucker conditions for the Lasso give $X_j^T(Y-X\hat{\beta })=t\mathrm{sign}({\hat{\beta }}_j)$ for a nonzero coefficient and $|X_j^T(Y-X\hat{\beta })|\le t$ for a zero coefficient. The latter interval permits exact zeros; the ridge equation instead is $X^T(Y-X\hat{\beta })=\lambda \hat{\beta }$.

Ridge is especially useful for stable prediction when many predictors carry small effects or are highly correlated. Its strictly convex penalty gives unique coefficients for $\lambda >0$ even in a rank-deficient design. It shares an effect among identical predictor columns: for a fixed coefficient sum the minimum quadratic penalty splits that sum equally. The Lasso can give a simpler sparse model when many effects are absent, but selection among strongly correlated predictors can be unstable. With identical columns, any same-sign split of a selected coefficient sum has the same absolute-value penalty and the same fitted values, so the coefficient vector need not be unique. This is a possible design degeneracy, not a claim that every Lasso fit is nonunique.

The penalties also encode different prior shapes: the Gaussian prior yields ridge, while independent double-exponential priors yield a Lasso posterior mode under Gaussian noise. A Lasso mode is not generally its posterior mean. Practically, sparse interpretability favors the Lasso, while stable distributed effects can favor ridge; prediction performance should be checked with cross-validation, and neither method universally dominates the other.

An orthonormal multiresolution analysis is a sequence of closed subspaces $V_j\subset L^2(\mathbb{R})$ satisfying $V_j\subset V_{j+1}$, $\overline{{\bigcup }_j{V}_j}=L^2(\mathbb{R})$, ${\bigcap }_j{V}_j=\{0\}$, and $f\in V_j$ iff $f(2\cdot )\in V_{j+1}$. A scaling function $\varphi$ has integer translates forming an orthonormal basis of $V_0$, and its scaled translates ${\varphi }_{j,k}(x)=2^{j/2}\varphi (2^{j}x-k)$ span $V_j$. Thus $V_j$ contains information at resolution about $2^{-j}$.

The detail space $W_j=V_{j+1}\cap V_j^{\perp }$ is generated by the corresponding wavelet functions ${\psi }_{j,k}(x)=2^{j/2}\psi (2^{j}x-k)$. The orthogonal splitting $V_{j+1}=V_j\oplus W_j$ iterates toAccordingly a square-integrable signal has a coarse-and-detail wavelet series,and its wavelet projection onto $V_J$ keeps the coarse coefficients and details below level $J$. Increasing $J$ adds finer-scale oscillations and localized features. On the real line there are infinitely many translates at each level; for a bounded observation interval, with an appropriate interval or periodic construction, the relevant approximation dimension satisfies $d_J\asymp 2^J$. For the Haar wavelet on a unit interval this is exactly $2^J$, corresponding to a separate mean on each dyadic subinterval.

This hierarchy gives a concrete meaning to wavelet approximation complexity. Coarse resolution uses few coefficients and can miss sharp features; fine resolution uses more coefficients and can fit noise. By orthogonality, projection bias is the energy of the omitted details,For an orthonormal sequence model with independent coefficient noise of variance ${\sigma }^2/n$, estimating the $d_J$ retained coefficients without shrinkage givesThe first term decreases with resolution while the second increases, making the bias-variance tradeoff explicit. Equivalently, a fixed orthogonal regression projection of rank $d_J$ has effective degrees of freedom $d_J$ and prediction optimism $2{\sigma }^2{d}_J/n$ under independent equal-variance errors.

Wavelets also permit local rather than only global complexity control. Thresholding small estimated detail coefficients, for example by soft thresholding, retains strong local structure while suppressing fine-scale noise elsewhere. A sparse retained coefficient set can represent localized features much more economically than keeping every coefficient through a high level. Its locations and scales matter as well as its count. If that set is selected using the same noisy responses, however, selection adds adaptivity: its raw coefficient count is not automatically its covariance-based degrees of freedom. Resolution or threshold tuning should therefore account for prediction error, using an appropriate risk estimate or cross-validation, rather than only decreasing training error.

Use the population as exposure regardless of whether a population-size category is also a covariate. Define a demographic factor with levels $\mathrm{Low}$ for at most 20%, $\mathrm{Medium}$ for $(20,30]$%, and $\mathrm{High}$ above 30%; define $H_{it}=1_{\{P_{it}>250\}}$ if following the suggested population dichotomy. Treat village-size codes as a categorical variable, not equally spaced areas.

A candidate multivariate Poisson generalized linear mixed model retains the random intercept and slope from part (c) and usesThis specifies the suggested village-size-by-population interaction while keeping its main effects. One may add selected time-by-covariate interactions to explain slope heterogeneity, provided information from 32 clusters supports them. Compare continuous demographic and population formulations as a sensitivity analysis: categorization loses information, and the coefficient of a size category measures association beyond the already included exposure effect.

Fit models with the same likelihood convention and conditional distribution when comparing Akaike information criteria; use joint fixed-effect likelihood-ratio tests, boundary-aware parametric bootstrap tests for random effects, residual diagnostics and village-level cross-validation to justify a parsimonious final model. Preserve interactions' main effects and report interval estimates from the final multivariate model, rather than interpreting separate univariate fits. For example, the adjusted inland-versus-coastal rate ratio is $e^{{\theta }_I}$, while a medium-versus-small village ratio is $e^{{\gamma }_2+{\kappa }_{2}H}$; a Wald interval for that contrast uses the full coefficient covariance matrix. Conditional time effects and standardized marginal predictions should be distinguished.

The displayed rows cannot identify this final model: population exceeds 250 in every displayed village-year, no large village is shown, and inland status is confounded with the identity of the two displayed villages. The complete 160-row file is needed for fitted effects, model selection, all-village plots and uncertainty. No numerical best model or evidence of the specified covariate associations can honestly be reported from this excerpt.

Finally, even a significant adjusted association with young-male percentage would not show that young males were involved in most incidents. This is the ecological fallacy, specifically that aggregate composition does not identify offender composition: the same village counts can be assigned to different individual age and sex groups without changing any observed aggregate predictor. Individual incident-level age, sex and involvement records are required to answer that last question. The aggregate dataset cannot establish the claimed majority involvement.

Begin by defining the scientific target: prediction for a new patient, or estimation of adjusted covariate effects. Check event definitions, follow-up origins, right-censoring codes and any delayed entry. The analysis assumes independent patients and censoring that is noninformative conditional on modeled covariates. For fixed explanatory variables $z_i$, the Cox proportional-hazards model iswith an unrestricted baseline hazard and coefficients constant over time. Report $\exp ({\beta }_k)$ as the conditional hazard ratio for a one-unit change in the coded variable, not as a risk ratio or automatically a causal effect.

For untied event times, fit the coefficients by maximizing the Cox partial likelihoodThe baseline cancels within each event risk set. Use an appropriate tied-event method, such as an Efron approximation or an exact method for a genuinely discrete event scale; the Breslow approximation for tied event times is another explicit approximation. Numerical score/information methods fit the model, and inverse information gives conventional covariance estimates for independent subjects. Report coefficient estimates, hazard ratios, uncertainty and the coding that makes their interpretation meaningful. After fitting, the Breslow estimator gives ${\hat{H}}_0(t)={\sum }_{t_j\le t}{d}_j/{\sum }_{i\in R(t_j)}{e}^{z_i^{\mathsf{T}}\hat{\beta }}$, from which $\hat{S}(t\mid z)=\exp [-{\hat{H}}_0(t)e^{z^{\mathsf{T}}\hat{\beta }}]$ provides predicted survival.

Adequacy is broader than significance of coefficients. Inspect influential observations, deviance residuals and data errors. Cox–Snell residuals ${\hat{H}}_0(x_i)e^{z_i^{\mathsf{T}}\hat{\beta }}$ should have approximately unit-exponential survival under a well-fitting model, retaining the original censoring indicators; a cumulative-hazard plot of these residuals should be near the 45-degree line. This is an overall diagnostic, not a substitute for the functional-form and proportionality checks below. For prediction, assess held-out discrimination and survival prediction calibration at prespecified time horizons with methods accounting for censoring. A high concordance index alone does not establish good calibration or correct hazard structure.

Use bootstrap or held-out cross-validation that repeats the complete modeling procedure, including imputation, variable selection and tuning. Validation of only the final fitted coefficients understates overfitting from earlier decisions. Report any substantive lack of fit rather than presenting a single time-independent hazard ratio when the data do not support that representation.

The ordinary least squares objective has gradient $2X^{T}X\beta -2X^{T}Y$. The normal equation therefore givesThe Gram matrix $X^{T}X$ is a positive-definite matrix because $X$ has full column rank. More explicitly, for any $b$,since $X^T(Y-X\hat{\beta })=0$. This proves that the displayed solution is the unique global minimum.

The fitted values are $\hat{Y}=X\hat{\beta }=HY$, where the hat matrix isThe inverse of the symmetric matrix $X^{T}X$ is symmetric, so $H^T=H$. Multiplication gives $H^2=X(X^{T}X{)}^{-1}(X^{T}X)(X^{T}X{)}^{-1}{X}^T=H$. Thus $H$ is the orthogonal projection onto the column space of the design matrix.

The regression residual vector is $e=Y-\hat{Y}=GY$ with $G=I_n-H$. Consequently $G^T=G$, $G^2=I_n-2H+H^2=G$, and the residual sum of squares isBy fitted-residual orthogonality, $GH=0$. Using $\mathrm{Cov}(Y)={\sigma }^2{I}_n$, the cross-covariance matrix isBoth vectors are jointly normal, so they are also independent by independence of uncorrelated jointly normal variables. The zero covariance calculation itself only needs the common error variance and lack of error correlations.

For the first wind fit, let $v_i$ be wind velocity and $Y_i$ electrical output. The simple linear regression is $Y_i=\alpha +\beta v_i+{\epsilon }_i$, with independent ${\epsilon }_i\sim N(0,{\sigma }^2)$. It has two fitted mean parameters and $25-2=23$ residual degrees of freedom. The missing analysis of variance entries are therefore velocity degrees of freedom $1$, velocity mean square $8.9296$, and residual degrees of freedom $23$. The residual mean square is $1.2816/23\simeq 0.05572$, and $8.9296/(1.2816/23)\simeq 160.25$, consistent with the printed value after rounding.

Because the model includes a regression intercept, the coefficient of determination is the explained sum of squares divided by the total centered sum of squares:A large coefficient of determination does not rule out a wrong mean function. In the PDF's first residual-versus-fitted plot, residuals are negative at both ends and positive in the middle. This curved pattern agrees with the visibly flattening output-versus-velocity relationship and motivates a polynomial regression with a quadratic term.

The second wind model is $Y_i=\alpha +{\beta }_1{v}_i+{\beta }_2{v}_i^2+{\epsilon }_i$, again with independent common-variance normal errors. Its fitted mean is $-1.155898+0.722936v_i-0.038121v_i^2$. The line marked (A) is a two-sided Student t-test of $H_0:{\beta }_2=0$ against $H_1:{\beta }_2\ne 0$, conditional on retaining the intercept and linear term. Under the null hypothesis,Its two-sided p-value is $6.59\times 10^{-8}$. This is far below $0.05$, so reject a purely linear mean in favour of the quadratic fit. Equivalently, the extra-term nested-model F-test has $F=t^2\simeq 63.15$ and null law $F_{1,22}$.

The third fit is a reciprocal-predictor regression, $Y_i=a+b/v_i+{\epsilon }_i$, with estimated mean $2.9789-6.9345/v_i$. It has two mean parameters rather than the quadratic model's three. Its residual standard error is smaller, $0.09417$ versus $0.1227$, and its coefficient of determination is larger, $0.9800$ versus $0.9676$. The corresponding residual sums of squares are approximately $23(0.09417{)}^2=0.20396$ and $22(0.1227{)}^2=0.33122$. The reciprocal fit is preferable on both these fit measures and parsimony. The two models are not nested, so an ordinary extra-term F-test between them is inappropriate. On the common normal-error likelihood, the difference ${\mathrm{AIC}}_3-{\mathrm{AIC}}_2=25\log (0.20396/0.33122)-2\simeq -14.12$ also favours the third model.

Neither lower residual sum of squares nor a higher coefficient of determination establishes adequate assumptions. The quadratic residual-versus-fitted plot removes the original pronounced curvature; the reciprocal plot also has no comparably obvious mean trend. The low-output residuals appear somewhat more spread out, so check scale-location plots and residuals against velocity for heteroscedasticity. Q-Q plots assess normality; residuals against observation order assess serial dependence; regression leverage and Cook's distance identify influential observations. Further cross-validation, replicate observations at comparable velocities, and prediction errors would help choose between their extrapolation behaviours. Both fitted shapes should be judged principally over the observed positive-velocity range.

The three-component finite Gaussian mixture with a common variance has the largest displayed maximized log-likelihood, but it also has more fitted parameters. For $k$ components there are $k-1$ free weights, $k$ means and one common variance, making $d=2k$ residual-distribution parameters. Applying the nominal Akaike information criterion and Bayesian information criterion to the three residual fits givesBoth criteria favour the two-component model. It improves the likelihood substantially over one normal component; the third component gains only $1.5550$ in log-likelihood at a cost of two further parameters. The Akaike information criterion difference between two and three components is small, about $0.89$, so that criterion alone gives only a slight preference; the Bayesian information criterion preference for two is stronger. Their nearly indistinguishable overlaid curves also support retaining the simpler two-component fit.

These calculations are model-selection aids, not an exact test. Ordinary chi-squared calibration of a likelihood-ratio test for the number of mixture components is invalid in general: under a smaller-component null, some weights lie on the boundary and extra-component parameters are unidentified. This is nonregular mixture model selection. A parametric bootstrap or predictive cross-validation is preferable for a formal comparison, and should account for the mean-adjustment stage. The table uses the provided residual likelihoods and their nominal parameter counts; first-stage regression uncertainty and possible local EM maxima remain qualifications. On the supplied evidence, select two components without claiming that two distinct biological groups have been established.

The residual smooth rises from negative values, arches above zero, then falls at high fitted values. This suggests a nonlinear altitude effect on the log mean. Since the fitted log mean in the linear model increases with altitude, the horizontal fitted-value axis is also an increasing rescaling of altitude. A generalized additive model with Quasi-Poisson regression is therefore a reasonable candidate:The centering constraint identifies the intercept. Here bs="cr" uses a penalized cubic regression spline. The plotted count residuals are deviance residuals, with the Q-Q panel using leverage-standardized versions. The quantile-quantile plot also shows tail discrepancies, but normality of count residuals is not a distributional assumption of this model: its main requirements are an adequate mean, variance relation, and independence. Investigate the labelled islands and possible excess variation as well.

The estimated smooth in the second plot increases fairly rapidly at low altitude, flattens around altitude 20 to 25, and is nearly level at the upper end. The pointwise bands widen where there are fewer observations. A penalized spline can describe a rise followed by a plateau without forcing a global parabola. A quadratic log mean has slope ${\beta }_1+2{\beta }_{2}a$, so negative curvature eventually forces a decline and its curvature is constant everywhere. A cubic regression spline allows different curvature over different ranges, with a penalty controlling unnecessary wiggles and natural linear tails outside its boundary knots. The displayed smooth supports this greater flexibility, but it is not by itself a formal rejection of every quadratic model. Select complexity and compare predictive performance using cross-validation, then reassess residuals and the Pearson dispersion estimator.

At a target $x$, approximate the unknown mean locally by $a+b(X_i-x)$ and solve the local linear regression weighted least-squares problemwhere $K$ is a nonnegative regression kernel and $h>0$ a smoothing bandwidth. A common factor $1/h$ in all weights makes no difference. Estimate $f(x)$ by the fitted intercept ${\hat{a}}_x$, and optionally $f^{\prime }(x)$ by ${\hat{b}}_x$. Repeat at each target rather than fitting a single global straight line.

For an explicit expression put $S_k={\sum }_i{w}_i(X_i-x{)}^k$ and $T_k={\sum }_i{w}_i(X_i-x{)}^k{Y}_i$. The normal equations yieldThe denominator is positive provided there are at least two distinct positively weighted predictor values. The weights sum to one and their first centered moment is zero, so local linear regression exactly reproduces affine functions and reduces boundary bias compared with a local constant fit. Smaller $h$ uses more local detail and increases variance; larger $h$ smooths more and can increase bias. Choose the bandwidth by cross-validation or another justified criterion. A nearest-neighbour span provides an adaptive bandwidth, as in the local fits used later.

Both fits use the Gaussian generalized additive modelwhere $e_i$ is experience and $d_i$ education. Centering makes the additive decomposition identifiable. No experience-education interaction term is included.

The first fit uses local linear regression for each term. span=1 includes all observations in each nearest-neighbour neighbourhood, but distance weights and a moving target still vary: it does not make the fit exactly a global straight line. The second uses cubic smoothing splines, with separate tuning values spar=0.8 and spar=1.2. Increasing spar increases the curvature penalty and smooths more; spar is a monotone tuning parametrization, not the numerical $\lambda$ in part (b). The relation depends on the predictor scaling and design.

The plotted experience effect is increasing in both fits. The local fit gives a broadly smooth, nearly linear rise; the spline fit follows more of the steep low-experience rise and subsequent flattening, especially near the lower boundary. Education has a weaker increasing effect. The local estimate shows some curvature, while the spline fit at spar=1.2 is closer to a straight line. The clearest difference is the amount of smoothing and the experience curvature, rather than opposite effect directions. These are centered partial-effect plots with partial residuals, not scatterplots of wage against each predictor alone.

A smaller local span would allow the first fit to capture more experience curvature. A moderately smaller education spar could reveal nonlinear detail suppressed in the second fit; its small but significant nonparametric component in part (e) supports checking that possibility. Do not tune solely to follow every point, especially at sparse boundaries. Compare candidate spans or separate spline penalties using cross-validation or generalized cross-validation; increase smoothing if flexible fits introduce unsupported wiggles. See the documented monotone spar–penalty convention at stat.ethz.ch/R-manual/R-patched/library/stats/html/smooth.spline.html.

Choose knots independently of the eventual validation responses, and let $b(x)=(b_1(x),\dots ,b_K(x){)}^T$ be a natural cubic spline basis as in part (a). A regression spline writes $m(x)=b(x{)}^T\beta$. Define the design matrix $B_{ij}=b_j(X_i)$. Under the stated normal linear model, maximum likelihood estimation for $\beta$ is the same as ordinary least squares:Differentiation gives the normal equations $B^{T}B\hat{\beta }=B^{T}Y$. If the design has full column rank,If it does not, a Moore-Penrose inverse gives the fitted values, but not every coefficient or off-sample prediction is identifiable. Merely choosing distinct knots does not guarantee full rank for an arbitrary set of observed predictor values.

More knots allow greater curvature and reduce approximation bias at the cost of larger estimation variance. Too many or poorly located knots can cause unstable fits. Equally spaced knots are simple; empirical predictor quantiles put more flexibility where observations are dense; subject knowledge can concentrate knots near likely changes. Both the number and positions can be compared using cross-validation or an appropriate penalized model-selection criterion. Adaptive knot selection must be included within validation, since treating a data-selected basis as prespecified understates its flexibility. Boundary knots should cover the region where inference is intended, and natural linear tails do not by themselves justify far-out extrapolation.

When $p>n$, the pooled within-class covariance has rank at most $n-L<p$, so it is singular and ordinary LDA is not defined. For every $\lambda >0$,is positive definite because $v^T{\hat{\Sigma }}_{\lambda }v\ge \lambda \Vert v{\Vert }^2>0$. Replacing the covariance by this matrix gives well-defined regularized LDA. Choose $\lambda$ for predictive performance by cross-validation.

Let $N_k(x)$ index the $k$ closest training covariates to $x$. The K-nearest neighbors algorithm estimatesand predicts one when this average is at least $1/2$.

Small $k$ gives low smoothing bias but high sampling variance and a jagged decision boundary. Larger $k$ averages more labels, reducing variance and producing a smoother boundary, but it mixes increasingly distant covariates and raises bias. The optimal balance depends on sample size, dimension, and smoothness of ${\eta }_1$, and is commonly selected by cross-validation.

A binary logistic regression setsand classifies by the sign of ${\gamma }_0+x^T\beta$. Its unpenalized maximum-likelihood estimator minimizes the empirical logistic lossThe plotted data are complete separation data: there is a vector $v$ with every signed margin $Y_i{X}_i^{\ast T}v>0$. For every finite $c>0$, increasing $c$ strictly decreases each term of $L(cv)$, and $L(cv)\to 0$ as $c\to \infty$. No finite parameter attains zero, so the unpenalized optimization has no solution.

Adding an $L^2$ penalty $\lambda \Vert \gamma {\Vert }_2^2$ with $\lambda >0$, constraining $\Vert \gamma {\Vert }_2$, or using a finite stopping rule makes the problem attain a finite approximate solution. The penalized option is preferable because cross-validation can select the strength of regularization.

A reciprocal-predictor linear regression uses $\mathbb{E}(Y\mid v)=\alpha +\beta /v$ for nonzero predictor $v$. It remains linear in its coefficients. Its flattening shape can suit a saturating response over a positive predictor range; residual checks and cross-validation must assess adequacy, and extrapolation toward zero is hazardous.

The residual sum of squares measures how far the observed responses lie from their fitted values. If $r_i=Y_i-{\hat{Y}}_i$ is the $i$th regression residual, thenIt is the squared Euclidean norm of the regression residual vector. Squaring prevents positive and negative deviations from cancelling and gives large deviations more weight. In the constructed five-point example in linear regression, the regression residuals are $(1/5,1/5,-4/5,1/5,1/5)$, so the residual sum of squares is $4/5$.

Ordinary least squares minimizes the residual sum of squares over the chosen linear regression family; ridge regression minimizes the residual sum of squares plus a penalty. Consequently the ridge regression fit can have a larger residual sum of squares than the ordinary least squares fit on exactly the same design matrix. For the one-predictor construction in ridge regression, the residual sum of squares rises from $4/5$ to $33/10$ when the penalty changes from $0$ to $10$; that increase is the cost of shrinking the fitted slope.

Dividing the residual sum of squares by $n$ gives the training error for squared-error loss, not an estimate based on fresh responses. Its units are squared response units, and it adds across observations, so raw totals need not be comparable across different response scales or sample sizes. Comparing fits to the same responses is useful, but choosing a ridge regression penalty by minimum training residual sum of squares favors the unpenalized fit. Assess prediction error using a validation set or cross-validation instead.^[ref]

Statistical learning constructs predictive rules from data and studies their prediction error, generalization, and computational fitting. In supervised learning, training data contain inputs $X$ and a response variable $Y$: regression analysis predicts a numerical response, while classification in statistical learning predicts a label. Unsupervised learning instead seeks structure without supplied responses, as in cluster analysis and principal component analysis. These tasks connect learning to probability and mathematical optimization. ^[ref]

For a fitted rule $\hat{f}$ and loss function $L$, the predictive goal iswhere $(X^{\ast },Y^{\ast })$ is a fresh observation, independent of the training data ${\mathcal{D}}_n$, from the intended target probability distribution. This prediction error fixes the fitted rule and averages over future observations; averaging again over training data gives the learning procedure's average performance. Its practical beauty is that a rule compresses past observations into useful future predictions. Its central difficulty is that excellent training error need not imply good generalization: cross-validation supports model selection, and a reserved test set assesses the resulting choice.

A test set is held out from both parameter fitting and model selection and is used to estimate the final selected procedure's prediction error. A validation set influences the choice; a test set assesses that choice after it is fixed. The distinction concerns how observations are used, not their file names.

For a fixed rule and an independent test set of size $m$ from the target probability distribution,This test error is an estimate, with sampling uncertainty; it is not a guarantee for each future prediction. Dataset shift changes what performance the estimate represents.

In the overfitting example, all degree choices are fixed in advance, cross-validation uses only training data, and the selected degree is locked before evaluating 400 separate points. Displaying the other curves afterward explains their behavior; it does not authorize selecting by those scores. Repeatedly inspecting a test set and changing the model makes it part of model selection, introducing data leakage and potentially optimistic assessment. ^[ref]

Underfitting occurs when a prediction rule fails to capture useful structure in the target relation. A straight-line polynomial regression fitted to a genuinely quadratic noiseless target illustrates a representational limitation. Poor prediction alone does not identify the cause: compare candidate rules using cross-validation or a validation set, while keeping the final test set reserved for assessment.

Variable selection chooses which predictor variables enter a statistical model. The Lasso produces exact zero slopes along its Lasso regularization path; ridge regression generally shrinks without excluding variables. Selection for prediction should use predictive cross-validation or another stated criterion, rather than equating a coefficient p-value with practical usefulness.