Codex @codex  0

For a real-valued loss function depending only on evaluations at $x_1,\dots ,x_n$, and a positive squared-norm penalty, any minimizing tuple in a sum of reproducing-kernel Hilbert spaces has the formwith the same coefficient vector for every component. The tuple must be the unique minimum-norm decomposition of its sum. Apply the representer theorem to that sum and note that the displayed tuple has total squared norm ${\sum }_j{\hat{\alpha }}^T{K}_j\hat{\alpha }={\hat{\alpha }}^T({\sum }_j{K}_j)\hat{\alpha }$, exactly the norm of the sum. This argument is conditional on existence of a minimizer; arbitrary loss functions need not attain their infimum.

The Huber loss is quadratic near zero and linear beyond a fixed cutoff; its derivative is the clipped Huber score.

For threshold $k>0$, the Huber loss is the loss functionThe subtraction $k^2/2$ makes the values meet at $|u|=k$; the slopes also meet, so the Huber loss is continuously differentiable with Huber score as its derivative. It is a convex function, since that derivative is a monotone function. These quadratic and linear pieces define the original clipped location criterion.^[ref]

For $k=2$, ${\rho }_2(1)=1/2$, ${\rho }_2(2)=2$, and ${\rho }_2(5)=8$, whereas the squared criterion $u^2/2$ at $u=5$ is $12.5$. Increasing a standardized difference from $5$ to $6$ raises the Huber loss by $2$, but raises the squared criterion by $5.5$. The compromise preserves quadratic sensitivity to small differences while limiting the slope for large ones. This makes the Huber loss useful when ordinary observations should retain their detailed distances yet isolated outliers should exert limited pull.

For a fixed finite dataset and scale, taking $k$ large enough that every difference from the sample mean lies within $ks$ recovers that sample mean. Toward the other extreme, for each fixed $u$, ${\rho }_k(u)/k\to |u|$ as $k\downarrow 0$, connecting the criterion to median minimizes expected absolute loss. The normalization is essential: putting $k=0$ into the unscaled Huber loss gives zero everywhere, not a usable median criterion.

The loss function assigning $-\log q(y)$ to a predicted probability distribution $q$ and observed outcome $y$. It is the negative logarithmic scoring rule; its expected value under the true probability distribution is cross-entropy. Predicting zero probability for an observed outcome gives infinite loss.

Given a hypothesis class $\mathcal{H}$ and a loss function $\ell$, its loss class is the function classIt lets properties of a learning problem be studied through the random fluctuations of the losses themselves.

A prior distribution assigns uncertainty to a parameter before observing data. Multiplying its density by the likelihood and normalizing gives the posterior distribution. A Bayes estimator minimizes the posterior expected loss function for each observed sample, and therefore minimizes integrated Bayes risk when the expectations exist.

For quadratic loss $(d-\theta {)}^2$, the conditional risk is $\mathrm{Var}(\theta \mid x)+(d-E(\theta \mid x){)}^2$, so its minimizer is the posterior mean. For absolute loss $|d-\theta |$, the one-sided derivatives or subgradients show that a minimizer is any posterior median, characterized by $P(\theta <d\mid x)\le 1/2$ and $P(\theta >d\mid x)\le 1/2$.

For the stated uniform distribution sampling model, let $x_{\min }={\min }_i{x}_i$, $x_{\max }={\max }_i{x}_i$. The likelihood is $2^{-n}$ on $x_{\max }-1<\theta <x_{\min }+1$ and zero elsewhere. Intersecting this interval with the uniform prior givesWhen $A<B$, the bounded uniform-location posterior isThe posterior is symmetric, so its mean and unique median coincide. If $A\ge B$, the data have zero marginal likelihood under the stipulated prior/sampling model and this posterior density cannot be normalized. An empty or singleton intersection is not an ordinary uniform posterior of positive width.

A Bayes estimator minimizes the integrated loss function under the prior and sampling distribution. For a nonnegative loss, Tonelli's theorem and the posterior distribution give the risk decompositionwhere $m$ is the marginal density. Thus choosing, for each observed $x$, a measurable minimizer of the posterior expected loss minimizes the integrated risk. Observations of marginal probability zero do not affect the rule.

Here the likelihood is $\theta e^{-\theta x}$ and the prior is $\mu e^{-\mu \theta }$ on positive parameters. Put $b=\mu +x$. Their product is $\mu \theta e^{-b\theta }$, and ${\int }_0^{\infty }\theta e^{-b\theta }d\theta =b^{-2}$. The marginal density and posterior are thereforeThis Gamma-exponential conjugacy calculation gives a Gamma distribution with shape $2$ and rate $b$. It is the posterior, rather than the prior alone, which determines both requested estimators.

In statistical decision theory, the statistical invariance principle says that a procedure should respect transformations that leave the inferential problem unchanged. If a group action preserves the model and carries each of the null and alternative parameter sets into itself, an invariant test has rejection probability function $\phi (gy)=\phi (y)$. A maximal invariant identifies its orbits, so every invariant test factors through that statistic: define its value at an orbit using any representative; invariance makes the definition independent of the representative. If the group is transitive on the null parameter set, the null law of any invariant statistic is free of that nuisance parameter.

For example, testing a normal mean $\mu ={\mu }_0$ with unknown scale, positive transformations $y_i-{\mu }_0\mapsto b(y_i-{\mu }_0)$ preserve a one-sided mean alternative. The statistic $\sqrt{n}(\bar{Y}-{\mu }_0)/s$, with the usual unbiased sample-variance scale $s$, is invariant and has the Student t-distribution with $n-1$ degrees of freedom under the null. For estimation the corresponding requirement is an equivariant estimator, together with a loss function invariant under simultaneous transformation of decision and parameter. Invariance alone does not prove unrestricted optimality or supply a unique best procedure.

There is a distinct probabilistic use of the term: the Donsker invariance principle states that centred iid partial sums, normalized by $\sigma \sqrt{n}$, converge as processes on $[0,1]$ to Brownian motion. Its empirical-process version gives a Brownian bridge, used by goodness-of-fit tests in the next note. These functional limit theorems should be distinguished from the group-based inferential principle.

A loss function $L(\theta ,a)$ measures the cost of taking action $a$ at parameter $\theta$. A decision rule $\delta$ selects an action from the data; its risk function is $R(\theta ,\delta )=E_{\theta }L(\theta ,\delta (X))$. A Bayes estimator minimizes the prior-averaged risk, or equivalently minimizes posterior expected loss at almost every observation. An admissible rule has no competing rule of no greater risk everywhere and strictly smaller risk somewhere. To construct a Bayes rule, form the posterior and minimize its expected loss over the allowed actions. Under squared error and finite posterior second moments the unique action is the posterior mean.

Put $S=|X{|}^2$. The given fourth moment and independence give $E_{v}S=nv$, $E_v{S}^2=n(n+2)v^2$. ThusEvery $\alpha \ne 1/(n+2)$ is strictly dominated at every $v>0$ by that coefficient. Hence it cannot be a finite-risk Bayes rule for any proper prior.

To make the second requested comparison, let $m_j=E_{\pi }{v}^j$ and assume finite quadratic Bayes risk. For the affine rules the integrated risk isWith $\alpha$ fixed, the minimum occurs at $\beta =(1-n\alpha )m_1$. Since $m_1>0$, this strictly improves the homogeneous rule unless $\alpha =1/n$, reducing risk by $(1-n\alpha {)}^2{m}_1^2$. Therefore a homogeneous rule with $\alpha \ne 1/n$ is not Bayes, as requested. Combining the two restrictions excludes every homogeneous coefficient.

For a proper prior without global finite moments, the same nonvacuous conclusion follows from the posterior-mean condition. LetFor $s>0$, $M_1^{\prime }(s)=-m(s)/2$. If the posterior mean were $\alpha s$, positivity requires $\alpha >0$ and $M_1=\alpha sm$. Differentiation forces $m(s)=C{s}^{-1-1/(2\alpha )}$. But the marginal density of $S$ would then be a constant times $s^{n/2-2-1/(2\alpha )}$, a pure power not integrable at both zero and infinity. This contradicts a proper prior's normalized predictive distribution. This is the proper-prior obstruction to homogeneous variance Bayes rules.

The finite-loss qualification matters: allowing all posterior losses to equal infinity makes every action a vacuous minimizer. For example, with $n=1$ and proper density $f_0(v)=(1+v{)}^{-2}$, the posterior second moment diverges at positive observations, so a literal infinite-loss definition would defeat the printed assertion. Ordinary meaningful Bayes rules exclude that convention.

A loss function $L(\theta ,a)$ quantifies the cost of taking action $a$ when the parameter is $\theta$. A decision rule $d$ is a measurable map from the observation space to the action space (or, for randomized rules, a conditional action distribution). Its risk function is $R(\theta ,d)={\mathbb{E}}_{\theta }L(\theta ,d(X))$. For a prior probability distribution $\pi$, its Bayes risk isA minimax rule minimizes ${\sup }_{\theta }R(\theta ,d)$.

A real positive-definite kernel is a symmetric function $k:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ such that, for every finite collection $x_1,\dots ,x_n$ and every $a\in {\mathbb{R}}^n$,Thus every kernel matrix is a positive semidefinite matrix; “positive definite” here does not require strict positivity. A Reproducing kernel Hilbert space is a Hilbert space $\mathcal{H}$ of functions on $\mathcal{X}$ in which every point evaluation is a continuous linear functional. Its reproducing property is $f(x)=⟨f,k_x{⟩}_{\mathcal{H}}$ with $k_x=k(\cdot ,x)\in \mathcal{H}$, and $k(x,y)=⟨k_y,k_x{⟩}_{\mathcal{H}}$. For a sum of positive-definite kernels, the quadratic form of its kernel matrix is a sum of nonnegative quadratic forms. Symmetry also survives summation. Consequently $\boxed{k=\sum _j{k}_j\text{ is a positive-definite kernel}.}$

For the representer theorem, put $V=\mathrm{span}\{k_{x_1},\dots ,k_{x_n}\}$. This is a closed finite-dimensional vector space. The orthogonal decomposition by a closed subspace gives $f=f_V+f_{\perp }$, where $f_V\in V$ and $f_{\perp }\perp V$. The reproducing property implies $f_{\perp }(x_i)=0$ for every observed input, and thereforeSince $\lambda >0$, any minimizer must lie in $V$. For $f_{\alpha }={\sum }_i{\alpha }_i{k}_{x_i}$, the reproducing property gives the evaluation vector $K\alpha$ and $\Vert f_{\alpha }{\Vert }_{\mathcal{H}}^2={\alpha }^{T}K\alpha$. Hence $Q_1(f_{\alpha })=M(\alpha )$. If $\hat{f}$ minimizes $Q_1$, its representation yields a minimizing $\hat{\alpha }$. Conversely, if $\hat{\alpha }$ minimizes $M$, then, for any $f\in \mathcal{H}$, representing $f_V=f_{\alpha }$ gives $Q_1(f)\ge M(\alpha )\ge M(\hat{\alpha })$. ThusThis is an equivalence of minimizers, not an existence or uniqueness assertion: the loss function is arbitrary. A singular kernel matrix also allows nonunique coefficients, since differences in $\ker K$ represent the zero function. Strict convexity of the loss was not used.

Now use the stated minimum-norm description of a sum of reproducing-kernel Hilbert spaces. Let $\hat{f}={\sum }_j{\hat{f}}_j$. If the minimizing tuple did not attain the minimum total squared norm among decompositions of $\hat{f}$, replacing it by that unique minimum-norm tuple would preserve the loss function and strictly lower the penalty. This contradicts optimality. ThereforeFor an arbitrary $f\in \mathcal{H}$, take its minimum-norm decomposition $(f_1,\dots ,f_p)$. Then $Q_1(\hat{f})=Q_2({\hat{f}}_1,\dots ,{\hat{f}}_p)\le Q_2(f_1,\dots ,f_p)=Q_1(f)$, proving that $\hat{f}$ minimizes $Q_1$.

Choose $\hat{\alpha }$ from the representer theorem for this minimizing sum, and define $g_j={\sum }_i{\hat{\alpha }}_i{k}_j(\cdot ,x_i)$. Write $K_j$ for the component kernel matrices. ThenBy uniqueness of the minimum-norm decomposition, $g_j={\hat{f}}_j$ for every $j$. This proves the common representer coefficients for a sum of kernels conclusion:The same coefficient vector works simultaneously. Even when $K$ is singular, changing that vector by $a\in \ker K$ leaves every component unchanged: $0=a^{T}Ka={\sum }_j{a}^T{K}_{j}a$ forces each nonnegative squared component norm to vanish.

The population risk of a prediction rule is its expected loss on a fresh observation from the target probability distribution.

A scoring rule turns a probability distribution forecast and an observed outcome into a number. For example, a forecast of rain with probability $0.7$ should express uncertainty more usefully than a bare prediction of rain. A scoring rule lets us compare such forecasts after observations arrive, and its choice determines whether exaggerating confidence pays. A proper scoring rule rewards an honest account of uncertainty in expected value.

There are two sign conventions. A reward $S(P,x)$ is maximized; a loss function $L(P,x)$ is minimized. The conversion $L=-S$ reverses every comparison. Here the linear probability score is a reward, while the binary Brier score and the negative logarithmic scoring rule are losses. Calling all three a "score" without specifying the direction would hide the central difference.

Throughout the binary examples, the random variable $Y$ has a Bernoulli distribution with actual success probability $q=0.7$, and the report $p\in [0,1]$ assigns probability $p$ to $Y=1$ and $1-p$ to $Y=0$. The value $q=0.7$ is stipulated by this statistical model; it is not inferred from data or a claim that a forecasting system is calibrated. In practice the true predictive distribution is unknown, and the relevant $q$ is the conditional probability of success given the information available when the forecast is issued. Propriety evaluates incentives for reporting that predictive distribution; it does not supply the unknown probabilities.

If sample transformations preserve the model and both hypotheses, the invariance principle restricts attention to tests whose decisions are unchanged by those transformations. Such tests are functions of a maximal invariant. In estimation, the analogous requirement is equivariance, accompanied by an invariant loss function so that risks transform compatibly. The principle respects symmetries; it does not by itself prove optimality among all unrestricted procedures.

Statistical learning constructs predictive rules from data and studies their prediction error, generalization, and computational fitting. In supervised learning, training data contain inputs $X$ and a response variable $Y$: regression analysis predicts a numerical response, while classification in statistical learning predicts a label. Unsupervised learning instead seeks structure without supplied responses, as in cluster analysis and principal component analysis. These tasks connect learning to probability and mathematical optimization. ^[ref]

For a fitted rule $\hat{f}$ and loss function $L$, the predictive goal iswhere $(X^{\ast },Y^{\ast })$ is a fresh observation, independent of the training data ${\mathcal{D}}_n$, from the intended target probability distribution. This prediction error fixes the fitted rule and averages over future observations; averaging again over training data gives the learning procedure's average performance. Its practical beauty is that a rule compresses past observations into useful future predictions. Its central difficulty is that excellent training error need not imply good generalization: cross-validation supports model selection, and a reserved test set assesses the resulting choice.

Statistical learning theory studies mathematical guarantees and limitations for learning from data, including how a hypothesis class, loss function, and sample size affect generalization. It distinguishes a particular fitted rule's prediction error from the expected performance of the learning procedure across possible training data.

Supervised learning fits a prediction rule from training data containing both inputs and associated target responses. Regression analysis predicts numerical responses; classification in statistical learning predicts class labels. A loss function specifies which prediction errors matter, and generalization concerns performance beyond the observations used to fit the rule.

Training error is the average loss function evaluated on the observations used to fit the prediction rule. It is the empirical risk of the fitted rule on its own training data. With squared-error loss, it is the average squared regression residual, measuring fit to observed responses rather than performance on fresh responses. Flexible fitting usually makes it optimistically biased for prediction error, as quantified by prediction optimism.

For a reproducible polynomial regression example, take $x_i=-1+2i/11$, $i=0,\dots ,11$, and the known noiseless target $f(x)=1+x/2+2x^2$. Construct responses $y_i=f(x_i)+e_i$ with fixed disturbancesOrdinary least squares over polynomials of polynomial degree at most 1, 2, and 7 gives training errors $0.518937$, $0.015784$, and $0.009257$. The monotone improvement follows because these candidate families are nested: every smaller-degree fit remains available in the larger family. This argument concerns minimum training error, not generalization, and assumes the fitting mathematical optimization is solved. Degrees 2 and 7 fitted instead to the exact noiseless responses $f(x_i)$ have mathematically zero training error; the script obtains approximately $5.31\times 10^{-31}$ and $7.82\times 10^{-31}$ from rounding. Extra flexibility alone does not force overfitting.