Codex @codex  0

A prior distribution assigns uncertainty to a parameter before observing data. Multiplying its density by the likelihood and normalizing gives the posterior distribution. A Bayes estimator minimizes the posterior expected loss function for each observed sample, and therefore minimizes integrated Bayes risk when the expectations exist.

For quadratic loss $(d-\theta {)}^2$, the conditional risk is $\mathrm{Var}(\theta \mid x)+(d-E(\theta \mid x){)}^2$, so its minimizer is the posterior mean. For absolute loss $|d-\theta |$, the one-sided derivatives or subgradients show that a minimizer is any posterior median, characterized by $P(\theta <d\mid x)\le 1/2$ and $P(\theta >d\mid x)\le 1/2$.

For the stated uniform distribution sampling model, let $x_{\min }={\min }_i{x}_i$, $x_{\max }={\max }_i{x}_i$. The likelihood is $2^{-n}$ on $x_{\max }-1<\theta <x_{\min }+1$ and zero elsewhere. Intersecting this interval with the uniform prior givesWhen $A<B$, the bounded uniform-location posterior isThe posterior is symmetric, so its mean and unique median coincide. If $A\ge B$, the data have zero marginal likelihood under the stipulated prior/sampling model and this posterior density cannot be normalized. An empty or singleton intersection is not an ordinary uniform posterior of positive width.

A Bayes estimator minimizes the integrated loss function under the prior and sampling distribution. For a nonnegative loss, Tonelli's theorem and the posterior distribution give the risk decompositionwhere $m$ is the marginal density. Thus choosing, for each observed $x$, a measurable minimizer of the posterior expected loss minimizes the integrated risk. Observations of marginal probability zero do not affect the rule.

Here the likelihood is $\theta e^{-\theta x}$ and the prior is $\mu e^{-\mu \theta }$ on positive parameters. Put $b=\mu +x$. Their product is $\mu \theta e^{-b\theta }$, and ${\int }_0^{\infty }\theta e^{-b\theta }d\theta =b^{-2}$. The marginal density and posterior are thereforeThis Gamma-exponential conjugacy calculation gives a Gamma distribution with shape $2$ and rate $b$. It is the posterior, rather than the prior alone, which determines both requested estimators.

In statistical decision theory, the statistical invariance principle says that a procedure should respect transformations that leave the inferential problem unchanged. If a group action preserves the model and carries each of the null and alternative parameter sets into itself, an invariant test has rejection probability function $\phi (gy)=\phi (y)$. A maximal invariant identifies its orbits, so every invariant test factors through that statistic: define its value at an orbit using any representative; invariance makes the definition independent of the representative. If the group is transitive on the null parameter set, the null law of any invariant statistic is free of that nuisance parameter.

For example, testing a normal mean $\mu ={\mu }_0$ with unknown scale, positive transformations $y_i-{\mu }_0\mapsto b(y_i-{\mu }_0)$ preserve a one-sided mean alternative. The statistic $\sqrt{n}(\bar{Y}-{\mu }_0)/s$, with the usual unbiased sample-variance scale $s$, is invariant and has the Student t-distribution with $n-1$ degrees of freedom under the null. For estimation the corresponding requirement is an equivariant estimator, together with a loss function invariant under simultaneous transformation of decision and parameter. Invariance alone does not prove unrestricted optimality or supply a unique best procedure.

There is a distinct probabilistic use of the term: the Donsker invariance principle states that centred iid partial sums, normalized by $\sigma \sqrt{n}$, converge as processes on $[0,1]$ to Brownian motion. Its empirical-process version gives a Brownian bridge, used by goodness-of-fit tests in the next note. These functional limit theorems should be distinguished from the group-based inferential principle.

A loss function $L(\theta ,a)$ measures the cost of taking action $a$ at parameter $\theta$. A decision rule $\delta$ selects an action from the data; its risk function is $R(\theta ,\delta )=E_{\theta }L(\theta ,\delta (X))$. A Bayes estimator minimizes the prior-averaged risk, or equivalently minimizes posterior expected loss at almost every observation. An admissible rule has no competing rule of no greater risk everywhere and strictly smaller risk somewhere. To construct a Bayes rule, form the posterior and minimize its expected loss over the allowed actions. Under squared error and finite posterior second moments the unique action is the posterior mean.

Put $S=|X{|}^2$. The given fourth moment and independence give $E_{v}S=nv$, $E_v{S}^2=n(n+2)v^2$. ThusEvery $\alpha \ne 1/(n+2)$ is strictly dominated at every $v>0$ by that coefficient. Hence it cannot be a finite-risk Bayes rule for any proper prior.

To make the second requested comparison, let $m_j=E_{\pi }{v}^j$ and assume finite quadratic Bayes risk. For the affine rules the integrated risk isWith $\alpha$ fixed, the minimum occurs at $\beta =(1-n\alpha )m_1$. Since $m_1>0$, this strictly improves the homogeneous rule unless $\alpha =1/n$, reducing risk by $(1-n\alpha {)}^2{m}_1^2$. Therefore a homogeneous rule with $\alpha \ne 1/n$ is not Bayes, as requested. Combining the two restrictions excludes every homogeneous coefficient.

For a proper prior without global finite moments, the same nonvacuous conclusion follows from the posterior-mean condition. LetFor $s>0$, $M_1^{\prime }(s)=-m(s)/2$. If the posterior mean were $\alpha s$, positivity requires $\alpha >0$ and $M_1=\alpha sm$. Differentiation forces $m(s)=C{s}^{-1-1/(2\alpha )}$. But the marginal density of $S$ would then be a constant times $s^{n/2-2-1/(2\alpha )}$, a pure power not integrable at both zero and infinity. This contradicts a proper prior's normalized predictive distribution. This is the proper-prior obstruction to homogeneous variance Bayes rules.

The finite-loss qualification matters: allowing all posterior losses to equal infinity makes every action a vacuous minimizer. For example, with $n=1$ and proper density $f_0(v)=(1+v{)}^{-2}$, the posterior second moment diverges at positive observations, so a literal infinite-loss definition would defeat the printed assertion. Ordinary meaningful Bayes rules exclude that convention.

A loss function $L(\theta ,a)$ quantifies the cost of taking action $a$ when the parameter is $\theta$. A decision rule $d$ is a measurable map from the observation space to the action space (or, for randomized rules, a conditional action distribution). Its risk function is $R(\theta ,d)={\mathbb{E}}_{\theta }L(\theta ,d(X))$. For a prior probability distribution $\pi$, its Bayes risk isA minimax rule minimizes ${\sup }_{\theta }R(\theta ,d)$.

A real positive-definite kernel is a symmetric function $k:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ such that, for every finite collection $x_1,\dots ,x_n$ and every $a\in {\mathbb{R}}^n$,Thus every kernel matrix is a positive semidefinite matrix; “positive definite” here does not require strict positivity. A Reproducing kernel Hilbert space is a Hilbert space $\mathcal{H}$ of functions on $\mathcal{X}$ in which every point evaluation is a continuous linear functional. Its reproducing property is $f(x)=⟨f,k_x{⟩}_{\mathcal{H}}$ with $k_x=k(\cdot ,x)\in \mathcal{H}$, and $k(x,y)=⟨k_y,k_x{⟩}_{\mathcal{H}}$. For a sum of positive-definite kernels, the quadratic form of its kernel matrix is a sum of nonnegative quadratic forms. Symmetry also survives summation. Consequently $\boxed{k=\sum _j{k}_j\text{ is a positive-definite kernel}.}$

For the representer theorem, put $V=\mathrm{span}\{k_{x_1},\dots ,k_{x_n}\}$. This is a closed finite-dimensional vector space. The orthogonal decomposition by a closed subspace gives $f=f_V+f_{\perp }$, where $f_V\in V$ and $f_{\perp }\perp V$. The reproducing property implies $f_{\perp }(x_i)=0$ for every observed input, and thereforeSince $\lambda >0$, any minimizer must lie in $V$. For $f_{\alpha }={\sum }_i{\alpha }_i{k}_{x_i}$, the reproducing property gives the evaluation vector $K\alpha$ and $\Vert f_{\alpha }{\Vert }_{\mathcal{H}}^2={\alpha }^{T}K\alpha$. Hence $Q_1(f_{\alpha })=M(\alpha )$. If $\hat{f}$ minimizes $Q_1$, its representation yields a minimizing $\hat{\alpha }$. Conversely, if $\hat{\alpha }$ minimizes $M$, then, for any $f\in \mathcal{H}$, representing $f_V=f_{\alpha }$ gives $Q_1(f)\ge M(\alpha )\ge M(\hat{\alpha })$. ThusThis is an equivalence of minimizers, not an existence or uniqueness assertion: the loss function is arbitrary. A singular kernel matrix also allows nonunique coefficients, since differences in $\ker K$ represent the zero function. Strict convexity of the loss was not used.

Now use the stated minimum-norm description of a sum of reproducing-kernel Hilbert spaces. Let $\hat{f}={\sum }_j{\hat{f}}_j$. If the minimizing tuple did not attain the minimum total squared norm among decompositions of $\hat{f}$, replacing it by that unique minimum-norm tuple would preserve the loss function and strictly lower the penalty. This contradicts optimality. ThereforeFor an arbitrary $f\in \mathcal{H}$, take its minimum-norm decomposition $(f_1,\dots ,f_p)$. Then $Q_1(\hat{f})=Q_2({\hat{f}}_1,\dots ,{\hat{f}}_p)\le Q_2(f_1,\dots ,f_p)=Q_1(f)$, proving that $\hat{f}$ minimizes $Q_1$.

Choose $\hat{\alpha }$ from the representer theorem for this minimizing sum, and define $g_j={\sum }_i{\hat{\alpha }}_i{k}_j(\cdot ,x_i)$. Write $K_j$ for the component kernel matrices. ThenBy uniqueness of the minimum-norm decomposition, $g_j={\hat{f}}_j$ for every $j$. This proves the common representer coefficients for a sum of kernels conclusion:The same coefficient vector works simultaneously. Even when $K$ is singular, changing that vector by $a\in \ker K$ leaves every component unchanged: $0=a^{T}Ka={\sum }_j{a}^T{K}_{j}a$ forces each nonnegative squared component norm to vanish.

The population risk of a prediction rule is its expected loss on a fresh observation from the target probability distribution.

Given observed data $x$, the posterior expected loss of action $a$ iswhere $\Pi (\cdot \mid x)$ is the posterior distribution and $L$ the loss function. Nonnegative losses define an extended nonnegative integral; meaningful comparisons require checking when finite loss is attainable. A Bayes decision rule chooses a measurable minimizer, or an approximate minimizer if an optimum is not attained.

With finite posterior second moment and squared-error loss, completing the square gives $\rho (a\mid x)=\mathrm{Var}(\Theta \mid x)+(a-\mathbb{E}[\Theta \mid x]{)}^2$. Thus the posterior mean minimizes the loss. With finite first moment and absolute-error loss, a posterior median minimizes it instead. Posterior expected loss conditions on the observed data and averages over parameter uncertainty; the frequentist risk of a decision rule fixes the parameter and averages over possible data.

Regularization constrains or penalizes fitting to control the complexity or instability of a learned rule. A penalty formulation iswhere $L_n$ is empirical loss function and $J$ measures a chosen notion of complexity. Ridge regression penalizes squared coefficient size, the Lasso penalizes the L1 norm, and a roughness penalty penalizes derivatives of a fitted function. Early stopping can regularize an iterative fitting procedure without an explicit added penalty.

For a scalar observation $Y=\theta +\epsilon$ with centered noise of variance ${\sigma }^2$, minimizing $(Y-a{)}^2+\lambda a^2$ gives $a=Y/(1+\lambda )$. Its mean squared error is $({\lambda }^2{\theta }^2+{\sigma }^2)/(1+\lambda {)}^2$: shrinkage reduces variance while introducing bias. This illustrates why regularization can improve prediction, but does not improve risk at every parameter value for every tuning choice. Cross-validation or a validation set can choose strength while a reserved test set assesses the fitted procedure.

Robust statistics studies how estimators and other statistical procedures respond to departures from a reference statistical model. Such departures can include a small fraction of outliers, a heavier-tailed probability distribution, or an incorrect assumed shape. A robustness statement specifies both the permitted departure and the quantity whose change is controlled.

For data $(0,1,2,3,M)$ with $M\ge 4$, the sample median remains $2$, whereas the sample mean is $(6+M)/5$ and grows without bound. This does not make the sample median an estimator of every desired expected value: the mean and median of an asymmetric probability distribution can differ. The estimand must therefore be specified before judging robustness.

Influence functions describe infinitesimal changes, breakdown points describe large finite-sample failures, and minimax asymptotic bias compares worst cases over a neighborhood. M-estimators connect these questions with an explicit loss function. Peter J. Huber's 1964 location paper developed a systematic treatment of estimation under contamination.^[ref]

A scoring rule turns a probability distribution forecast and an observed outcome into a number. For example, a forecast of rain with probability $0.7$ should express uncertainty more usefully than a bare prediction of rain. A scoring rule lets us compare such forecasts after observations arrive, and its choice determines whether exaggerating confidence pays. A proper scoring rule rewards an honest account of uncertainty in expected value.

There are two sign conventions. A reward $S(P,x)$ is maximized; a loss function $L(P,x)$ is minimized. The conversion $L=-S$ reverses every comparison. Here the linear probability score is a reward, while the binary Brier score and the negative logarithmic scoring rule are losses. Calling all three a "score" without specifying the direction would hide the central difference.

Throughout the binary examples, the random variable $Y$ has a Bernoulli distribution with actual success probability $q=0.7$, and the report $p\in [0,1]$ assigns probability $p$ to $Y=1$ and $1-p$ to $Y=0$. The value $q=0.7$ is stipulated by this statistical model; it is not inferred from data or a claim that a forecasting system is calibrated. In practice the true predictive distribution is unknown, and the relevant $q$ is the conditional probability of success given the information available when the forecast is issued. Propriety evaluates incentives for reporting that predictive distribution; it does not supply the unknown probabilities.

Statistical decision theory compares procedures by the losses their decisions incur under a statistical model. Specify observation laws $P_{\theta }$, an action space $\mathcal{A}$ and a loss function $L(\theta ,a)$. A decision rule $\delta$ maps data to an action, and its risk function isFor estimating a normal mean from $n$ observations of known variance ${\sigma }^2$, the sample mean has squared-error loss risk ${\sigma }^2/n$. Under zero-one loss, the same framework describes classification in statistical learning and statistical hypothesis testing.

Different criteria compare the whole risk function differently. A Bayes decision rule minimizes risk averaged over a prior distribution. A minimax decision rule minimizes the worst-case risk. An admissible decision rule has no competitor with no greater risk everywhere and smaller risk somewhere. These criteria require the model, allowed actions and loss to be specified; accuracy alone does not define what mistakes cost.

If sample transformations preserve the model and both hypotheses, the invariance principle restricts attention to tests whose decisions are unchanged by those transformations. Such tests are functions of a maximal invariant. In estimation, the analogous requirement is equivariance, accompanied by an invariant loss function so that risks transform compatibly. The principle respects symmetries; it does not by itself prove optimality among all unrestricted procedures.

Statistical learning constructs predictive rules from data and studies their prediction error, generalization, and computational fitting. In supervised learning, training data contain inputs $X$ and a response variable $Y$: regression analysis predicts a numerical response, while classification in statistical learning predicts a label. Unsupervised learning instead seeks structure without supplied responses, as in cluster analysis and principal component analysis. These tasks connect learning to probability and mathematical optimization. ^[ref]

For a fitted rule $\hat{f}$ and loss function $L$, the predictive goal iswhere $(X^{\ast },Y^{\ast })$ is a fresh observation, independent of the training data ${\mathcal{D}}_n$, from the intended target probability distribution. This prediction error fixes the fitted rule and averages over future observations; averaging again over training data gives the learning procedure's average performance. Its practical beauty is that a rule compresses past observations into useful future predictions. Its central difficulty is that excellent training error need not imply good generalization: cross-validation supports model selection, and a reserved test set assesses the resulting choice.

Statistical learning theory studies mathematical guarantees and limitations for learning from data, including how a hypothesis class, loss function, and sample size affect generalization. It distinguishes a particular fitted rule's prediction error from the expected performance of the learning procedure across possible training data.

Supervised learning fits a prediction rule from training data containing both inputs and associated target responses. Regression analysis predicts numerical responses; classification in statistical learning predicts class labels. A loss function specifies which prediction errors matter, and generalization concerns performance beyond the observations used to fit the rule.

Training error is the average loss function evaluated on the observations used to fit the prediction rule. It is the empirical risk of the fitted rule on its own training data. With squared-error loss, it is the average squared regression residual, measuring fit to observed responses rather than performance on fresh responses. Flexible fitting usually makes it optimistically biased for prediction error, as quantified by prediction optimism.

For a reproducible polynomial regression example, take $x_i=-1+2i/11$, $i=0,\dots ,11$, and the known noiseless target $f(x)=1+x/2+2x^2$. Construct responses $y_i=f(x_i)+e_i$ with fixed disturbancesOrdinary least squares over polynomials of polynomial degree at most 1, 2, and 7 gives training errors $0.518937$, $0.015784$, and $0.009257$. The monotone improvement follows because these candidate families are nested: every smaller-degree fit remains available in the larger family. This argument concerns minimum training error, not generalization, and assumes the fitting mathematical optimization is solved. Degrees 2 and 7 fitted instead to the exact noiseless responses $f(x_i)$ have mathematically zero training error; the script obtains approximately $5.31\times 10^{-31}$ and $7.82\times 10^{-31}$ from rounding. Extra flexibility alone does not force overfitting.