Bayes act 2026-10-14
A Bayes act minimizes for a specified parameter distribution . Applying the same minimization to the posterior after observing data gives a Bayes decision rule. The optimal data-based Bayes risk is the expectation of the optimal posterior no-data loss.
Bayes estimator Created 2026-09-24 Updated 2026-11-05
A Bayes estimator chooses an action minimizing posterior expected loss after observing data:
The answer depends on the loss function: squared error gives the posterior mean, absolute error a posterior median, and discrete zero-one loss a posterior mode. For squared error, expanding proves the mean rule when the second moment is finite. Pointwise posterior minimization also minimizes integrated Bayes risk under the usual measurability and integrability conditions.
Bayesian decision problem 2026-10-14
A prior on a parameter, a conditional observation distribution, an action space and a loss specify a Bayesian decision problem. The normal form minimizes prior expected loss over decision rules; the extensive form minimizes posterior expected loss after each observation. Conditional expectation gives the same Bayes risk for these formulations, provided measurable optimal or approximate choices can be made.
Under the hypotheses for the one-nearest-neighbour asymptotic risk, the limiting risk satisfies . Put : then since . Integrate, using the definition of Bayes risk. Constant gives limiting risk and Bayes risk , so the lower comparison is not generally equality.
One-nearest-neighbour classifier Created 2026-10-05 Updated 2026-10-14
The classifier that assigns the query the label of its closest training feature, using feature-measurable nearest-neighbour tie-breaking. Its misclassification risk need not converge to Bayes risk, even with infinitely many independent and identically distributed random variables as training data.
A loss function measures the cost of taking action at parameter . A decision rule selects an action from the data; its risk function is . A Bayes estimator minimizes the prior-averaged risk, or equivalently minimizes posterior expected loss at almost every observation. An admissible rule has no competing rule of no greater risk everywhere and strictly smaller risk somewhere. To construct a Bayes rule, form the posterior and minimize its expected loss over the allowed actions. Under squared error and finite posterior second moments the unique action is the posterior mean.
Put . The given fourth moment and independence give , . Thus
Every is strictly dominated at every by that coefficient. Hence it cannot be a finite-risk Bayes rule for any proper prior.
To make the second requested comparison, let and assume finite quadratic Bayes risk. For the affine rules the integrated risk is
With fixed, the minimum occurs at . Since , this strictly improves the homogeneous rule unless , reducing risk by . Therefore a homogeneous rule with is not Bayes, as requested. Combining the two restrictions excludes every homogeneous coefficient.
For a proper prior without global finite moments, the same nonvacuous conclusion follows from the posterior-mean condition. Let
For , . If the posterior mean were , positivity requires and . Differentiation forces . But the marginal density of would then be a constant times , a pure power not integrable at both zero and infinity. This contradicts a proper prior's normalized predictive distribution. This is the proper-prior obstruction to homogeneous variance Bayes rules.
The finite-loss qualification matters: allowing all posterior losses to equal infinity makes every action a vacuous minimizer. For example, with and proper density , the posterior second moment diverges at positive observations, so a literal infinite-loss definition would defeat the printed assertion. Ordinary meaningful Bayes rules exclude that convention.
A loss function quantifies the cost of taking action when the parameter is . A decision rule is a measurable map from the observation space to the action space (or, for randomized rules, a conditional action distribution). Its risk function is . For a prior probability distribution , its Bayes risk is
A minimax rule minimizes .
Interpret the printed expression as . If , the Gaussian exponential-moment formula gives
Its supremum is one, attained at zero. For every rule , , so and . Hence
In the language of part (b), the point-mass prior makes every rule Bayes with Bayes risk one, and has exactly that worst-case risk.
Write , with independent of the prior . Then , so
Completing the square in the joint density gives . Thus the Bayes rule is , with Bayes risk . The rule has integrated risk one for every , so its excess Bayes risk is as . The priors remain proper at every positive . Therefore is extended Bayes, without treating the limiting improper flat prior as a proper Bayes prior.
For a prior and loss , the Bayes risk of a rule is . A Bayes decision rule minimizes this integrated risk. An Extended Bayes rule is a rule whose excess over the optimal Bayes risk can be made arbitrarily small by a suitable proper prior: for every some has , with finite risks.
Condition on the observation and use iterated expectation. A rule is Bayes precisely when it chooses a minimizer of the posterior expected loss almost surely, up to the usual measurability requirement. Under squared Euclidean loss, with a finite posterior second moment, write . The identity
shows that the Bayes rule is the posterior mean.
Write and . An equaliser rule has constant risk . An Extended Bayes rule has, for every , a proper prior such that , where . If a rule has both properties, every competitor satisfies
Letting proves that its maximum risk is the minimax risk.
For the given normal distribution model, completing the square in prior times likelihood gives the posterior , with and . Another completion of the square gives
This is maximized uniquely at . Consequently the Bayes act and posterior posterior expected loss are
The posterior loss is independent of , so averaging over the predictive distribution gives the same value for the Bayes risk of this Bayes decision rule.
The sample mean obeys , so its risk is the constant . Take any fixed and let the proper prior precisions . The optimal Bayes risks just calculated tend to , while the integrated risk of is exactly for every prior. Thus its excess Bayes risk tends to zero: it is extended Bayes as well as equaliser. The preceding argument proves
In the normal form of a Bayesian decision problem, choose a measurable decision rule before seeing the data to minimize the prior Bayes risk . In the extensive form, after observing , choose an action minimizing posterior expected loss. Conditional expectation makes these equivalent, subject to the usual measurable selection conditions. If is the posterior distribution and is the best no-data loss, then
The inequality follows by ignoring the data.
For a measurable region , posterior or prior expected loss is . It is minimized by including exactly the locations where the integrand is negative; including equality points is harmless. Thus the highest density region is a Bayes act and
For , if the answer is zero. Otherwise put . The optimal interval is , and
For the specified prior and , the prior density is everywhere below , so . Completing the square gives . Its peak is , so a positive-measure region has posterior density strictly above and its optimal expected loss is strictly negative. That loss is independent of its mean, hence independent of . Therefore . Explicitly, with , .
For fixed-design nonparametric regression, let and . The degree- local polynomial estimator takes the fitted intercept , where
Use a nonnegative regression kernel and an invertible local polynomial Gram matrix to obtain a unique fit. For , solving the single weighted least squares normal equation gives the local constant estimator:
At the interior point , use the standard unit-integral regression kernel convention . Otherwise the leading variance below is ; unlike a kernel density estimator, the local constant estimator itself is unchanged by rescaling .
The local constant estimator is a linear estimator in nonparametric regression with weights . The independent unit-variance errors imply
To control the bias of an estimator, differentiability at this one point is enough. Write
Symmetry makes , and the supplied moment approximation gives . Therefore the linear contribution to the bias of an estimator is . The remaining contribution has absolute value at most , because only receive positive weight. This is the interior first-order bias cancellation for local constant regression, and gives squared bias of an estimator . Combining with the variance proves
For fixed , take . It satisfies both asymptotic smoothing bandwidth conditions, and the displayed remainder, multiplied by , tends to zero for this fixed . Thus
Letting proves the optimized scaled pointwise error tends to zero. The order of these limits matters: no remainder uniform in has been assumed.
For the lower bound over the whole function class, the absence of a uniform bound on derivatives permits a particularly direct argument. It also avoids assuming a normal distribution for errors when the model only supplies their first two moments. Fix and . Choose a smooth bump function on the interval, with , whose support contains no design point except possibly . Such a bump exists because the design is finite; at an endpoint use the restriction of a smooth bump on the real line. Every , , belongs to the differentiability class.
If is not a design point, all observation means are zero for every . The Le Cam two-point lemma applied to has total variation distance zero and gives a lower bound tending to infinity with . In fact, the supremum of the mean squared error is infinite at every such .
If is a design point, only its response depends on . The resulting location family is the single observation ; all remaining observations are independent of and give no extra information. The single-observation location minimax bound is
Here is a proof covering discrete as well as continuous errors. Give the prior distribution , independently of the noise. Put and . Consider an easier experiment in which an oracle also reveals whether , and reveals itself on the complementary event. On the event and , every possible truncated noise value places within . The flat prior distribution then makes the Bayesian posterior noise law exactly its truncated original law. The smallest conditional squared-error loss is its conditional variance , attained by the posterior mean.
The probability of that event is at least for , by restricting further to . Thus every rule's supremum risk function is at least the original Bayes risk, which is at least the easier experiment's Bayes risk, and therefore at least . Let and then . Finite second moment and zero mean give . The rule has constant mean squared error one, proving the equality.
Consequently, at every , the unrestricted differentiability class satisfies the stronger conclusion
Thus works for every . There is no conflict with the preceding fixed-function limit: the worst functions, including arbitrarily narrow bumps, can vary with . This is the pointwise versus uniform risk distinction. A smoothness ball with a common derivative bound would pose a different minimax risk problem.
Choose a Borel measurable function version of the posterior success probability . Conditional on , predicting zero has error probability and predicting one has error probability . A Bayes classifier for zero-one loss is therefore
Either decision is optimal when . The displayed Bayes risk also equals and is unaffected by changing on a -null set.
For any measurable set , independence and the uniform distribution of give
Likewise the probability with label zero is . These sets determine the joint probability distribution, so
The same construction makes all training pairs independent and identically distributed random variables.
Fix a feature-measurable nearest-neighbour tie-breaking rule, for example increasing original index for observations at equal Euclidean norm distance from the query. This is important: the ordering may depend on the features, but not on the or labels. Let be the label attached to the th ordered feature. The classifier from the K-nearest neighbors algorithm is
The displayed rule resolves a tied vote in favour of one; any fixed vote convention would also define the method. When , this is the one-nearest-neighbour classifier .
Write for the selected original index. Conditional on all features , it is fixed, and is still uniform on . The two decisions being compared are
The latter is an oracle comparison rule: its relabeling is recomputed at each query , rather than being a fixed relabeling of the training data for all queries. For a variable with the uniform distribution on , the two threshold indicators at differ exactly when lies between them, an interval of length . This Bernoulli coupling by a shared uniform random variable proves
Take the test pair independent of the training data and its auxiliary uniforms. Let , and interpret as conditional misclassification risk given . The tower property of conditional expectation gives . Conditional on the test feature and all training features, and are independent variables with a Bernoulli distribution of parameter . Consequently, for every ,
This is an exact finite-sample equality, not merely a limit.
The two error indicators can differ only if the two classifiers disagree, so
The permitted nearest-neighbour approximation theorem states that the last expected value tends to zero for the feature-based selection rule. This establishes the one-nearest-neighbour asymptotic risk
For , put . Then , and . Integrating gives the Bayes risk bound for one-nearest-neighbour classification
Thus a one-nearest-neighbour classifier need not achieve the Bayes risk: for constant , its limiting misclassification risk is , while .
Feature-based neighbour tie-breaking is required. The convention cannot be dropped for arbitrary . For a counterexample, take almost surely and constant , and choose among the coincident neighbours the one with the smallest . Then the predicted label is one with probability , so its misclassification risk tends to , exceeding the claimed upper bound . The corrected theorem uses ties determined independently of the uniforms and labels, as above.
Condition on . For , the posterior expected loss is a strictly convex quadratic in the decision . Its minimizer is
The beta density gives and , so . If , every has infinite posterior expected loss because of the nonintegrable term near zero, whereas has finite loss expectation. The analogous argument at forces . Hence the Bayes estimator is
Indeed is unbiased and has variance , which cancels the loss denominator. Its uniform-prior Bayes risk is .
The printed loss is undefined at , despite the stated closed parameter interval. A convention is necessary there. The natural extended loss assigns zero to the correct endpoint decision and infinity to an incorrect one; it gives endpoint risks zero for . Alternatively a continuous extension of this estimator's risk gives endpoint values . The interior result and the minimax value below are unchanged by either convention, and a uniform prior gives the endpoints measure zero. We do not identify an undefined with a value without specifying this choice.
for an estimator and loss , its risk function is , where expectation is over repeated data at fixed . Its Bayes risk under prior is , equivalently the joint expectation under the prior and sampling model.
Combining the Gaussian likelihood with the prior gives the posterior precision and mean
Thus the posterior in the Gaussian normal-mean shrinkage Bayes estimator is
The Bayes estimator under squared error loss is the posterior mean, so
Its posterior expected loss is the trace of the posterior covariance and does not depend on . Therefore its Bayes risk is
For , let be the Bayes classifier with class-zero prior , and define its two class-conditional error probabilities by
Writing the likelihood ratio as , the rule chooses class one when
The Gaussian likelihood-ratio level sets have probability zero, so dominated convergence shows that and depend continuously on . As , the threshold tends to zero and the classifier chooses class one almost surely, giving
As , it chooses class zero almost surely, giving the opposite limit . The intermediate value theorem therefore provides such that
The rule is an equalizer rule with worst-case risk . For any classifier ,
because minimizes the integrated risk for its prior. Hence is a minimax decision rule, as summarized by the minimax Gaussian Bayes classifier from equal class errors.
The prior is indeed least favorable. Its Bayes risk is , while for any other prior the Bayes risk is at most the integrated risk of the same equalizer rule , namely
Thus no prior has larger Bayes risk.
The conditional class probability is . A Bayes classifier chooses
and its Bayes risk is
A sequence of classifiers is consistent when as , with convergence interpreted in probability or in expectation according to whether the training sample is conditioned upon.
By the Cauchy-Schwarz inequality, part (b), and part (e),
This bound tends uniformly to zero if
The plug-in classifier excess-risk bound then shows that the misclassification risk of the nearest-neighbour classifier converges to the Bayes risk.