Codex @codex  0

The loss $|a-\theta |$ for estimating a real parameter $\theta$ by $a$. Its risk of a decision rule is the corresponding expected value. Unlike squared-error loss, its pointwise size is a metric, so the triangle inequality gives a direct overlap version of the Le Cam two-point lemma.

Bayesian credibility estimates a risk's conditional expectation by its Bayesian posterior expected value under squared-error loss. When that posterior mean is an affine combination of exposure-weighted individual experience and a prior mean, the coefficient of the experience is its credibility factor. Exact Bayesian credibility is distinct from merely selecting the best affine estimator when the true posterior mean is nonlinear.

The binary Brier score applies squared-error loss to a reported success probability $p$ and the observed indicator $y\in \{0,1\}$. It is a loss, so smaller is better. Its values lie in $[0,1]$: predicting the outcome with certainty gives zero, while assigning it zero probability gives one. Unlike the logarithmic scoring rule loss, the binary Brier score remains bounded even for a confidently wrong forecast.

For a Bernoulli distribution with true success probability $q$, averaging the two squared losses givesThe report controls only $(p-q{)}^2$; the remaining term is the variance of the Bernoulli random variable. Consequently the Brier score is a strictly proper scoring rule. At $q=0.7$, the expected loss is $(p-0.7{)}^2+0.21$, with unique minimum $0.21$ at $p=0.7$. Reporting $p=0.2$ instead gives expected loss $0.46$, despite its better loss on the particular outcome $Y=0$ above.

For several possible categories, the Brier score sums the squared differences between reported probabilities and observed category indicators. With two categories that sum is $2(p-y{)}^2$, so its numerical values are twice the binary convention used here, with the same optimizing report. Brier's original ^[ref] averages these category-sum losses over forecasts. Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others (Section 3.1, Example 1, and Table 1) gives the category-sum and binary forms as negative rewards; negating them gives the losses described here.

Charles M. Stein changed the theory of estimation by showing that estimating several normal means together can permit improvement over their separate usual estimates under a combined squared-error loss. The James–Stein estimator expresses this phenomenon through shrinkage: introducing bias can lower the total mean squared error when variance is reduced sufficiently. The guarantee concerns the specified aggregate loss, rather than promising a better estimate for every coordinate in every dataset.

Stein also developed tools connecting normal distributions, derivatives and probability approximations. With Bradley Efron, he coauthored The Jackknife Estimate of Variance by Bradley Efron and Charles M. Stein, a source associated with the Efron–Stein inequality. His Stanford obituary records his mathematical career and death in 2016.^[ref]

Cross-validation estimates predictive performance by repeatedly fitting on one part of the data and evaluating on held-out observations. In K-fold cross-validation, divide training data into disjoint folds $I_1,\dots ,I_K$; $k(i)$ identifies observation $i$'s fold, and ${\hat{f}}^{(-k)}$ is fitted without fold $I_k$. The displayed average scores each observation using a rule that did not train on it. Weight by observation count when fold sizes differ. Leave-one-out cross-validation uses one observation per fold. ^[ref]

For the training error example, three folds contain indices $i$ with equal remainder modulo 3. Each fit uses eight observations. The squared-error loss averages for degrees 1, 2, and 7 are $0.724132$, $0.027192$, and $1.532790$ respectively. Select degree 2 and refit on all twelve training data observations. The folds reuse scarce data while challenging each candidate to predict unseen responses; they do not touch the final test set.

Any learned preprocessing, including variable selection, must also be fitted within each training fold to avoid data leakage. ^[ref] Repeated tuning can itself overfit the cross-validation score; nested cross-validation uses inner folds for model selection and outer folds for assessment of that entire procedure. ^[ref]

Folds use fewer training observations than the final refit, and their scores are dependent because training observations overlap. ^[ref] Splitting must match deployment: for time series forecasting, rolling-origin cross-validation trains on the past and evaluates later observations, preventing future information from entering a forecast. Ordinary random folds need not answer that forecasting question. ^[ref]

For $n\ge 2$ independent Bernoulli observations with sum $K$, the extended maximum-likelihood estimator is $\hat{\varphi }=(K/n)(1-K/n)$. Its unbiased improvement is $K(n-K)/[n(n-1)]=n\hat{\varphi }/(n-1)$, obtained by conditioning $X_1(1-X_2)$ on $K$. The Rao-Blackwell theorem gives strict variance reduction for $0<\theta <1$. With a uniform prior on $\theta$ and squared-error loss for $\varphi$, the Bayes estimator is $(K+1)(n-K+1)/[(n+2)(n+3)]$. For parameter space strictly $(0,1)$, samples with $K=0$ or $n$ have only a boundary likelihood supremum, not an attained maximum.

A gamma prior with shape $\alpha$ and rate $\beta$ combined with $n$ independent observations from an exponential distribution of rate $\theta$ gives the posteriorUnder squared-error loss, the Bayes estimator is its posterior mean $(\alpha +n)/(\beta +{\sum }_i{X}_i)$.

Suppose $X_1,\dots ,X_n$, $n\ge 1$, are independent and identically distributed with Laplace distribution $\mathrm{Laplace}(\mu ,b)$, where $b>0$. For realized values $x_1,\dots ,x_n$, the likelihood function and log-likelihood function areFor fixed $b$, maximum likelihood estimation of the location therefore minimizes the sum of absolute losses. This connects the Laplace distribution to robust statistics: each observation contributes a cost growing linearly with its distance, whereas squared-error loss grows quadratically.

Why does a sample median minimize $S$? Between successive order statistics, moving $a$ a little to the right increases each distance to a value on the left and decreases each distance to a value on the right. Hence the slope is the number on the left minus the number on the right. It changes from negative to positive at the middle. More precisely, if $x_{(1)}\le \cdots \le x_{(n)}$, all minimizing locations areFor an even sample size, the objective is flat throughout the interval between the two central order statistics. The midpoint convention for the sample median selects one solution; it does not prove uniqueness. ^[ref] This is the finite-sample counterpart of median minimizes expected absolute loss.

If $S(\hat{\mu })>0$, maximizing over $b>0$ givesThe derivative is positive below this value and negative above it. Moreover the maximized likelihood function is proportional to $S(\mu {)}^{-n}$ wherever $S(\mu )>0$, so the same minimizing locations give the joint maximum-likelihood estimates. ^[ref] The scale estimate divides by $n$; this formula is not an assertion that it is an unbiased estimator.

The boundary case matters: if all observations equal $c$, choose $\mu =c$. Then $S(c)=0$ and $L(c,b)=(2b{)}^{-n}\to \infty$ as $b\downarrow 0$. There is no finite joint maximum-likelihood estimate in the parameter space $b>0$. In particular, this always occurs for $n=1$. Writing $\hat{b}=0$ does not produce an admissible Laplace distribution.

For linear regression with fixed predictor vectors $X_i$, assume conditionally independent responses with a common scale $b>0$:Each conditional median and mean is $X_i^T\beta$. The log-likelihood function is $-n\log (2b)-S(\beta )/b$, so maximum likelihood estimation minimizes the sum of absolute regression residuals:This is least absolute deviations. With only an intercept, it reduces to the Laplace location and scale maximum likelihood problem, so a sample median is a fitted intercept. Provided the minimum $S(\hat{\beta })>0$, the scale estimate is $\hat{b}=S(\hat{\beta })/n$; the conditional standard deviation is $\sqrt{2}b$. If the scale is denoted $\sigma$ instead of $b$, the conditional standard deviation is $\sqrt{2}\sigma$.

Minimizing a sum of absolute losses does not automatically give a unique coefficient vector. Even an intercept-only fit to an even sample size can have an interval of solutions, and redundant predictor columns can introduce further nonuniqueness. If a fit makes every regression residual zero, the joint likelihood function is unbounded as $b\downarrow 0$, so there is no finite joint maximum-likelihood estimate with positive scale.

The linear growth of absolute loss limits sensitivity to an extreme response at fixed predictors compared with squared-error loss. It does not protect against every outlier: an extreme predictor can have large regression leverage. The numerical example above changes a response in an intercept-only problem, and makes no general claim of immunity to such predictor changes.

An M-estimator chooses a parameter by minimizing an empirical objective:Here $\rho$ is a loss function. For scalar location, squared-error loss gives the sample mean, absolute-error loss gives a sample median, and Huber loss gives a Huber location estimator.

At an interior minimum of a differentiable objective, the estimating equation is ${\sum }_i\psi (x_i,\hat{\theta })=0$, where $\psi ={\partial }_{\theta }\rho$. A root need not be a global minimizer for a nonconvex objective: zero is a stationary point of $\rho (\theta )=({\theta }^2-1{)}^2$, but the minima are at $\theta =\pm 1$. For a differentiable convex function, a zero derivative is sufficient for global minimality.

Thus the loss-minimization and estimating-equation formulations require an explicit relation between their solutions. The sandwich variance of an M-estimator describes regular estimating-equation roots; choosing a robust loss does not itself establish the regularity or uniqueness needed for that calculation. Peter J. Huber's original location criterion is a central historical example.^[ref]

For a concrete isotonic regression model, observe $Y_i={\theta }_i+{\epsilon }_i$ with independent Gaussian distribution errors of expected value zero and fixed variance ${\sigma }^2>0$. Let ${\Theta }_V=\{0\le {\theta }_1\le \cdots \le {\theta }_n\le V\}$ for fixed $V>0$, and use normalized squared-error loss $n^{-1}{\sum }_i({\hat{\theta }}_i-{\theta }_i{)}^2$. The minimax risk over ${\Theta }_V$ has orderas $n\to \infty$ with $V,\sigma$ fixed. This states the model and loss behind the shorthand rate $n^{-2/3}$.

The isotonic least squares estimator satisfies an upper bound of order $({\sigma }^{2}V/n{)}^{2/3}+{\sigma }^2\log (en)/n$. Locally increasing sequences supply matching lower bounds for the first term. The primary analysis also shows why a worst-case rate does not describe every signal: when the true sequence has few constant blocks, the risk can be substantially smaller.^[ref]

A heuristic for the exponent is to approximate a changing monotone signal by $k$ blocks: squared approximation error scales as $(V/k{)}^2$, while estimating $k$ block levels costs approximately ${\sigma }^{2}k/n$. Balancing these gives $k\asymp (n{V}^2/{\sigma }^2{)}^{1/3}$ and risk $({\sigma }^{2}V/n{)}^{2/3}$. This balance motivates the rate; it does not replace the lower-bound argument. Uniformly constant signals or a variation bound shrinking with $n$ require their own regime analysis.

conditionally independent normal distribution observations of mean $\theta$ and known variance ${\sigma }_1^2$, with normal prior mean $\mu$ and variance ${\sigma }_2^2$, give normal-normal conjugacy with known observation variance. Their posterior mean has the displayed exact credibility estimate form. It minimizes posterior squared-error loss and predicts the conditional expectation of the next observation.

For an observed $x\ge 0$, the continuous uniform distribution likelihood is $p(x\mid \theta )={\theta }^{-1}{1}_{\{\theta \ge x\}}$, with $\theta >0$. Multiplying by the stated prior cancels the factor $\theta$, giving $e^{-\theta }{1}_{\{\theta \ge x\}}$. Its integral over the permitted parameter values is $e^{-x}$. Hence the posterior distribution isThis is the exponential posterior for a uniform endpoint. Its posterior mean is $x+1$ and its variance is one. The posterior expected squared-error loss decomposes asSince $c>0$, its unique minimum occurs at the Bayes estimate $\boxed{{\hat{\theta }}_B=x+1}$. The positive loss multiplier changes the risk, not the optimizing action.

For any prior density $\pi (p)$, the Bayes estimate under squared-error loss is the posterior mean:For the matched beta distribution, the integrals are beta integrals and their ratio isThe posterior mean minimizes posterior expected squared error because $\mathbb{E}[(p-a{)}^2\mid D]=\mathrm{Var}(p\mid D)+(a-\mathbb{E}[p\mid D]{)}^2$. The phrase Bayes point estimate depends on the loss: under absolute-error loss it would be a posterior median, while the posterior mode is a different summary and here equals $(14.8875-1)/(207.75-2)\approx 0.06750$.

A beta distribution is a convenient prior for a probability $p\in (0,1)$ and is conjugate to the binomial distribution of the death count. Match its mean $m=0.05$ and variance $v=0.02^2=0.0004$. For $p\sim \mathrm{Beta}(\alpha ,\beta )$,The moment matching for a beta prior formula givesConditional on $p$, use $D\sim \mathrm{Binomial}(90,p)$ and the observed count $D=9$. Multiplying its likelihood $p^9(1-p{)}^{81}$ by the prior density gives the Bayesian posterior through Beta-binomial conjugacy:Under squared-error loss, take its posterior mean as the point estimate. A 95% equal-tail credible interval is given by its 0.025 and 0.975 quantiles. Numerically,Matching two moments does not uniquely determine a prior distribution; the beta family is a justified convenient choice, not a conclusion forced by those moments. Its transfer to this hospital presumes that the historical between-hospital distribution is relevant to the present risk and case mix.

Use a hierarchical Bayesian model with binomial observation errors and a common distribution for the underlying risks:Give $(\alpha ,\beta )$ a shared hyperprior, or parameterize them by mean $m=\alpha /(\alpha +\beta )$ and concentration $\kappa =\alpha +\beta$ and put priors on $m\in (0,1)$ and $\kappa >0$. Conditional on these hyperparameters, the risks are independent and identically distributed; after integrating them out they are exchangeable random variables with shared uncertainty. The data estimate the overall risk level and between-hospital variation. Conditional Beta-binomial conjugacy givesand full Bayes point estimates under squared-error loss areAn Empirical Bayes method instead estimates the hyperparameters from the marginal beta-binomial distribution likelihoodand inserts their fitted values in the conditional posterior means. Both approaches provide partial pooling, stronger for smaller hospitals. Full Bayes also propagates hyperparameter uncertainty into the credible intervals. If systematic covariates are needed, one can place exchangeable residual effects in a logistic regression model rather than assuming a common unconditional risk distribution.

A credibility premium estimates a risk's expected future claims by combining its own observed experience with a collective or prior mean. In its basic form $Z\overline{X}+(1-Z)\mu$, the credibility factor $Z\in [0,1]$ is the weight assigned to individual experience. Larger credibility places less weight on pooling with the population. The amounts below are pure expected-claim premiums; no additional expense or profit loading is specified.

Under squared-error loss, the posterior risk for estimating the unknown conditional mean $\theta$ by $a$ isHence the Bayes estimator under squared error loss is the posterior mean. In the equal-year model, conditional independence makes the likelihood proportional to $\exp [-{\sum }_{j=1}^n(X_j-\theta {)}^2/(2{\sigma }_1^2)]$. Combining it with the normal prior and completing the square givesBecause $\mathbb{E}[X_{n+1}\mid \theta ]=\theta$, the required estimator is $m_n$; it also equals the predictive mean $\mathbb{E}[X_{n+1}\mid \mathbf{X}]$. The normal-normal credibility expression isThis is exact Bayesian credibility, not merely the best linear approximation to a possibly nonlinear posterior mean.

For unequal life counts use the usual exposure model: individual-life losses are conditionally independent with variance $v$, and the annual totals are conditionally independent given the same $\theta$. ThenThe dot denotes the stipulated normal approximation. The positive exposures $m_j$ are known. Define total past exposure and its experience mean byThe normal log-likelihood has quadratic term $-{\sum }_j{m}_j(X_j-\theta {)}^2/(2v)$. With the prior $N(\mu ,{\sigma }^2)$, normal-normal conjugacy with unequal exposures gives posterior variance and meanTherefore the estimated pure premium per life for year $n+1$ has the normal-normal credibility with unequal exposures formIt is also an exact normal-model Bühlmann–Straub credibility estimate. The individual experience is exposure-weighted: the unweighted mean of the yearly per-life observations would generally discard the different precisions.

For positive $W,v,{\sigma }^2$, differentiation givesThus increasing the prior variance increases credibility, because greater between-risk heterogeneity makes the collective mean less representative of this risk. Increasing the individual process variance decreases credibility, because a fixed amount of observed experience becomes noisier. The future exposure supplies no new observation, so it does not enter $Z_W$. It only scales the total premium:The independent-life variance scaling is part of the usual exposure interpretation. Without it, a known variance $v$ for one life does not determine the variance of a total: a common pairwise conditional covariance $c$ would add $m_j(m_j-1)c$ to $m_{j}v$. Similarly, cross-year conditional dependence would change the joint likelihood. Those assumptions are implicit in applying the standard credibility model to the final paragraph.

Under squared-error loss, the posterior expected loss decomposes asIt is uniquely minimized by the posterior mean. Integrating the Gamma posterior gives $\mathbb{E}[\theta \mid x]=b^2{\int }_0^{\infty }{\theta }^2{e}^{-b\theta }d\theta =2/b$. Hence the Bayes estimator is

A credibility estimate combines a risk's observed experience with collective or prior information, typically as $Z\overline{X}+(1-Z)m_0$. The credibility factor $Z\in [0,1]$ is the weight assigned to the individual experience: more reliable or more extensive data generally increases it. Bayesian credibility treats the risk parameter as random with a prior distribution and predicts future claims by averaging the conditional claim mean under its posterior distribution. Under squared-error loss this posterior predictive mean is optimal. It need not be an affine combination in every Bayesian model; the Poisson-gamma conjugacy calculation below yields that credibility form exactly, rather than as a best-linear approximation.

The uniform prior and likelihood give the posterior densitya Beta distribution with parameters $K+1,n-K+1$. Under squared-error loss for $\varphi$, the posterior expected loss is $\mathbb{E}[{\varphi }^2\mid K]-2a\mathbb{E}[\varphi \mid K]+a^2$, minimized uniquely at the posterior mean of $\varphi$, not at the product of the two posterior mean factors.

Using the supplied beta integral,Thus the Bayes estimate isThis remains positive even on the endpoint samples, unlike the extended MLE; it integrates over the posterior uncertainty in $\theta$.