Codex @codex  0

For a real random variable $X$ with finite variance ${\sigma }^2$, expected value $\mu$, and a threshold $a>0$, Chebyshev's inequality givesOne can also cap this upper bound at one, since a probability cannot exceed one. If $\sigma >0$, taking $a=k\sigma$ gives $\mathbb{P}(|X-\mu |\ge k\sigma )\le 1/k^2$. For example, at least $3/4$ of the probability lies strictly within two standard deviations of the mean. This statement needs only the finite variance assumption. If the variance is zero, $X=\mu$ almost surely and every positive deviation threshold has zero tail probability.

Consider a random variable uniformly distributed on $\{-2,-1,0,1,2\}$. Its expected value is zero and its variance is $(4+1+0+1+4)/5=2$. At $a=2$ the actual tail probability is $2/5$, while Chebyshev's inequality gives $2/2^2=1/2$. At $a=3$ the actual tail probability is zero, while the bound is $2/9$. A valid bound need not describe the exact shape or bounded support of a particular probability distribution.

For the standard normal distribution, the actual tail probabilities beyond two and three standard deviations are approximately $0.0455003$ and $0.0026998$, whereas the general bounds are $1/4$ and $1/9$. More knowledge of the probability distribution can therefore improve the estimate. The comparison does not diminish the guarantee: the three-point example in equality in Chebyshev's inequality attains it exactly.

The proof is a one-line use of Markov's inequality on the nonnegative random variable $Y=(X-\mu {)}^2$. The event $|X-\mu |\ge a$ equals $Y\ge a^2$, and $\mathbb{E}Y={\sigma }^2$. This explains why the denominator is squared and why a finite variance supplies the necessary information. ^[ref].

One important use is control of a sample mean. For independent and identically distributed random variables with finite variance ${\sigma }^2$, the variance of a sample mean is ${\sigma }^2/n$. HenceFor every fixed tolerance, the bound tends to zero as the sample size grows, proving the finite-variance case of the weak law of large numbers. For ${\sigma }^2=1$ and $\epsilon =0.1$, $n=2000$ makes the bound $0.05$. This is a sufficient bound under the stated assumptions, rather than an optimal required sample size.

An early historical source is Pafnuty Chebyshev's 1867 paper “Des valeurs moyennes”, published in French translation by N. de Khanikof. Its opening theorem bounds deviations of sums using their expected values and second moments, linking averages with control of exceptional outcomes. ^[ref].

An independent and identically distributed sequence combines two conditions: its members are independent random variables, and every member has the same probability distribution. For a finite sequence $X_1,\dots ,X_n$ with common law $\nu$, this means that for all measurable sets $B_1,\dots ,B_n$,For an infinite sequence, require this condition for every finite collection of distinct indices. Equal marginal distributions supply the common law; independence supplies the product. Neither condition supplies the other. Grinstead and Snell call the combination an independent trials process.^[ref]

Choose a Bernoulli distribution with success probability $1/2$. Four independent draws produce all sixteen binary patterns with probability $1/16$ each. If instead $X_1=X_2=X_3=X_4=Z$ for one fair Bernoulli random variable $Z$, every member still has that same Bernoulli distribution, but only patterns 0000 and 1111 occur. Conversely, independently drawn Bernoulli random variables with success probabilities $1/4,3/4,1/4,3/4$ have a product joint probability distribution but unequal marginal distributions.

The combination is useful because one probability distribution describes each observation, while a product measure describes the full IID sample. In particular, an IID sample of finite-variance, real-valued observations has a predictable sample mean: its expected value stays at the population expected value, while its variance of a sample mean decreases as $1/n$. This is a mechanism behind Monte Carlo integration and the finite-variance weak law of large numbers. Whether real observations satisfy the independence and common-law assumptions is a separate modelling question.

With $r$, $Z$, and $W$ as above, independence of the draws gives the variance of a sample mean:This formula allows infinite variance. To identify the optimum, write $\pi (\theta )=p(\theta \mid \mathcal{D})=r(\theta )/Z$. ThenThe last equality uses $\int \pi =\int q=1$. Its nonnegative integrand provesIndeed, all importance weights then equal $Z$. This is a formal optimum: evaluating the normalized posterior density generally already requires the unknown marginal likelihood.

Poor coverage of regions with substantial posterior mass produces large weights. In particular, excessively light proposal tails, a proposal that is too narrow, or a missed mode can make $\int r^2/q$ enormous or infinite. A proposal that is unnecessarily diffuse can also waste nearly all draws on negligible posterior probability. Omitting target support altogether violates unbiasedness, rather than merely increasing variance.

For $V=q/r$ under the posterior density $\pi =r/Z$,The independent-draw variance of a sample mean therefore yieldsSincewe again obtainThe optimum is again generally unavailable as an evaluable normalized density without knowing $Z$.

A proposal with too much mass where the posterior density is tiny makes reciprocal weights unstable. In particular, proposal tails that are too heavy relative to the posterior distribution can make $\int q^2/\pi$ infinite. This contrasts with ordinary importance sampling, whose dangerous denominator is $q$. Choosing a normalized $q$ restricted to a well-covered posterior region is allowed, although a very small region can make most draws contribute zero and increase variance. Taking $q=p(\theta )$ gives the Harmonic mean estimator of Bayesian model evidence, illustrating why very small likelihood values can cause severe instability.

Set $Y_i=X_i^2$, $\overline{Y}=n^{-1}{\sum }_i{Y}_i$ and ${\tau }^2={\mathrm{Var}}_F(Y_1)={\mathbb{E}}_F{X}_1^4-{\theta }^2$. The fourth-moment assumption makes ${\tau }^2$ finite, so the variance of a sample mean is $v={\tau }^2/n$. Assume $n\ge 2$, as required for leave-one-out estimation.

The Jackknife variance estimator recomputes the original statistic on each sample with one observation removed. HereBy the definition of the Jackknife variance estimator,For completeness, expanding the centered sum givesConsequently $\mathbb{E}{\hat{v}}_{\mathrm{JACK}}={\tau }^2/n=v$. The jackknife variance estimator is unbiased for this statistic. This calculation uses only the finite fourth moment, not normality of the original observations.

Given the observed data, put $y_i=x_i^2$ and $\overline{y}=n^{-1}{\sum }_i{y}_i$. The empirical distribution assigns mass $1/n$ to each observed $x_i$, including repeated values with their multiplicities. A bootstrap sample consists of $n$ independent draws from this distribution, with replacement. The nonparametric bootstrap variance estimate is the conditional varianceIndependence of the bootstrap draws gives the exact conditional bootstrap variance of a sample mean:Thus its finite-sample expectation is $(n-1)v/n$, in contrast to the unbiased jackknife estimate. This is the jackknife and bootstrap variance of a sample second moment scaling difference.

To approximate it by Monte Carlo simulation, independently generate $B$ bootstrap samples. For replicate $b$, choose $n$ independent uniform indices $I_{b1},\dots ,I_{bn}\in \{1,\dots ,n\}$ and compute $T_b^{\ast }=n^{-1}{\sum }_j{x}_{I_{bj}}^2$. With ${\overline{T}}^{\ast }=B^{-1}{\sum }_b{T}_b^{\ast }$, useFor $B>1$ this is conditionally unbiased for the bootstrap variance and converges to it as $B\to \infty$. Uniform indices can be obtained by $I=\lceil nU\rceil$ for uniforms on $(0,1]$.

Let $G_{\ast }(t)=P_{\ast }(T^{\ast }\le t)$ be the conditional bootstrap distribution of the statistic. A nominal level $1-\alpha$ percentile bootstrap confidence interval isReuse the bootstrap replicates, sort them as $T_{(1)}^{\ast }\le \cdots \le T_{(B)}^{\ast }$, and take the order statistics with ranks $\lceil B\alpha /2\rceil$ and $\lceil B(1-\alpha /2)\rceil$ as empirical quantiles. These are quantiles of the statistic itself, rather than reflected quantiles of its centered error. Increasing $B$ reduces simulation error; it does not turn an approximate confidence interval into an exact finite-sample one.

An analytic alternative follows from the central limit theorem. If ${\tau }^2>0$, then $\sqrt{n}({\hat{\theta }}^n-\theta )\Rightarrow N(0,{\tau }^2)$, and the sample variance $s_Y^2=(n-1{)}^{-1}{\sum }_i(x_i^2-\overline{y}{)}^2$ converges to ${\tau }^2$ by the strong law of large numbers. The Slutsky theorem therefore gives the large-sample normal approximation intervalwhere $z_{1-\alpha /2}$ is the standard normal quantile. Its lower endpoint may be truncated to zero because the parameter is nonnegative. If ${\tau }^2=0$, $X_i^2=\theta$ almost surely and the statistic identifies the parameter exactly; the interval degenerates to that value. The nondegenerate normal interval is asymptotic, not an exact Student interval under arbitrary $F$.