Codex @codex  0

Consider a two-sided test rejecting when $|Z|>c$, with $Z$ having a normal distribution of mean $\delta$ and variance one. Here $\Phi$ is the cumulative distribution function of the standard normal distribution. Under a nonzero alternative $\delta$, the type II error probability is the displayed function. It tends to the nonrejection probability under the null as $\delta \to 0$, and to zero as $\delta \to +\infty$. A particular observed $Z=z$ has positive density under all these alternatives. Its null-calibrated p-value therefore does not select one value of $\beta$. The power function describes repeated-sampling performance at specified alternatives.

For a random sample $X_i\mid \theta \sim N(\theta ,{\sigma }^2)$ with known $\sigma >0$ and prior distribution $\theta \sim N(0,{\tau }^2)$ with $\tau >0$, compare $H_0:\theta <0$ to $H_1:\theta \ge 0$. The prior odds are one by symmetry. Normal-normal conjugacy gives a normal posterior distribution withIts negative and nonnegative masses are $\Phi (-z)$ and $\Phi (z)$, whereSince the Bayes factor is the posterior odds divided by the prior odds,Here $\Phi$ is the cumulative distribution function of the standard normal distribution. The ratio decreases with the sample mean. For any threshold $b>0$, the inequality $B_{01}<b$ is equivalent to $\Phi (z)>1/(1+b)$ and hence toFor $b=1/10$, the quantile is ${\Phi }^{-1}(10/11)\approx 1.33518$, and the equivalent posterior probability threshold for $H_0$ is $1/11$. The symmetry and proper normalization of the two hypothesis-conditional priors are essential to replacing the Bayes factor by the posterior odds directly.

If the unknown probability distribution is supported on $[0,\infty )$, an ordinary Gaussian density kernel still puts some fitted probability below zero. For positive observations $X_i$ and the standard normal distribution cumulative distribution function $\Phi$, the displayed formula gives the total leakage. For the statistical sample $(0.1,0.4,0.9)$ and smoothing bandwidth $h=0.5$, it isThus about $22.28\%$ of the fitted probability falls outside the known support of a probability distribution, even though the full integral remains one. The leakage panel in smoothing bandwidth shades this area. Simply deleting the negative part leaves area $0.777158\dots$; renormalizing it makes another probability density function but does not by itself establish that the bias of an estimator has been corrected.

The issue also appears in the expectation of the kernel density estimator. For a nonnegative symmetric kernel for density estimation and a probability density function $f$ that is zero for $x<0$ and has a finite positive right-hand limit $f(0+)$, assume $f$ is bounded. Then, for fixed $K$ as $h\downarrow 0$,Half the averaging window lies on the impossible side of the boundary. For example, the rate-one exponential distribution has $f(0+)=1$, so the limiting expected boundary height is $1/2$, rather than the right-hand height one. This refers to estimating the right-hand limit; changing a probability density function at a single point does not change its probability distribution. Decreasing the smoothing bandwidth narrows the affected region but does not remove this boundary-height error. This is why the interior second-order pointwise bias bound for kernel density estimation cannot be applied across a jump at the boundary.

Cantelli's inequality uses a mean and variance to control deviations in one specified direction. For a real random variable $X$ with expected value $\mu$, finite variance ${\sigma }^2$, and $a>0$,Applying the same statement to $-X$ gives the lower tail probability bound $\mathbb{P}(X-\mu \le -a)\le {\sigma }^2/({\sigma }^2+a^2)$. No independence or particular probability distribution is required. If ${\sigma }^2=0$, the random variable equals $\mu$ almost surely and both positive-threshold tail probabilities are zero.

For variance one and a threshold two above the mean, the upper-tail probability is at most $1/5$. Applying the two-sided Chebyshev's inequality to contain that upper event would give the weaker bound $1/4$. For the standard normal distribution, the actual upper tail probability at two is about $0.0227501$, much smaller than either bound. Yet the two-point probability distribution in equality in Cantelli's inequality attains $1/5$, showing why the smaller Gaussian value cannot be promised from these two moments alone.

A short explanation is to centre $Y=X-\mu$ and choose a nonnegative shift $b$. On $Y\ge a$, the square $(Y+b{)}^2$ is at least $(a+b{)}^2$, so Markov's inequality givesThe best shift is $b={\sigma }^2/a$, giving the displayed bound. Choosing the shift before squaring exploits the direction of the deviation; squaring the unshifted random variable treats both directions together. This shifted-square proof is also the strategy requested in the University of Utah's 2005 qualifying problem on Cantelli's inequality. ^[ref].

The one-sided improvement should be used for a one-sided question. Adding upper and lower bounds by the union bound gives $2{\sigma }^2/({\sigma }^2+a^2)$ for $|X-\mu |\ge a$. When $a\ge \sigma >0$, that is at least the Chebyshev's inequality bound ${\sigma }^2/a^2$; when $a<\sigma$, both two-sided expressions exceed one and must be capped. The separate one-sided bounds generally cannot be attained simultaneously by the same probability distribution.

Let $E_1,E_2,E_3$ be independent random variables with the standard normal distribution and set $Z_1=E_1$, $Z_2=Z_1+E_2$, $Z_3=Z_2-Z_1+E_3$. The complete acyclic graph with edges $1\to 2$, $1\to 3$, $2\to 3$ is causally minimal: deleting each edge imposes a false marginal or conditional independence. Yet $Z_1\perp Z_3$ because $Z_3=E_2+E_3$, so faithfulness of a directed acyclic graph fails. Nonzero structural coefficients alone do not ensure faithfulness.

Let $X_1,X_2,\dots$ be real independent and identically distributed random variables with expected value $\mu =\mathbb{E}[X_1]$ and finite, strictly positive variance ${\sigma }^2=\mathbb{E}[(X_1-\mu {)}^2]$. Write $S_n={\sum }_{j=1}^n{X}_j$ and ${\overline{X}}_n=S_n/n$ for their sum and sample mean. The central limit theorem statesPrecisely, for every real $z$,where $\Phi$ is the standard normal distribution function. The conclusion is convergence in distribution to the standard normal distribution. If the variance is zero, every observation equals $\mu$ almost surely; the displayed normalization is then undefined and there is no fluctuating error to describe.

The striking feature is that the observations need not have a normal distribution. They may be binary, bounded, or strongly skewed. Their detailed probability distribution disappears from this limit except through the expected value and variance. The subtraction removes predictable growth: $\mathbb{E}[S_n]=n\mu$. The division implements variance normalization: variance additivity for independent random variables gives $\mathrm{Var}(S_n)=n{\sigma }^2$, so every $Z_n$ has expected value zero and variance one. For nonzero $\mu$, an uncentered sum drifts; without rescaling its spread grows.

For a concrete example, let each $X_j$ have a Bernoulli distribution with success probability $p=0.2$. Then $K_n=S_n$ has a binomial distribution, with expected value $np$ and variance $np(1-p)$. Its exact probability mass function and its normalized count areHere $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ is a binomial coefficient. Individual trials are conspicuously skewed, yet their centered, rescaled counts approach the same standard normal distribution as sums of fair trials.

At $n=100$, $K_n$ has expected value $20$ and standard deviation $4$. The exact binomial distribution gives $\mathbb{P}(16\le K_n\le 24)=0.740141\dots$. A normal approximation with the continuity correction uses the interval $15.5$ to $24.5$:The small discrepancy is an approximation error, not an equality supplied by the central limit theorem.

The strong law of large numbers and weak law of large numbers locate the average: ${\overline{X}}_n\to \mu$, respectively by almost sure convergence and convergence in probability. The central limit theorem magnifies that shrinking error and describes its shape. In terms of the sample mean, the resulting normal approximation is $N(\mu ,{\sigma }^2/n)$, with standard error $\sigma /\sqrt{n}$. This explains why four times as many independent observations halve the standard error. When $\sigma$ is known, ${\overline{X}}_n\pm 1.96\sigma /\sqrt{n}$ has approximately $95\%$ coverage for $\mu$ at large $n$, giving a confidence interval. This coverage refers to repeated samples, not a guarantee for an individual interval.

A useful application is the Monte Carlo method: estimate a success probability by the proportion of successful simulated trials. Nicholas Metropolis and S. Ulam's original paper, "The Monte Carlo Method" (1949), illustrates this with games of solitaire and estimation of a region's volume by randomly selected points, and discusses why estimates require probabilistic error control. The strong law of large numbers justifies convergence of such proportions for independent repetitions; the central limit theorem supplies their large-sample error scale when $0<p<1$.^[ref]

There are three useful limits to this picture:

A historical original-paper reference is J. W. Lindeberg's "Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung" (1922), published in Mathematische Zeitschrift 15, pp. 211–225. The Lindeberg condition and Lindeberg-Feller central limit theorem below describe a version for sums whose summands need not share a probability distribution.^[ref]

A brief proof outline uses the characteristic function of $Y=(X_1-\mu )/\sigma$. Its expected value is zero and its variance is one, so the finite second moment givesThe Taylor expansion of the complex exponential, with its remainder controlled by $Y^2$, and the dominated convergence theorem justify this step without a third-moment assumption. Independence then gives the characteristic function of $Z_n$:The limit is the characteristic function of the standard normal distribution; the Lévy continuity theorem turns this into convergence in distribution.

Let $X_i$ be real independent and identically distributed random variables, with mean $m$, variance $v$ and finite fourth central moment ${\mu }_4=\mathbb{E}[(X_i-m{)}^4]$. For the sample variance $S_n^2=(n-1{)}^{-1}{\sum }_i(X_i-{\overline{X}}_n{)}^2$, the displayed convergence in distribution holds.

Indeed $n^{-1}{\sum }_i(X_i-{\overline{X}}_n{)}^2=n^{-1}{\sum }_i(X_i-m{)}^2-({\overline{X}}_n-m{)}^2$. The central limit theorem for the sample mean implies that $\sqrt{n}({\overline{X}}_n-m{)}^2$ tends to zero in probability. Changing the divisor from $n$ to $n-1$ also has negligible effect at this scale, by the law of large numbers. Apply the central limit theorem to the independent variables $(X_i-m{)}^2$, whose variance is ${\mu }_4-v^2$, and use Slutsky's theorem. Standardization gives a standard normal distribution limit when ${\mu }_4>v^2$. The familiar special case $2v^2$ holds for the normal distribution because then ${\mu }_4=3v^2$.

Let $X_1,X_2,\dots$ be independent and identically distributed random variables taking $-1$ and $1$ with equal probabilities, the Rademacher distribution. The two opposite complex exponentials give ${\phi }_X(t)=(e^{it}+e^{-it})/2=\cos t$. Their expected value is zero and their variance is one. For $S_n={\sum }_{j=1}^n{X}_j$, the characteristic functions of its sample mean and its variance normalization are respectivelyFor each fixed $t$, the bases are positive for all sufficiently large $n$. The Taylor expansion $\log \cos u=-u^2/2+O(u^4)$ at zero then givesExponentiating proves the two displayed pointwise limits. The first limit is the characteristic function of the constant zero; the second is that of the standard normal distribution. The Lévy continuity theorem gives the corresponding convergence in distribution statements. Averaging has variance $1/n$, whereas dividing the sum by $\sqrt{n}$ preserves variance one. This sign calculation illustrates why the law of large numbers and the central limit theorem use different scales; it is not a proof for arbitrary probability distributions.

For a real random variable $X$ with finite variance ${\sigma }^2$, expected value $\mu$, and a threshold $a>0$, Chebyshev's inequality givesOne can also cap this upper bound at one, since a probability cannot exceed one. If $\sigma >0$, taking $a=k\sigma$ gives $\mathbb{P}(|X-\mu |\ge k\sigma )\le 1/k^2$. For example, at least $3/4$ of the probability lies strictly within two standard deviations of the mean. This statement needs only the finite variance assumption. If the variance is zero, $X=\mu$ almost surely and every positive deviation threshold has zero tail probability.

Consider a random variable uniformly distributed on $\{-2,-1,0,1,2\}$. Its expected value is zero and its variance is $(4+1+0+1+4)/5=2$. At $a=2$ the actual tail probability is $2/5$, while Chebyshev's inequality gives $2/2^2=1/2$. At $a=3$ the actual tail probability is zero, while the bound is $2/9$. A valid bound need not describe the exact shape or bounded support of a particular probability distribution.

For the standard normal distribution, the actual tail probabilities beyond two and three standard deviations are approximately $0.0455003$ and $0.0026998$, whereas the general bounds are $1/4$ and $1/9$. More knowledge of the probability distribution can therefore improve the estimate. The comparison does not diminish the guarantee: the three-point example in equality in Chebyshev's inequality attains it exactly.

The proof is a one-line use of Markov's inequality on the nonnegative random variable $Y=(X-\mu {)}^2$. The event $|X-\mu |\ge a$ equals $Y\ge a^2$, and $\mathbb{E}Y={\sigma }^2$. This explains why the denominator is squared and why a finite variance supplies the necessary information. ^[ref].

One important use is control of a sample mean. For independent and identically distributed random variables with finite variance ${\sigma }^2$, the variance of a sample mean is ${\sigma }^2/n$. HenceFor every fixed tolerance, the bound tends to zero as the sample size grows, proving the finite-variance case of the weak law of large numbers. For ${\sigma }^2=1$ and $\epsilon =0.1$, $n=2000$ makes the bound $0.05$. This is a sufficient bound under the stated assumptions, rather than an optimal required sample size.

An early historical source is Pafnuty Chebyshev's 1867 paper “Des valeurs moyennes”, published in French translation by N. de Khanikof. Its opening theorem bounds deviations of sums using their expected values and second moments, linking averages with control of exceptional outcomes. ^[ref].

Under sparsity, compatibility, and inverse-Gram estimation conditions, the standardized coordinatewise error of the Debiased Lasso converges in distribution to a standard normal distribution.

For a normal distribution with mean $m$ and positive variance ${\sigma }^2$, expand the logarithm of its probability density function:The second term has expected value $1/2$, hence $h_e(X)=\frac{1}{2}\ln (2\pi e{\sigma }^2)$. Translation of the mean has no effect, while shrinking the standard deviation lowers differential entropy. A standard normal distribution has approximately $1.41894$ nats. This connects the continuous spread calculation with entropy power, whose normalization matches a Gaussian distribution's variance.

For a real strongly additive arithmetic function with uniformly bounded values on primes, put $A(x)={\sum }_{p\le x}f(p)/p$ and $B(x{)}^2={\sum }_{p\le x}f(p{)}^2/p$. If $B(x)\to \infty$, then $(f(n)-A(N))/B(N)$ under uniform sampling from $[N]$ converges to the standard normal distribution. For the prime omega function, the mean and variance scales are $\log \log N$.

An estimation error is the difference between an estimator and the statistical parameter it estimates. It is a random variable before observing the sample. Its mean is the bias of the estimator, and its mean square is the sum of the variance and squared bias. Standardizing an estimation error by its standard error often yields an approximate standard normal distribution, allowing construction of a Wald confidence interval.

Excess kurtosis subtracts $3$ from kurtosis, making the normal distribution baseline zero. The non-excess convention is sometimes called raw kurtosis; this still uses a fourth central moment, not the uncentered fourth moment $\mathbb{E}[X^4]$. Software may report either kurtosis or excess kurtosis, so the convention must be stated.^[ref]

For a standard normal distribution, the moment-generating function of a standard normal variable gives $\mathbb{E}[Z^4]=3$, hence excess kurtosis zero. For a continuous uniform distribution on $[-\sqrt{3},\sqrt{3}]$, the mean is zero, the variance is one, and direct integration givesThe rare-outlier three-point distribution can instead have kurtosis $100$ and excess kurtosis $97$ with mean zero and variance one. These are calculated properties of specified probability distributions, rather than a measured statistic.

For $X=\mu +\sigma Z$ with $Z$ having the standard normal distribution and an increasing concave utility function $U$, define $f(\mu ,\sigma )=\mathbb{E}[U(X)]$. Then $f_{\mu }=\mathbb{E}[U^{\prime }(X)]>0$ and Gaussian integration by parts gives $f_{\sigma }=\sigma \mathbb{E}[U^{\prime \prime }(X)]\le 0$. Moreoveris negative semidefinite, so $f$ is jointly concave in $\mu$ and $\sigma$.

For $Z$ with a standard normal distribution and $a\ge 0$, split the Gaussian integral at zero and complete the square to obtain $\mathbb{E}{e}^{a|Z|}=2e^{a^2/2}\Phi (a)\le 2e^{a^2/2}$, where $\Phi$ is the standard normal distribution function. This converts a bound on an absolute Gaussian random variable into a two-sided Gaussian tail bound using Markov inequality.

For times $t_1,\dots ,t_n$, the corresponding finite-dimensional distribution of a stochastic process $X$ is the joint probability distribution of the random vector $(X_{t_1},\dots ,X_{t_n})$. The collection of these distributions records every finite set of coordinates of the process.

Knowing only the one-time marginal distributions is insufficient. Let $X_n$ be independent standard normal random variables, and let $Y_n=Z$ for every $n$, where $Z$ is one standard normal random variable. Every $X_n$ and $Y_n$ has the same standard normal distribution. Nevertheless, for $m\ne n$, the difference $X_n-X_m$ has a normal distribution with variance two, whereas $Y_n-Y_m=0$. The joint probability distribution of $(Y_m,Y_n)$ is concentrated on the diagonal; that of $(X_m,X_n)$ is spread across the plane. A model for tomorrow's value therefore needs the relation between observations as well as their separate distributions.

For a real-valued stochastic process, its finite-dimensional distributions determine its probability law on the product sigma-algebra. With an uncountable index set, a path property such as continuity need not be measurable in that sigma-algebra; specifying a suitable space of sample paths and a regular modification of a stochastic process is a separate step. This distinction is explained in MIT, The basics of stochastic processes, section 1.1.

Here is an original reproducible finite set with 40 points and 512 coordinates. Let $e_1,\dots ,e_4$ be the first four vectors of the standard basis of ${\mathbb{R}}^{512}$. With point indices $i=0,\dots ,39$, generateThe four groups have different centers, and the standard normal random variables perturb all 512 coordinates. Generate the coordinates in row order using NumPy's PCG64 pseudorandom number generator with seed 20261013. Then keep this finite set fixed. A second PCG64 stream with seed 424242 generates a $32\times 512$ matrix of independent random variables with the standard normal distribution; divide it by $\sqrt{32}$ to obtain the Gaussian Johnson–Lindenstrauss transform.

For all 780 unordered pairs, compute the distance ratio $r_{ij}=\Vert T{x}_i-T{x}_j{\Vert }_2/\Vert x_i-x_j{\Vert }_2$. In this draw, its minimum is approximately $0.733010$ and its maximum $1.478785$, giving a maximum relative error in Euclidean distance of about $0.478785$. The squared-distance ratios $r_{ij}^2$ instead range from approximately $0.537304$ to $2.186806$: their maximum relative error is about $1.186806$. Thus this particular draw fails a 20% distance tolerance. The Johnson–Lindenstrauss lemma does not assert that every draw into 32 dimensions must succeed.

The two bands of original Euclidean distances distinguish pairs within a group from pairs between groups. Their spread after the linear map shows why checking an average distance is insufficient: the Johnson–Lindenstrauss lemma controls every pair. The dashed 20% distance band is an illustrative tolerance, not a confidence interval or a squared-distance band. The affine hull of 40 points has dimension at most 39; 512 is the ambient coordinate count, and no full-dimensionality assumption is required. For this draw the bi-Lipschitz distortion is approximately $1.478785/0.733010=2.017414$.

Now keep the same points and draw 48 fresh Gaussian Johnson–Lindenstrauss transforms for each target dimension $k\in \{8,16,32,64,128\}$. NumPy's SeedSequence(314159).spawn(240) supplies distinct PCG64 streams, ordered first by target dimension, then by draw. Each stream generates a new $k\times 512$ random matrix, normalized by $1/\sqrt{k}$; no map is a row prefix of another map. Each plotted value is

The observed median maximum relative errors in Euclidean distance are approximately $0.735$, $0.520$, $0.381$, $0.267$, and $0.184$, respectively. All pairs fall inside the 20% distance tolerance in $0$, $0$, $0$, $0$, and $35$ of the 48 draws, respectively. These are computations on one synthetic finite set, rather than optimal dimension bounds or measured physical data. The displayed quantiles describe the sampled maps; they do not certify a success probability for other sets. Even at 128 dimensions, 13 of the sampled maps fail this tolerance.

The practical benefit is that each point acquires fewer coordinates while its pairwise Euclidean distances can remain useful. The choice of target dimension balances compression, accuracy, and failure probability. For example, a conservative sufficient squared-distance tolerance $\epsilon =0.2$ with $n=40$ and $\delta =0.05$ gives $k\ge 1195$ by the explicit bound above. Here that exceeds the ambient dimension, so the identity map is better for a certified guarantee. The much smaller dimensions in the plots illustrate observed behavior without claiming that this sufficient bound is sharp.

Here take $X,Y$ iid, centered with variance one, and with finite moment-generating functions at all real arguments. Independence of their sum and difference gives $M(2t)=M(t{)}^{3}M(-t)$. Dividing the positive and negative argument identities shows $M(t)/M(-t)$ satisfies the symmetry step of dyadic rigidity of a moment-generating function, hence $M$ is even. The identity becomes $M(2t)=M(t{)}^4$, and the unit-variance expansion then forces $M(t)=e^{t^2/2}$. The uniqueness theorem for moment-generating functions identifies the standard normal distribution. This is a precise sufficient-hypothesis version; the displayed implication is understood with those stated assumptions.

In a systematic Gibbs sampler for a standardized bivariate normal distribution with correlation $\rho$, update $Z_1$ conditionally on the old $Z_2$, then update $Z_2$ using the new $Z_1$. Substitution gives the displayed autoregressive process of order one, with independent noises with the standard normal distribution ${\eta }_{1,r},{\eta }_{2,r}$. The sweep-to-sweep autocorrelation of $Z_2$ at lag $h$ is ${\rho }^{2h}$, so the chain mixes slowly as $|\rho |$ approaches one.