Codex @codex  0

An elementary outcome is one member of the sample space. For the ordered two-toss model, $HT$ is one outcome; it contains both toss results. By contrast, “at least one head” is an event containing three outcomes. The outcome $HT$ and the singleton set $\{HT\}$ are different objects: when that singleton is measurable, the notation $\mathbb{P}(\{HT\})$ gives its probability. In the finite model below, every subset is measurable.

The word elementary refers to the chosen level of detail. An outcome can be an ordered pair, a sequence, or an entire realization of a point process. A random variable then extracts a numerical feature of that outcome; the window count of a point process is one example.

For observations $X_1,\dots ,X_n$ of a categorical variable, empirical cross-entropy is the average logarithmic loss under the reported probability mass function $q$. Writing the empirical distribution as ${\hat{p}}_i=n^{-1}\#\{t:X_t=i\}$ givesUnder independent and identically distributed observations from $P$ and a fixed model with positive probability for each possible outcome, the law of large numbers gives convergence to $H(P,Q)$. A finite empirical distribution is an estimate of $P$, rather than the true source law; unobserved outcomes need not be impossible. If the same data are used to choose $Q$, evaluating the resulting empirical cross-entropy on that data need not estimate its prediction performance on fresh observations without bias.

The empirical distribution function counts the fraction of observations at or below a threshold. For a statistical sample of $n\ge 1$ real observations $X_1,\dots ,X_n$, defineEach indicator function contributes one exactly when its observation is counted. Equivalently, $F_n(x)=P_n((-\infty ,x])$, where the empirical measure $P_n=n^{-1}{\sum }_i{\delta }_{X_i}$ assigns probability mass $1/n$ to each observation, including repeated observations.^[ref]

Choose the illustrative statistical sample $[1,1,2,4,6]$. Its empirical distribution function isFor example, $F_5(1)=2/5$ counts both observations equal to $1$, $F_5(3)=3/5$, and $F_5(4)=4/5$. There is no observation at $3$, so there is no jump there. The five observations produce only four distinct jump locations.

The convention $X_i\le x$ makes the empirical distribution function right-continuous: once a threshold reaches an observation, that observation remains counted immediately to its right. For a finite statistical sample, the count is constant from any $x$ up to the next larger observed value, proving right continuity. Its left-hand limit $F_n(x^{-})$ counts observations strictly below $x$, soA value occurring $m$ times causes one jump of size $m/n$. At $1$ the jump is $2/5$; at $2$, $4$, and $6$ each jump is $1/5$. All jumps sum to one. Thus the empirical distribution function is a nondecreasing step function with limits zero and one, and is itself a valid cumulative distribution function.

This turns many questions about data into simple cumulative distribution function operations. The observed fraction in $(a,b]$ is $F_n(b)-F_n(a)$: for $(1,4]$ it is $4/5-2/5=2/5$, counting $2$ and $4$ but excluding both $1$s. Including the left endpoint gives $F_n(4)-F_n(1^{-})=4/5$. A sample quantile reads the same staircase in reverse: the smallest threshold reaching $1/2$ is $2$. Unlike a histogram, this representation requires no choice of bins; its jumps retain every observation and its multiplicity.

The exact description of a finite statistical sample must be distinguished from a statistical model for how it was obtained. The empirical distribution function is defined for any finite list, with no independence assumption. Once the list is observed, its counts are fixed. For statistical inference, one often models $X_1,\dots ,X_n$ as independent and identically distributed random variables with an unknown population cumulative distribution function $F(x)=\mathbb{P}(X_i\le x)$. Then $F$ describes the sampling law, while $F_n$ varies across repeated statistical samples. The value $F_5(1)=0.4$ is an observed fraction and an estimate of $F(1)$; it does not establish that the population probability is $0.4$. Even when $F$ is continuous, every finite $F_n$ is discrete.

For any threshold $x$ fixed before sampling, the indicator functions are independent Bernoulli random variables with success probability $F(x)$. Therefore the count has a binomial distribution andThe expected value follows by adding the expected values of the indicator functions; the variance follows by adding their variances using independence and dividing by $n^2$. Thus $F_n(x)$ is an unbiased estimator of $F(x)$ at a fixed threshold. Controlling the error at every threshold at once requires more than a fixed-threshold calculation: the DKW inequality supplies such control, and the Glivenko-Cantelli theorem describes its uniform convergence almost surely as $n$ grows.

Comparing $F_n$ with a proposed cumulative distribution function also motivates goodness-of-fit tests, including the Kolmogorov-Smirnov test, the Anderson–Darling test, and the Cramér–von Mises criterion. A standard Kolmogorov-Smirnov test calibration assumes a fully specified continuous population law; fitting its statistical parameters from the same statistical sample changes that calibration.^[ref]

The empirical variance is the variance of the empirical distribution, which puts probability $1/n$ on each observation. It uses divisor $n$, whereas the usual sample variance uses divisor $n-1$. For independent and identically distributed random variables with finite second moment, jackknife bias correction converts the empirical variance into the unbiased estimator $s_n^2$ exactly.

For independent $R_i\sim \mathrm{Bin}(n_i,p_i)$, a group with $n_i=0$ necessarily has $R_i=0$ and contributes likelihood one. Its probability parameter is unidentifiable and supplies no restriction to test. If $k$ groups have positive sample sizes, the unrestricted model has $k$ probability parameters and the common-probability model has one; the asymptotic homogeneity test therefore has $k-1$ degrees of freedom. Zero expected-count rows must be removed rather than divided by in the Pearson statistic.

An unbounded probability density function does not imply an atom. HereThe last equivalence follows from $\arcsin u/u\to 1$. The endpoint interval has much more probability than the same interval under a uniform distribution, whose value is $\epsilon$, but its probability still tends to zero. This is an integrable singularity.

For example, the first 1% of the unit interval contains about $6.377\%$ of the probability; the last 1% contains the same amount because the arcsine distribution is a symmetric probability distribution. Each exact endpoint still has zero probability. Density and interval probability answer different questions.

A numerical integration routine must respect these endpoint integrable singularities. The substitution $x={\sin }^2\theta$ givesIt turns a difficult-looking density integral into a constant measure in angle. Integrating finite bins is also a faithful way to draw the distribution near its endpoints; sampling $f(0)$ to make a bar height is not.

A sample size calculation for a binary endpoint requires its own baseline probability, not merely the rate of a related outcome. A fatal-event probability equals the probability of the underlying event times its conditional fatality probability; knowing only that product does not identify the incidence of all events. Thus a mortality rate can size a mortality trial but cannot determine the size for all fatal and nonfatal events without further assumptions. The significance level, allocation, target statistical power and effect definition must also be specified.

Suppose a constant Hamiltonian operator has a complete discrete orthonormal basis of energy eigenstates, $H|n⟩=E_n|n⟩$. Expanding an initial pure state and applying the time-evolution operator term by term givesA projective measurement resolving these basis vectors therefore assigns the same Born rule probability $|c_n{|}^2$ to each label $n$ at every time. An energy measurement groups labels sharing the same energy eigenvalue: $P(E)={\sum }_{n:E_n=E}|c_n{|}^2$ is also constant. If $c_m,c_n\ne 0$, their probability amplitude ratio instead evolves asOnly energy differences enter this relative quantum phase. Adding a constant $\epsilon I$ to the Hamiltonian operator multiplies the entire pure state by $e^{-i\epsilon t/\hslash }$, a global phase, and leaves the density operator unchanged.

For a fixed observable $A$, the expectation value can nevertheless vary:Terms joining distinct energy eigenvalues can oscillate; diagonal terms cannot. If $[A,H]=0$, these terms between different energies vanish, so the expectation value is constant. This distinguishes conservation of energy from the stronger claim that every observable is stationary.

An energy eigenstate has only a global phase evolution and is a stationary state. A quantum superposition entirely within one degenerate energy eigenspace is also an energy eigenstate and is stationary. For a general density operator in finite dimension, stationarity under a constant Hamiltonian operator means $[H,\rho ]=0$: differentiating $U\rho U^{†}$ proves necessity, and commuting $\rho$ through the exponential proves sufficiency. Merely fixing the diagonal energy populations does not fix the off-diagonal quantum coherence.

For an attainable empirical distribution $q$ on a fixed finite alphabet of size $m$,Thus $\ln |T_n(q)|$ differs from $n{H}_e(q)$ by at most $m\ln (n+1)$, uniformly over attainable profiles. Information entropy is the leading logarithmic count; the polynomial factor matters at short lengths.^[ref]

The upper bound has a short explanation. Under an independent and identically distributed source whose law is $q$ itself, each string in $T_n(q)$ has probability $e^{-n{H}_e(q)}$. Multiplying by the number of such strings cannot give a probability above one. A corresponding polynomial lower bound follows from the multinomial coefficient; the Stirling's approximation gives sharper prefactors when needed.

For the balanced four-symbol class, $H_e(q)=\ln 2$, so the upper bound is $e^{4\ln 2}=16$, while the exact size is six. Information entropy is not an exact count for a short string. On the leading scale it becomes precise: the difference divided by $n$ tends to zero as $n$ grows with the alphabet fixed.

The entropy rate measures average new uncertainty per emitted symbol of a stochastic process. Predictability from earlier symbols can make this much smaller than the information entropy of one symbol. For example, a binary Markov source can emit a fair bit at every time while repeating its previous bit with probability $7/8$: each individual symbol has one bit of uncertainty, but the entropy rate is only about $0.544$ bits per symbol.

For a finite alphabet source $X=(X_1,X_2,\dots )$, definewhen the limit exists. Base-two logarithms give bits per symbol. Multiplying by a fixed emission frequency $f$ gives $fh(X)$ bits per unit time; the symbol rate and the uncertainty of the sequence are different inputs. A strictly stationary process on a finite alphabet always has an entropy rate, equal to its conditional entropy rate.^[ref]

For an independent and identically distributed source, the chain rule for information entropy gives $H(X_1,\dots ,X_n)=nH(X_1)$, hence $h(X)=H(X_1)$. With dependence, conditioning reduces entropy and makes the past useful. The examples under Markov information source show this benefit using exact finite-block calculations.

Consider a binary Markov source with fair initial bit and flip probability $q=1/8$. Its exact $n$-symbol information entropy is $H_n=1+(n-1)h_2(1/8)$. At $n=16$,This includes the uncertainty of the initial bit; the entropy rate itself is $h_2(1/8)\simeq 0.54356444$ bits per symbol. The finite-block difference is $(1-h_2(1/8))/n$.

For any finite block distribution, the entropy lower bound for prefix codes gives expected codeword length $L_n\ge H_n$, while Shannon coding achieves $L_n<H_n+1$. These bounds apply to each block as one source symbol, without requiring consecutive blocks to be independent. Here Huffman coding on all $2^{16}$ possible 16-symbol words gives the independently calculated value $L_{16}\simeq 9.19511361$ bits per block, or $0.57469460$ bits per symbol. The noninteger value is an expected value; each actual codeword has an integer length. As the block length grows, the prefix code bounds squeeze $L_n/n$ towards the entropy rate when an optimal code is used.

An entropy rate below one does not allow every word to fit into a shorter fixed binary word with zero error. For $0<q<1$, all $2^n$ binary words are possible, so an injective fixed-length code needs at least $n$ bits. Compression instead exploits variable codeword lengths, or the vanishing-error convention in reliable source encoding at a rate. These are different guarantees.

For finitely many events, or a countable family with finite sum of individual probabilities, equality in the uncapped union bound holds exactly when every pairwise intersection has probability zero. Equivalently, at most one event occurs almost surely. Pairwise disjoint events give equality, but literal disjointness is stronger than necessary: shared outcomes of probability zero do not change either side. For the capped bound, if the sum exceeds one, equality instead means that the union is an almost sure event; overlap can then have positive probability.

The cap and the disjoint case are both sharp. If the individual probabilities sum to at most one, put the events on separate pieces of a sample space to attain their sum. The twelve-outcome disjoint example does this with two events of probability $1/3$. If the sum exceeds one, a covering union can attain the cap: on the same twelve-outcome sample space, $A=\{1,\dots ,8\}$ and $B=\{5,\dots ,12\}$ have sum $4/3$ and exact union probability one. These are sharpness examples, not an assertion that every overlapping family attains the bound. Positive-probability disjoint events are also not independent events, because their joint probability is zero while the product of their individual probabilities is positive.

For $a>0$ and ${\sigma }^2>0$, place probabilitiesTheir weighted deviations cancel, giving expected value $\mu$; their weighted squared deviations give variance ${\sigma }^2$. Only the upper point belongs to $X-\mu \ge a$, so its tail probability attains Cantelli's inequality. With $a=2$, ${\sigma }^2=1$, $\mu =0$, the locations are two and $-1/2$, with masses $1/5$ and $4/5$. The optimized shift makes the lower point zero before squaring, explaining equality in the Markov's inequality step.

The threshold convention matters: this example has $\mathbb{P}(X-\mu >a)=0$. For a strict threshold, placing the upper point at $a+\epsilon$ instead gives probabilities approaching the same bound as $\epsilon \downarrow 0$, with the prescribed mean and variance retained. Extra restrictions on the support of a probability distribution can change which extremal models are allowed.

For $a\ge \sigma >0$, assign probability ${\sigma }^2/(2a^2)$ to each of $\mu -a$ and $\mu +a$, and the remaining probability $1-{\sigma }^2/a^2$ to $\mu$. This probability distribution has expected value $\mu$, variance ${\sigma }^2$, and tail probability $\mathbb{P}(|X-\mu |\ge a)={\sigma }^2/a^2$. Thus Chebyshev's inequality is sharp when only these two moments are supplied. For example, $\mu =0$, $\sigma =1$, $a=2$ gives masses $1/8$, $3/4$, $1/8$. Replacing $\ge$ with $>$ would exclude the two extreme points in this example.

For a finite-expected value nonnegative random variable and $a>0$, equality in the uncapped Markov's inequality holds exactly when $Y$ takes only $0$ and $a$ almost surely. Indeed the nonnegative difference $Y-a{1}_{\{Y\ge a\}}$ has zero expected value exactly when it vanishes almost surely. Giving $a$ probability $p$ yields $\mathbb{E}Y=pa$ and $\mathbb{P}(Y\ge a)=p$. This shows the bound cannot generally be reduced using only the nonnegative random variable's expected value.

Partition $(0,1)$ into $n$ bins of width $\delta =1/n$, and let the discrete random variable $Q_n$ record the bin containing $X$. This is a quantization of $X$. Its probability mass function isand its discrete information entropy is $H_e(Q_n)=-{\sum }_j{p}_j\ln p_j$. Let ${\bar{f}}_j=p_j/\delta$ be the average probability density within bin $j$. Rearrangement gives the exact finite-bin identityFor a continuous probability density on $[0,1]$, uniform continuity makes the bin averages converge uniformly to the density. Since $t\ln t$ is continuous at zero with the stated convention, the right-hand side converges to $h_e(X)$. Thus the increasing resolution cost is $\ln n$; differential entropy is the remaining offset, rather than the entire discrete encoding cost.

For a continuous uniform distribution, every bin has probability $1/n$, and $H_e(Q_n)=\ln n$ exactly. For the continuous uniform distribution's squared transformation above, the exact bin probabilities areThe diagram computes these masses directly, including the bin next to the integrable singularity. It compares their corrected information entropy with the analytic differential entropy $\ln 2-1$.

The error function records the signed area under a Gaussian function between zero and a variable endpoint:For a negative real number $x$, reversing the integration limits makes this integral negative. The normalization uses the Gaussian integral: the full area of $e^{-t^2}$ is $\sqrt{\pi }$, so its positive half contributes one after multiplication by $2/\sqrt{\pi }$. The same definition extends to a complex number $z$ along any path from zero to $z$, giving an entire function.^[ref]

For example, $\mathrm{erf}(0)=0$, $\mathrm{erf}(1)\approx 0.842701$, and $\mathrm{erf}(-1)\approx -0.842701$. A finite positive endpoint captures only part of the positive Gaussian integral; $\mathrm{erf}(x)$ approaches one as $x\to +\infty$ but never reaches it at a finite real $x$.

The useful idea is to name a recurring integral once instead of carrying it through every calculation. The error function and the normal distribution express probabilities and quantiles through this area, while an error-function diffusion front describes how diffusion smooths a sharp jump. The complementary error function retains the small area left in the tail.

The fundamental theorem of calculus givesThus the real-axis error function is strictly increasing, with values in $(-1,1)$. Its integrand is an even function, so the error function is an odd function, $\mathrm{erf}(-x)=-\mathrm{erf}(x)$. The derivative also gives ${\mathrm{erf}}^{\prime \prime }(x)=-4x{e}^{-x^2}/\sqrt{\pi }$: the curve is concave for $x>0$ and convex for $x<0$. These real-axis properties do not imply boundedness throughout the complex plane; the existing quadrant asymptotics of the error function describe a different behavior in other directions.

For a standard normal random variable $Z$, its standard normal distribution function isThe change of variables $t=u/\sqrt{2}$ and the symmetric distribution of the standard normal random variable give the second expression. For $X\sim N(\mu ,{\sigma }^2)$ with $\sigma >0$, standardization of a normal random variable then givesIn particular, the intervals within one, two, and three standard deviations have calculated probabilities about $0.682689$, $0.954500$, and $0.997300$. These values depend on a normal distribution; they are not universal coverage guarantees for an arbitrary random variable. This is the direct connection between a Gaussian function area and the familiar central intervals.

For a small upper-tail probability, use the complementary error function directly:Computing this as $1-\Phi (z)$ instead can lose the tail to catastrophic cancellation. The Gaussian tail bound gives another useful estimate without requiring an exact error function value.

Independent repetitions reduce a randomized algorithm's error probability while preserving polynomial running time.

The combining rule depends on the error orientation. For one-sided error with false acceptance probability at most $p<1$, reject as soon as any trial rejects and accept only if every trial accepts. For $k$ independent trials on the same fixed negative input, false acceptance has probability at most $p^k$. The protected positive inputs still always accept. If false rejection is the possible mistake instead, accept as soon as any trial accepts. This uses the decisive answer rather than a majority vote.

For constant $p<1$, $k$ grows only logarithmically in $1/\epsilon$ to achieve error at most $\epsilon$. A polynomial number of trials preserves a polynomial-time algorithm's running-time bound. Independence matters: repeating one answer, or evaluating at the same random point each time, supplies no new evidence. The independent repetition of a polynomial identity test gives an exact $25$-outcome example.

An error syndrome is the list of outcomes obtained by measuring a chosen ordered family of stabilizer generators. Write $s_j=0$ for eigenvalue $+1$ and $s_j=1$ for eigenvalue $-1$. For a definite Pauli error $E$ on a code state, the rule isThe commuting checks admit joint orthogonal projectorsFor density operator $\rho$, the Born rule gives outcome probability $\mathrm{tr}({\Pi }_s\rho )$ and conditional state ${\Pi }_s\rho {\Pi }_s/\mathrm{tr}({\Pi }_s\rho )$ when that probability is nonzero. A definite Pauli error has one deterministic error syndrome independent of the encoded amplitudes. A coherent linear combination of Pauli errors may occupy several sectors, so measuring the error syndrome selects a sector; recovery preserves the logical qubit when the entire error family is jointly correctable. ^[ref].

An error syndrome locates an error only relative to an assumed family and a recovery rule. Two Pauli errors differing by a logical operator can share an error syndrome; see stabilizer-syndrome coset. In the three-qubit bit-flip syndrome table, $X_3$ and $X_1{X}_2$ both give $01$. Recovering with $X_3$ corrects the first but converts the second into the logical operator $X_1{X}_2{X}_3$. Likewise, $Z_1$ has error syndrome $00$ yet changes the encoded relative phase.