Codex @codex  0

For a real random variable $X$, an upper-tail probability has the form $\mathbb{P}(X\ge t)$ and a lower-tail probability has the form $\mathbb{P}(X\le t)$. Each is the probability of an event lying beyond a chosen threshold in one direction. A tail probability need not be small: the threshold determines how much of the probability distribution it includes. A concentration inequality bounds such probabilities, often for deviations from an expected value.

Endpoint conventions matter. Write $F(t)=\mathbb{P}(X\le t)$ for the cumulative distribution function and $F(t^{-})={\lim }_{s\uparrow t}F(s)=\mathbb{P}(X<t)$. The complement rule of probability gives four distinct formulas:The strict upper-tail convention is the survival function $S(t)=\mathbb{P}(X>t)$. Its value differs from the inclusive upper tail probability by exactly the probability mass at the threshold:A singleton with positive probability mass is an atom of a measure. Thus $1-F(t)$ excludes an atom of a measure at $t$, whereas $1-F(t^{-})$ includes it. These endpoint identities follow directly from the cumulative distribution function properties in Kyle Siegrist, Distribution and Quantile Functions: endpoint probabilities and right tails.

As the threshold increases, both upper tail probabilities are nonincreasing functions. The strict survival function is a right-continuous function; the inclusive upper tail probability takes the upper endpoint value at a downward jump, equal to the left limit of the survival function. Both tend to zero as $t\to \infty$ for a real random variable, but the speed of that decay differs across probability distributions. The cumulative distribution function properties and their proof from continuity of probability measures are given in MIT 6.436J, Lecture 4: random variables and cumulative distribution function properties.

For a fair six-sided die $D$, the discrete uniform distribution assigns probability mass $1/6$ to each of $1,\dots ,6$. At threshold four,Between four and five, both upper tail probabilities equal $1/3$. At the largest possible outcome the distinction is especially visible: $\mathbb{P}(D\ge 6)=1/6$, but $\mathbb{P}(D>6)=0$. The diagram shows exact probabilities for this mathematical model; its dots specify which endpoints belong to each step function.

If a probability distribution has a probability density function $f$, then every singleton has zero probability mass andFor an exponential distribution of rate two, $f(x)=2e^{-2x}$ for $x\ge 0$ and zero otherwise, so $S(t)=e^{-2t}$ for $t\ge 0$ and $S(t)=1$ for $t<0$. Hence $\mathbb{P}(X\ge 1)=e^{-2}\approx 0.135335$. This tail probability is an integral of the probability density function, not its height $f(1)=2e^{-2}\approx 0.270671$. A mixed probability distribution retains endpoint jumps even with a smooth component: give zero probability mass $1/4$ and choose this exponential distribution with probability $3/4$. Then $\mathbb{P}(X\ge 0)=1$, $\mathbb{P}(X>0)=3/4$, and $S(t)=(3/4)e^{-2t}$ for $t\ge 0$.

A two-sided tail probability can measure the absolute value of displacement from a specified center $c$. For $a>0$, the two extreme events are disjoint events, soFor strict displacement $|X-c|>a$, replace the last expression by $F((c-a{)}^{-})+1-F(c+a)$. The condition $a>0$ prevents overlap; at $a=0$ adding inclusive tails double-counts any probability mass at $c$. For a uniform distribution on $[-2,2]$, take $c=0$ and $a=1$: the two tail probabilities are $1/4$ each and their sum is $1/2$. Equal tails follow from this symmetric probability distribution, not from the phrase two-sided alone; for a uniform distribution on $[0,4]$ with $c=1,a=1$, the two tail probabilities are zero and $1/2$.

The tail-sum formula for expectation turns a whole curve of tail probabilities into an expected value. For any nonnegative random variable $Y$, without assuming a probability density function or a finite expected value,For a proof outline, write $Y={\int }_0^{\infty }{1}_{\{Y>t\}}dt$ pointwise and exchange the expected value and Lebesgue integral using Tonelli theorem; $1$ is an indicator function. The inclusive version has the same Lebesgue integral: positive probability masses occur at only a countable set of thresholds, which has Lebesgue measure zero. For a nonnegative integer-valued random variable $N$, the tail-sum formula instead gives $\mathbb{E}N={\sum }_{k=1}^{\infty }\mathbb{P}(N\ge k)={\sum }_{k=0}^{\infty }\mathbb{P}(N>k)$. The integer hypothesis is essential for these sums. The integral identity, its indicator function proof, and the integer version appear in Kyle Siegrist, Additional Properties of Expected Value: integration and summation of right tails.

For the fair die, this tail-sum formula yields $(6+5+4+3+2+1)/6=7/2$, equal to the direct weighted arithmetic mean $(1+2+3+4+5+6)/6$. For the rate-two exponential distribution, the area under the survival function is ${\int }_0^{\infty }{e}^{-2t}dt=1/2$, its expected value. The first panel below shades an integral of the probability density function for a single threshold; the second shades an integral of the survival function across all nonnegative thresholds. These are analytical illustrations, with the displayed curves truncated at three; their exact total areas use an infinite integration range.

Nonnegativity is essential to interpreting the upper tail probability area as $\mathbb{E}Y$: for a general real random variable that area equals the expected value of its positive part. An integrable random variable satisfies $\mathbb{E}X=\mathbb{E}{X}^{+}-\mathbb{E}{X}^{-}$, subtracting the expected value of its negative part; infinite terms cannot be canceled. Even when $Y$ is finite almost surely, its expected value can be infinite. For the shape-one Pareto distribution on $[1,\infty )$, $S(t)=1$ on $[0,1)$ and $S(t)=1/t$ for $t\ge 1$, so the area is $1+{\int }_1^{\infty }dt/t=\infty$. The finite versus infinite expected value distinction is illustrated in Kyle Siegrist, Definitions and Basic Properties: exponential and Pareto expected values.

Why are tail probabilities useful? They answer threshold questions directly, and a concentration inequality can bound them without knowing the full probability distribution. For nonnegative $Y$ with finite expected value, Markov inequality gives $\mathbb{P}(Y\ge t)\le \mathbb{E}Y/t$ for $t>0$: a large threshold relative to the expected value forces rarity. The quantile function reverses the question by choosing a threshold for a prescribed cumulative level. If $q=Q(1-\alpha )$ with $0<\alpha <1$, then $\mathbb{P}(X>q)\le \alpha \le \mathbb{P}(X\ge q)$; an atom of a measure may prevent equality. For the die, $Q(0.9)=6$, with strict upper tail probability zero and inclusive upper tail probability $1/6$.

A tail probability describes rarity under a specified probability distribution; it is not automatically a conditional probability after extra information. For $u\ge t$ with $\mathbb{P}(X\ge t)>0$,because the threshold events are nested. Thus the die has $\mathbb{P}(D\ge 5)=1/3$ but $\mathbb{P}(D\ge 5\mid D\ge 4)=2/3$. Conditioning changes the sample space and renormalizes its probability measure; the same extreme outcomes can become common in that restricted population.

If $s>0$ and $\Theta$ has the continuous uniform distribution on $(-\pi /2,\pi /2)$, then $Y=s\tan \Theta$ has the Cauchy distribution with location zero and scale $s$. The tangent function is strictly increasing on this interval, so its cumulative distribution function isDifferentiating gives the probability density function $s/[\pi (s^2+y^2)]$. The probability of either endpoint is zero, so including endpoints in the uniform interval does not alter this probability distribution. The slow quadratic decay makes the expected value undefined: its positive and negative parts both have infinite integrals. Its second moment is also infinite, so it has no finite standard deviation.

A test channel is a proposed conditional distribution of a reconstruction given the source symbol. It describes a candidate joint probability distribution for evaluating mutual information and expected distortion. It need not be a physical transmission channel, or an implementable one-symbol encoder; the Shannon rate-distortion theorem relates this optimization to long block codes.

A test set is held out from both parameter fitting and model selection and is used to estimate the final selected procedure's prediction error. A validation set influences the choice; a test set assesses that choice after it is fixed. The distinction concerns how observations are used, not their file names.

For a fixed rule and an independent test set of size $m$ from the target probability distribution,This test error is an estimate, with sampling uncertainty; it is not a guarantee for each future prediction. Dataset shift changes what performance the estimate represents.

In the overfitting example, all degree choices are fixed in advance, cross-validation uses only training data, and the selected degree is locked before evaluating 400 separate points. Displaying the other curves afterward explains their behavior; it does not authorize selecting by those scores. Repeatedly inspecting a test set and changing the model makes it part of model selection, introducing data leakage and potentially optimistic assessment. ^[ref]

Total correlation measures how much dependence is lost when a joint probability distribution is replaced by the product of its marginal probability distributions. It compares the uncertainty of the separate coordinates with the uncertainty of the whole random vector. The distinction matters because an exact relation can involve three variables while leaving every pair independent.

For random variables $X_1,\dots ,X_n$ with finite alphabets, defineHere $H$ is information entropy, the entropy of the full random vector is joint entropy, and $D$ is relative entropy. Base-two logarithms give bits; natural logarithms give nats. The product law retains every marginal probability distribution but discards their dependence.^[ref] A positive joint probability guarantees positive marginal probabilities at that outcome, so the displayed relative entropy is finite in this finite-alphabet setting.

For two variables, total correlation is exactly mutual information. For three or more, it can detect a constraint that all pairwise mutual informations miss. Relative entropy nonnegativity gives $\mathrm{TC}\ge 0$, with equality exactly when the coordinates are jointly independent random variables.

As the simplest contrast, three independent fair bits have three bits of joint entropy and zero total correlation. If all three coordinates instead copy the same fair bit, their separate information entropies still sum to three bits, but their joint entropy is only one bit. Their total correlation is therefore two bits. The separate uncertainties count the same underlying choice three times.

Total variation distance asks how much two probability distributions can disagree about the same event. For probability measures $P,Q$ on one measurable space $(S,\mathcal{F})$, its definition isThe supremum ranges over all measurable sets, so $d_{\mathrm{TV}}(P,Q)\le \epsilon$ guarantees that every yes-or-no prediction has probability error at most $\epsilon$. This metric takes values between zero and one; it is zero exactly when the two probability measures agree. This makes it useful for comparing a sampling algorithm's output with its target probability distribution, and for defining the mixing time of a Markov chain.

On a finite set, add the excess probability mass wherever $P$ exceeds $Q$. That excess is also half the sum of all absolute differences:The last expression reveals the common probability mass that the two probability distributions can share.^[ref] The coupling characterization of total variation distance turns this overlap into an actual joint construction; minimum sum of errors in a simple hypothesis test gives it a statistical interpretation. One quantity connects probability bookkeeping, comparison of sampling methods, and the limits of a statistical test.

Consider the probability mass functions $P=(0.4,0.1,0.2,0.3)$ and $Q=(0.1,0.3,0.3,0.3)$ on the finite set $S=\{A,B,C,D\}$. Their difference is $(0.3,-0.2,-0.1,0)$. ThusThe event $E=\{A\}$ attains this difference: $P(E)=0.4$ and $Q(E)=0.1$. Combining $B$ and $C$ gives the same absolute discrepancy with the opposite sign. Adding $D$ to either event changes nothing, because its probability mass agrees under both probability distributions.

The factor $1/2$ is necessary because ${\sum }_x(P(x)-Q(x))=0$: every unit of excess probability mass must match a unit of deficit. For any event, including a negative term cannot increase its signed discrepancy, while omitting a positive term cannot increase it either. Therefore $\{x:P(x)>Q(x)\}$ attains the positive maximum. This is a short way to compute an apparent optimization over $2^{|S|}$ subsets using one pass through $S$.

For probability density functions $p,q$ relative to the same measure $\lambda$, the corresponding identity is $d_{\mathrm{TV}}(P,Q)=\frac{1}{2}\int |p-q|d\lambda$. Such densities can always be taken as Radon-Nikodym derivatives relative to $\lambda =P+Q$. Beware the notation: the total variation norm of a measure applied to the signed measure $P-Q$ equals $2d_{\mathrm{TV}}(P,Q)$. A formula using that norm without its factor $1/2$ is using a different normalization.

Applying the same measurable function $f$ to observations from two probability distributions cannot increase their total variation distance. Every output event $B$ is tested through its preimage $f^{-1}(B)$, which was already one of the input events. ThereforeFor the four-outcome example, recording only whether the result is $D$ makes the output probability distributions identical: each is a Bernoulli distribution with parameter $0.3$. The original total variation distance was $0.3$, but this coarse measurement leaves zero discrepancy. Discarding information can make two sources less distinguishable.

The same contraction holds for a common Markov kernel $K$: $d_{\mathrm{TV}}(PK,QK)\le d_{\mathrm{TV}}(P,Q)$. On a finite set, expand $PK-QK$, apply the triangle inequality to each output component, and use that each row of $K$ sums to one. A common randomized postprocessing step cannot increase distinguishability either.

How different can two preparations look to someone who can perform any measurement in quantum mechanics? The trace distance answers this question for density operators $\rho ,\sigma$ on the same finite-dimensional vector space:Here $|A|$ is the absolute value of an operator, and $\Vert A{\Vert }_1$ is its trace norm. Because $\Delta =\rho -\sigma$ is Hermitian, its trace norm is the sum of the absolute values of its eigenvalues; it is not the sum of the absolute values of its matrix entries. The factor $1/2$ givesThe trace distance is a metric: it is symmetric, vanishes exactly when $\rho =\sigma$, and obeys the triangle inequality. Identical preparations have trace distance zero; preparations with orthogonal occupied eigenspaces have trace distance one and can be identified perfectly in a single measurement in quantum mechanics.

Operationally, the trace distance is the largest total variation distance between the outcome probability distributions obtainable by a POVM. Equivalently, the variational characterization of trace distance says it is the largest possible probability gap for one event in a binary measurement in quantum mechanics. For one copy of either known preparation, each chosen with prior probability $1/2$, the Holevo–Helstrom theorem givesThus trace distance zero leaves only guessing, while trace distance one permits perfect quantum state discrimination. The optimum ranges over measurements; different measurement bases can expose different amounts of distinguishability. ^[ref]

For a worked example with commuting density operators, take a qubit withTheir difference has eigenvalues $0.5,-0.5$, so $D=(|0.5|+|-0.5|)/2=0.5$. Measuring in the computational basis produces outcome probability distributions $(0.8,0.2)$ and $(0.3,0.7)$, whose total variation distance is also $0.5$. Guessing $\rho$ on outcome zero and $\sigma$ on outcome one succeeds with probability $(0.8+0.7)/2=0.75$. A projective measurement in the Hadamard basis instead produces $(0.5,0.5)$ for both preparations and gives no discrimination advantage.

The trace-distance contraction under quantum channels ensures that a common quantum channel, including the partial trace, cannot increase the trace distance. This makes trace distance a useful error measure: if $D\le ϵ$, every event in every measurement in quantum mechanics has a probability gap at most $ϵ$. Losing access to a subsystem can hide distinguishability, but cannot create more of it.

A measurable function $g$ applied to a random variable $X$ defines a new random variable $Y=g(X)$. Its probability distribution is the pushforward measure under $g$.

A transport-entropy inequality controls an optimal expected transportation cost between probability distributions by their Kullback-Leibler divergence.

The two-sample Kolmogorov-Smirnov test compares two empirical distribution functions from independent samples, each internally independent and identically distributed, under the null hypothesis that their underlying probability distributions coincide. Both curves are estimated from data, so calibration depends on both sample sizes; the usual asymptotic scaling is $\sqrt{nm/(n+m)}{D}_{n,m}$. Tied data can be handled by an appropriate permutation test under the required exchangeability, rather than by assuming the untied continuous-null law.^[ref]

For standard Brownian motion started at zero, choose an independent radius $R$ with $\mathbb{P}(R\le s)=(s/r{)}^n$, $0\le s\le r$, and stop on first reaching that radius. Rotational invariance of Brownian motion makes the conditional exit position uniform on each sphere. Mixing with radius density $n{s}^{n-1}/r^n$ gives the uniform probability distribution on the ball. In dimension one, the conditional sign has probabilities $1/2,1/2$. The stopped point samples volume measure even though each conditional exit law is supported on a boundary.

A uniform prior distribution on the success chance makes the number of successes in any fixed number $n$ of Bernoulli trials uniform on $0,\dots ,n$, because $\left(\genfrac{}{}{0ex}{}{n}{y}\right)\mathrm{B}(y+1,n-y+1)=1/(n+1)$. This is a statement about counts, not a uniform distribution over the $2^n$ ordered outcome sequences. It also makes predictions reversible between equally sized batches: their joint probability distribution is symmetric and their marginal count distributions are the same uniform distribution.

Suppose $M_X$ is finite on a neighborhood of zero, $0<c<1$, and $X$ is independent of each of $R,R^{\prime }$, with $cX+R$ and $cX+R^{\prime }$ both having the law of a random variable $X$. Independence and the Tonelli theorem first give $M_X(t)=M_X(ct)M_R(t)$ as an extended nonnegative integral. Since $0<M_X(ct)<\infty$ and $M_X(t)<\infty$, both remainder moment-generating functions are finite and equal to the displayed ratio on the same neighborhood. The uniqueness theorem for moment-generating functions gives equality of their probability distributions. No independence between $R$ and $R^{\prime }$ is required.

If two random vectors have moment-generating functions that are finite and equal on a neighborhood of the origin, then they have the same probability distribution.

Write $X_{-i}$ for all inputs except $X_i$. Under the Efron–Stein inequality hypotheses, variance tensorization is the equivalent conditional variance formConditioning freezes the other inputs. Then $Z,Z_i^{\prime }$ are independent with the same conditional probability distribution, givingTaking expected values explains the factor $1/2$.^[ref]

For Bernoulli input $X_i$ with parameter $p_i$, let ${\Delta }_{i}f$ be the value of $f$ with coordinate $i$ set to one minus its value with that coordinate set to zero, keeping $X_{-i}$ fixed. The two-point conditional variance isFor the maximum example, ${\Delta }_{1}f=1-X_2$ and ${\Delta }_{2}f=1-X_1$. The two averaged conditional variances are $0.3(0.7)(0.4)=0.084$ and $0.6(0.4)(0.7)=0.168$, precisely half the respective resampling terms. In the dependent example $X_1=X_2=B$, each other coordinate already determines $W$, so both conditional variances vanish; variance tensorization also fails without independence.

For a short induction proof, separate the last input using the law of total variance. Apply the smaller-dimensional statement to the conditional expectation; Jensen's inequality for the square bounds its coordinate contributions by the original ones. One input gives equality, starting the induction.^[ref]

The variational characterization of trace distance turns a matrix distance into the best possible binary experiment. For density operators $\rho ,\sigma$ on the same finite-dimensional vector space,An operator $P$ with $0\le P\le I$ is an effect of a POVM: both $P$ and $I-P$ are positive operators. The Born rule assigns the first outcome probabilities $\mathrm{Tr}(P\rho )$ and $\mathrm{Tr}(P\sigma )$. Their signed gap is $\mathrm{Tr}(P\Delta )$, where $\Delta =\rho -\sigma$. Since $\mathrm{Tr}\Delta =0$, replacing $P$ by $I-P$ reverses this gap, explaining the equality with the absolute-value maximum.

One optimal $P$ is the spectral projector $P_{+}$ onto the positive eigenspaces of $\Delta$. In the spectral decomposition of $\Delta$, this choice includes all positive eigenvalues and excludes all negative eigenvalues. Their positive sum equals the trace distance by the spectral-parts formula for trace distance. Any zero eigenspace can be assigned to either outcome without changing the gap. This is the effect selected by the equal-prior Holevo–Helstrom theorem. ^[ref]

The same optimum has a classical description. For any finite-outcome POVM $\{E_i\}$, let $p_i=\mathrm{Tr}(E_i\rho )$ and $q_i=\mathrm{Tr}(E_i\sigma )$. ThenThe expression $\frac{1}{2}{\sum }_i|p_i-q_i|$ is the total variation distance of the entire outcome probability distributions; the expression for $A$ is the gap for one event. Grouping outcomes in $A$ gives the single effect $P_A={\sum }_{i\in A}{E}_i$. Conversely, an effect $P$ gives the binary POVM $\{P,I-P\}$, whose two probability gaps are opposites, so their total variation distance is $|\mathrm{Tr}(P\Delta )|$. Thus the optimum over all POVMs is already attained by one binary projective measurement. ^[ref]

For quantum state discrimination with equal prior probabilities, declare $\rho$ on outcome $P$ and $\sigma$ on outcome $I-P$. Directly averaging the two conditional success probabilities givesOptimizing this expression gives $(1+D)/2$. For the diagonal example in trace distance, $P=|0⟩⟨0|$ gives a gap $0.8-0.3=0.5$. For zero versus plus, that same effect gives only $1-1/2=1/2$, while the effect in Helstrom measurement for the zero and plus states gives the larger gap $1/\sqrt{2}$. The trace-distance-preserving binary measurement depends on the pair of preparations; a fixed measurement basis does not generally attain the trace distance for every pair.

Variational inference approximates a posterior distribution by minimizing a Kullback-Leibler divergence over a chosen family of probability distributions. For an unnormalized posterior density $h$ and a trial probability density function $q$, minimizing ${\mathbb{E}}_q[\log q-\log h]$ is equivalent to minimizing $\mathrm{KL}(q\Vert h/\int h)$.

The probability distribution of a nonnegative duration until an event. For a continuous duration with survival function $S(t)=P(T>t)$, its probability density function is $p(t)=-S^{\prime }(t)$. A constant hazard gives an exponential distribution; successive independent first-order waits give a convolution, as in the two-step catalytic waiting time.