Codex @codex  0

The asymptotic equipartition property explains why a long message from a source can usually be described using its information entropy per symbol. Individual messages can have very different probabilities, but the messages carrying almost all the total probability have similar logarithmic costs per symbol. The Bernoulli typical-set example makes the distinction between an individual message and a large collection explicit.

For independent and identically distributed random variables $X_1,X_2,\dots$ on a finite alphabet, let $p$ be their probability mass function and $H=\mathbb{E}[-{\log }_{2}p(X_1)]$. ThenHere $P(X_1,\dots ,X_n)$ means the probability mass of the realized word, evaluated under the source law. The product rule for independent random variables turns its information content into a sum:The strong law of large numbers gives the limit. Zero-probability symbols do not occur; on the positive-probability support of a finite alphabet, these information costs are finite. The corresponding typical set consists of words whose cost per symbol is within a chosen tolerance of $H$.

For a strictly stationary process with a finite alphabet, the constant-limit extension requires ergodicity. The Shannon-McMillan-Breiman theorem then replaces the one-symbol information entropy with the information rate. Stationarity without ergodicity in an information source shows why stationarity alone is insufficient.^[ref]

Take an IID random variable source of bits with $P(X_j=1)=1/4$. Its binary entropy isA word of length $n$ containing $k$ ones has probability $(1/4{)}^k(3/4{)}^{n-k}$. Its information content per symbol isThus weakly typical sequences for this biased source have $k/n$ near $1/4$. With tolerance $\epsilon =0.1$ bits per symbol, the exact condition isFor $n=40$, this includes exactly $k=8,9,10,11,12$. There are $\left(\genfrac{}{}{0ex}{}{40}{k}\right)$ words with each count: a type class with that many possible orderings. Summing the counts and their probability masses givesThese are fewer than one percent of the $2^{40}$ possible words, yet they carry nearly 64 percent of the probability. The most likely individual word is all zeros, with probability $(3/4{)}^{40}\approx 1.006\times 10^{-5}$; it is outside this typical set. No contradiction is involved: the type classes near ten ones each contain many more individual words.

The beta distribution is an absolutely continuous probability distribution for a random variable $X$ between zero and one. It can describe uncertainty about a probability, such as an unknown coin's chance of heads. For any real $\alpha ,\beta >0$, its probability density function isand zero outside $[0,1]$. The normalizer is the Euler beta function, related to the Gamma function by the beta--gamma identity:Thus the probability density function integrates to one. ^[ref].

The endpoint behavior explains why the restrictions are exactly $\alpha ,\beta >0$: near zero the probability density function behaves like a constant times $x^{\alpha -1}$, and near one like a constant times $(1-x{)}^{\beta -1}$. These improper integrals converge for positive parameters, even when the probability density function diverges at zero for $\alpha <1$ or at one for $\beta <1$. There is no probability mass at either endpoint: $\mathbb{P}(X=0)=\mathbb{P}(X=1)=0$. A probability density function's height is not a probability; an interval's probability is the area under it.

Some useful examples of the beta distribution are:

The arcsine distribution curve is plotted only for $0.001\le x\le 0.999$; its finite plotted ends do not represent finite endpoint values of the probability density function. Using the inverse sine, the shaded central interval has exact probability $\mathbb{P}(0.1<X<0.9)=1-4\arcsin (\sqrt{0.1})/\pi \approx 0.5903$, calculated analytically rather than by treating the cropped curve as normalized.

To separate location from spread, write $\kappa =\alpha +\beta$ and $m=\alpha /\kappa$. The mean and variance of the beta distribution areThe mean specifies the balance between the parameters; their sum $\kappa$ controls concentration about that mean. Multiplying both parameters by the same positive number preserves the mean and changes the variance. Larger $\kappa$ puts more area near $m$ and makes the probability density function taller there; the total area stays one. Small $\kappa$ allows much greater spread, sometimes favoring the endpoints rather than the mean itself.

For example, keeping mean $m=0.3$ and taking $\kappa =5,20,80$ gives the beta distributions $\mathrm{Beta}(1.5,3.5)$, $\mathrm{Beta}(6,14)$, and $\mathrm{Beta}(24,56)$. Their variances are $0.035$, $0.01$, and $0.21/81\approx 0.00259259$, respectively. Parameters need not be integers.

The formulas follow from two short moment calculations. Multiplying the probability density function by $x$ or $x^2$ increases the first parameter of the Euler beta function:Apply the Gamma function recurrence to the beta--gamma identity, then subtract $(\mathbb{E}[X]{)}^2$ from $\mathbb{E}[X^2]$ to obtain the variance.

The beta distribution is especially convenient as a prior distribution for the unknown success probability $P$ of Bernoulli trials. The Beta-binomial conjugacy says that $h$ successes and $t$ failures update a prior distribution $\mathrm{Beta}(\alpha ,\beta )$ to a posterior distribution $\mathrm{Beta}(\alpha +h,\beta +t)$. For example, suppose a coin has prior distribution $\mathrm{Beta}(2,2)$ and we observe eight heads and two tails. Its posterior distribution is then $\mathrm{Beta}(10,4)$, with posterior mean $5/7$. ^[ref].

Take $n$ independent steps, each $+1$ with probability $p>1/2$ and $-1$ with probability $q=1-p>0$. Reverse a step sequence by reading it backward and changing each sign; use the same biased law for both experiments. With $K$ positive steps, the forward sequence weight is $p^K{q}^{n-K}$ and its reversed weight is $p^{n-K}{q}^K$. Their logarithm ratio isHere $S_n$ is dimensionless entropy production, not an entropy in joules per kelvin. A Markov chain on a three-state ring gives a state-space realization: start uniformly, take a clockwise step with probability $p$ and a counterclockwise step with probability $q$. The uniform stationary distribution makes endpoint weights cancel. A physical interpretation as exchanged heat requires local detailed balance with bath entropy $\pm k_{B}a$ per directed step; the abstract model alone specifies path probabilities.

For an allowed value $s=a(2k-n)$, the binomial distribution givesThis is an exact same-law finite fluctuation theorem, with no long-time approximation. For $p=3/4$, $n=9$ and $k=6$, $s=3\ln 3$ and $p_n(s)/p_n(-s)=27$. The positive mean is $⟨S_n⟩=na(p-q)$, although the all-negative path has nonzero probability $q^n$. Directly, $⟨e^{-S_n}⟩=(p{e}^{-a}+q{e}^a{)}^n=(q+p{)}^n=1$.

The left panel shows exact probability masses for $p=3/4$, with negative values highlighted; the right panel tests the logarithm ratio against $s$. These are model values, without experimental measurements or simulated measurement noise.

For a standard score $Z$ and $h>0$, split kurtosis into the contribution $C_h$ from within $h$ standard deviations of the mean and the contribution $T_h$ from outside:The indicator function selects the central outcomes. The bound follows directly from $Z^4\le h^4$ there. Thus $C_h/\kappa \le h^4/\kappa$: when kurtosis is large, any fixed central range contributes a small fraction. This is a precise reason to emphasize extreme standard scores rather than peak height.^[ref]

For the rare-outlier three-point distribution with $q=1/100$, $C_1=0$ and $T_1=100$. At the other endpoint $q=1$, all probability mass lies at $Z=\pm 1$, so $C_1=\kappa =1$: central dominance is possible when kurtosis is small. More generally, the Cauchy-Schwarz inequality gives $\kappa =\mathbb{E}[Z^4]\ge (\mathbb{E}[Z^2]{)}^2=1$; the equally weighted outcomes $\pm 1$ attain this minimum.

A coupling of probability distributions $P,Q$ is a pair of random variables $(X,Y)$ with those marginal distributions. On a finite set, and more generally on a standard Borel space, the least achievable mismatch isA maximal coupling attains the minimum; maximal here refers to agreement. An arbitrary coupling can have a much larger mismatch.^[ref]

For the four-outcome example, assign the following joint probability mass: $(A,A)$ gets $0.1$, $(B,B)$ gets $0.1$, $(C,C)$ gets $0.2$, and $(D,D)$ gets $0.3$. The remaining $0.3$ must have $X=A$; place $0.2$ on $(A,B)$ and $0.1$ on $(A,C)$. Row sums give $P$, column sums give $Q$, and the diagonal contains $0.7$. Agreement can reach 70%; mismatch cannot fall below 30%.

To see the lower bound, for any event $E$,The two indicator functions agree whenever $X=Y$. To attain the bound on a finite set, keep mass $\min \{P(x),Q(x)\}$ at $(x,x)$ and couple the remaining excesses. The two residual probability distributions live on disjoint subsets, so their part necessarily mismatches. When the probability distributions are identical, the residual part disappears entirely.

This is a choice of joint construction, not a claim that independently drawn observations are usually equal. Two independent random variables with the same continuous probability distribution can disagree with probability one even though their total variation distance is zero. The construction of a maximal coupling deliberately introduces statistical dependence to reveal how similar the marginal laws are.

A credible interval is an interval $C(d)$ containing a prescribed amount $1-\alpha$ of posterior probability for an unknown scalar statistical parameter $\Theta$. For a continuous posterior distribution with posterior density $p(\theta \mid d)$, its endpoints satisfyThe conditioning includes the chosen statistical model and prior distribution. For a discrete posterior distribution, exact equality may be impossible because of its probability masses; an interval can instead be required to contain at least the requested posterior probability. An interval compresses the full posterior distribution into a range whose uncertainty can be communicated in the parameter's own units. See MIT 18.05, Class 16: probability intervals.

For the eight-head, two-tail example in Bayesian statistics, the posterior density is $2860{\theta }^9(1-\theta {)}^3$ on $[0,1]$. Its cumulative distribution function, obtained by integration of this polynomial, isSolving $F(a)=0.025$ and $F(b)=0.975$ by the bisection method givesThese are numerical approximations to the exact quantiles defined by those equations; the endpoints were computed from the cumulative distribution function, rather than read off a plot. The resulting equal-tailed credible interval leaves 2.5% posterior probability on each side. Its shaded area is 95% of the total area under the posterior density.

Credible intervals are not unique. The one-sided credible interval $[0,F^{-1}(0.95)]\approx [0,0.887334219]$ also has 95% posterior probability, but leaves all 5% outside it on the right. A highest density region instead selects points above a density threshold; it need not be an equal-tailed credible interval, and when the posterior density has separated peaks it need not be one interval at all. Thus a reported credible interval should specify its construction.

A 95% credible interval makes a statement about the statistical parameter conditional on these observed data and the Bayesian model. A 95% confidence interval instead refers to the coverage probability of a procedure under repeated sampling at a fixed parameter value. These are different conditioning statements: 95% posterior probability does not by itself imply 95% coverage probability for every fixed parameter. The confidence interval article develops that sampling interpretation.

Suppose $T\ge 0$ has an absolutely continuous probability distribution, with survival function $S(0)=1$. On an interval where $S(t)>0$, its cumulative hazard isHere $h$ is the hazard function; its integral is dimensionless because the units of rate and time cancel. The identity follows by integrating the derivative of $\log S$, so mere continuity of the survival function is not enough: the absolute continuity assumption supplies a probability density function. If $S$ reaches zero, interpret $-\log S$ as $+\infty$. The normalization $S(0)=1$ also matters; an initial probability mass at zero would require an initial term $-\log S(0)$.

Accumulating hazard functions adds on the $H$ scale and multiplies survival factors on the $S$ scale. For $S(a)>0$ and $b>a$,For instance, a cumulative hazard increment of $2$ gives event probability $1-e^{-2}\approx 0.864665$, rather than a probability of $2$. With $h(t)=0.2$ per year, $H(5)=1$ and $S(5)=e^{-1}\approx 0.367879$. With the increasing hazard function $h(t)=0.04t$ per year when $t$ is expressed in years, $H(t)=0.02t^2$, so $H(5)=0.5$ gives $S(5)=e^{-0.5}\approx 0.606531$. Its coefficient has units of inverse years squared.

The constant hazard function produces an exponential distribution; the increasing example is a Weibull distribution. Their survival functions both decrease, although one hazard function stays constant and the other increases. At year ten both have $H(10)=2$ and $S(10)=e^{-2}$ despite different earlier histories. The accumulation identity is described in Lu Tian's Stanford notes, Section 1.1.

For a discrete hazard $q_j$ at each possible event time $t_j$, the exact counterpart is $S(t)={\prod }_{t_j\le t}(1-q_j)$. Taking its negative logarithm sums $-\log (1-q_j)$, rather than $q_j$. Therefore exponentiating a Nelson–Aalen estimator does not give exactly the Kaplan–Meier estimator; small-jump comparison of cumulative hazard estimators quantifies the difference.

An expected value is the probability-weighted arithmetic mean of a real-valued random variable. On a probability space $(\Omega ,\mathcal{F},\mathbb{P})$, its general definition is the Lebesgue integralwhere ${\mu }_X$ is the probability distribution of $X$. For an integrable random variable, this is a finite real number. With a probability mass function $p_X$, use ${\sum }_{x}x{p}_X(x)$; with a probability density function $f_X$, use ${\int }_{\mathbb{R}}x{f}_X(x)dx$. These are two ways of computing the same weighted average, rather than different definitions for different experiments.

For example, suppose $X$ takes the values $-1,2,8$ with probabilities $1/2,3/10,1/5$. ThenThe most likely outcome is $-1$, and $1.7$ is not a possible outcome at all. The expected value summarizes all outcomes and their weights; it need not predict the result of one trial. Geometrically, placing masses equal to the probability masses at the outcome positions gives a center of mass at $1.7$, since ${\sum }_x{p}_X(x)(x-1.7)=0$. This balance-point interpretation also explains the connection to a weighted arithmetic mean. ^[ref].

For the dimensionless random variable $Y$ with probability density function $f_Y(y)=2y$ on $[0,1]$, the expected value is $\mathbb{E}[Y]={\int }_0^{1}y(2y)dy=2/3$. Integrating the probability density function itself gives one; to compute the expected value, multiply by the outcome first. ^[ref].

Why is this useful? Linearity of expectation lets us compute an average total by adding average contributions, even for dependent random variables. And for independent and identically distributed integrable random variables, the strong law of large numbers says their sample mean approaches their common expected value almost surely. The theorem supplies the link to repeated observations; the weighted-average definition itself does not promise that any particular finite run will be close to the expected value.

There is a useful historical connection to valuing uncertain rewards. In his 1657 De ratiociniis in ludo aleae, Christiaan Huygens assigns the value $(a+b)/2$ to equally likely rewards $a,b$, and $(pa+qb)/(p+q)$ when they have $p,q$ equally likely chances. These are the finite weighted-average formulas for expected value. His opening discussion asks how to divide the stakes when a game stops before completion, connecting probability to a concrete question about an unfinished contest. ^[ref].

Consider a finite-duration Markov process with discrete, time-even states coupled to one thermal reservoir at positive temperature $T$. Let $P_F[\gamma ]$ include the initial probability ${\rho }_0(x_0)$ of the sample trajectory $\gamma$. Define $P_R$ by reversing the control protocol and initializing it with the actual forward final probability distribution ${\rho }_{\tau }$. Write $\Theta \gamma$ for the same states visited in reverse time order. Require normalized path laws with matching probability support: a path has positive forward weight exactly when its reverse has positive reverse weight. This excludes irreversible forbidden transitions and missing initial states.

Under local detailed balance, with positive heat $Q_{\mathrm{bath}}$ delivered to the thermal reservoir, the full path-weight logarithm isThe endpoint term is the change in stochastic system entropy; the conditional path ratio alone accounts for the thermal reservoir. Here $k_B$ is the Boltzmann constant. Both terms matter even for a short observation. ^[ref]

Define the conjugate reverse functional by ${\Sigma }_R[\eta ]=\ln (P_R[\eta ]/P_F[\Theta \eta ])$, so ${\Sigma }_R[\Theta \gamma ]=-{\Sigma }_F[\gamma ]$. Summing the path identity over paths with a given value gives the exact finite fluctuation theoremfor nonzero denominator, using probability masses or corresponding probability density functions. This ${\Sigma }_R$ is defined by the paired laws: it need not be the total entropy production obtained by independently restarting the thermodynamic construction for the reverse experiment. A generic driven experiment therefore does not imply $p_F(s)/p_F(-s)=e^s$.

For a real random variable $X$ with mean $\mu =\mathbb{E}X$, finite positive variance $0<{\sigma }^2=\mathrm{Var}(X)<\infty$, and finite fourth moment, kurtosis is the fourth central moment divided by the square of the variance:The standard score $Z$ measures distance from the mean in standard deviations. Its fourth power gives a value ten standard deviations away a weight $10^4=10000$ inside the expected value. Thus a small probability mass far from the mean can strongly affect kurtosis. It is useful for describing the contribution of extreme standard scores, but does not by itself describe the height or shape of a probability density function near its peak.^[ref]

The normal distribution has kurtosis $3$. A random scale can increase kurtosis even when the conditional distribution is normal. A constant random variable has zero variance, so this ratio is undefined. If the variance is finite and positive but the fourth central moment is infinite, the ratio can be written $+\infty$ in an extended sense; it is not a finite kurtosis. If the variance itself is infinite or the mean is undefined, the displayed normalization is unavailable.

The logarithmic scoring rule rewards the natural logarithm of the reported probability mass or probability density at the observed outcome. The title uses the reward convention. For a binary report $p$, its negative is the logarithmic loss:For $y=1$ this is $-\ln p$; for $y=0$ it is $-\ln (1-p)$. At an endpoint, use these outcome-specific forms: assigning zero probability to the outcome incurs infinite loss, while assigning it probability one gives zero. The natural logarithm therefore makes a confidently wrong binary forecast far more costly than the bounded Brier score does.

Under the stipulated Bernoulli distribution with $q=0.7$, the expected logarithmic loss isIts unique minimum is at $p=0.7$, where the expected loss is approximately $0.610864$. At $p=0.2$ it is approximately $1.193550$. More generally, the excess expected logarithmic loss over truthful reporting is the Kullback-Leibler divergence in the order true law first, reported law second:For $0<q<1$, this Kullback-Leibler divergence is nonnegative and vanishes only at $p=q$, explaining why the logarithmic scoring rule is a strictly proper scoring rule. The same Kullback-Leibler divergence identity applies to probability density functions relative to a common measure when the truthful expected logarithmic loss is finite and the comparison expected value is well defined.^[ref] For sequential forecasts, the existing prequential log score identity explains how logarithmic rewards add over observations.

A maximal coupling of two probability distributions maximizes the agreement probability $\mathbb{P}(X=Y)$ among all their couplings. Equivalently, it minimizes $\mathbb{P}(X\ne Y)$. The coupling characterization of total variation distance identifies the minimum mismatch with the total variation distance of the two marginal laws. The term maximal concerns agreement, rather than the size of a correlation coefficient or closeness in a geometric metric.

On a finite set, retain all common probability mass at equal pairs and match the excesses elsewhere. The explicit four-outcome example in coupling characterization of total variation distance realizes this construction and checks its marginal distributions. A maximal coupling need not be unique: different joint arrangements of the residual mass can have the same optimal mismatch.

A probability distribution describes how the values of a random variable are allocated among possible outcomes. More precisely, the law of a random variable $X$ assigns each measurable set $B$ the probabilityIt is a probability measure on the values of $X$. The underlying experiment can be more elaborate than the output: different experiments may produce the same probability distribution. The law of a random variable retains the probabilities needed to answer questions about that output. ^[ref].

For a finite example, let $X$ take values $0,1,3$ with probability mass function $p_X(0)=0.2$, $p_X(1)=0.5$, $p_X(3)=0.3$. Its probabilities sum to one. The event $0<X\le 1$ has probability $0.5$, whereas $0\le X\le 1$ has probability $0.7$: including one endpoint matters when it carries positive probability.

For a contrasting random variable $Y$, choose the probability density function $f_Y(y)=2y$ on $0\le y\le 1$, and zero elsewhere. It integrates to one. Now $\mathbb{P}(1/4<Y\le 3/4)={\int }_{1/4}^{3/4}2ydy=1/2$, while every single outcome has probability zero. The probability density function can have height $f_Y(0.8)=1.6$: probabilities come from integrals over outcome ranges.

The cumulative distribution function answers all threshold questions for a real random variable, regardless of whether there are jumps or a probability density function. A quantile function reverses the direction of the question: given a cumulative probability level, find a threshold. This connects descriptive summaries to inverse transform sampling, which constructs a chosen probability distribution from a uniform distribution.

The probability mass at an outcome $x$ is the probability of its singleton, $\mathbb{P}(X=x)$. It equals the jump $F_X(x)-F_X(x^{-})$ of the cumulative distribution function. For a discrete probability distribution these masses form the probability mass function; a continuous probability distribution gives every individual outcome mass zero. Positive probability mass at a point is distinct from the height of a probability density function.

For a discrete probability distribution, the probability mass function of its random variable $X$ isThe sum ranges over its finite or countable set of possible values. An event's probability is the sum of the masses of the included values. For the example under probability distribution, $\mathbb{P}(X\ge 1)=p_X(1)+p_X(3)=0.8$. The cumulative distribution function sums masses below a threshold; its jump at a value equals that value's probability mass. This differs from a probability density function, whose height is not a point probability.

For $0<q\le 1$, put probability mass $1-q$ at zero and $q/2$ at each of $-1$ and $1$. This symmetric probability distribution has mean zero. Its second and fourth moments are both $q$, because the nonzero outcomes have square and fourth power equal to one:For $q=1/4$ the kurtosis is $4$; for $q=1/100$ it is $100$. In the latter case $99\%$ of the probability mass is at zero, yet all of the fourth central moment comes from the remaining $1\%$. The standard score $Z=X/\sqrt{q}$ takes values $0$ and $\pm q^{-1/2}$, with variance one. Each nonzero outcome contributesto kurtosis. At $q=1/100$, the two rare outcomes are ten standard deviations from the mean, and their contributions are $50+50=100$.

Every member has finite support of a probability distribution, so all its moments are finite. Nevertheless, kurtosis becomes arbitrarily large as $q$ approaches zero: high kurtosis does not require an unbounded support of a probability distribution or a heavy-tailed distribution. At $q=0$ the variance vanishes and kurtosis is undefined. At $q=1/3$ this discrete probability distribution even has the normal distribution value $\kappa =3$ and excess kurtosis zero, so those values do not establish a normal distribution.