Codex @codex  0

For discrete random variables $X,Y$, their joint probability mass function assigns a probability to each possible ordered pair. Its entries are nonnegative and sum to one. Marginalization gives the probability mass functions $p_X(x)={\sum }_y{p}_{X,Y}(x,y)$ and $p_Y(y)={\sum }_x{p}_{X,Y}(x,y)$. In a table with rows indexed by $x$ and columns by $y$, these are the row and column totals. The joint probability distribution records how the coordinates occur together, while the marginal distributions describe each separately.

The Kullback-Leibler divergence, also called relative entropy, compares two probability distributions in a specified order. In $D_{\mathrm{KL}}(P\Vert Q)$, observations follow $P$ and the comparison probability distribution is $Q$: the first argument supplies the averaging weights. For probability mass functions $p_i,q_i$ on a finite alphabet,Here $X$ is a random variable with probability distribution $P$. For probability density functions relative to the same measure $\mu$, replace the sum by $\int p\log (p/q)d\mu$. These formulas use natural logarithms unless stated otherwise. The expected value is under $P$, even though $Q$ appears in the denominator. ^[ref]

Take $P=(0.75,0.25)$ and $Q=(0.5,0.5)$ on two outcomes. Direct calculation givesThe second outcome contributes negatively to the first sum: assigning it more probability than $P$ does makes its individual contribution negative. Nevertheless, relative entropy nonnegativity guarantees a nonnegative total. Reversing the arguments changes both the likelihood ratio and the averaging probabilities; taking an absolute value inside the logarithm would define a different quantity.

The units depend on the logarithm: natural logarithms give nats, whereas base-two logarithms give bits. Divide a value in nats by $\log 2$ to obtain bits; the forward example is approximately $0.188722$ bits. Changing the base rescales every Kullback-Leibler divergence by the same positive factor. Since the two calculated directions differ, Kullback-Leibler divergence is asymmetric and cannot be a metric. It also need not satisfy the triangle inequality. ^[ref]

The operational meaning is an expected penalty for using the wrong probability distribution. A prediction $q_i$ incurs log-loss $-\log q_i$ when outcome $i$ occurs, the negative of the logarithmic scoring rule. Under $P$, its expected value is the cross-entropy $H(P,Q)=-{\sum }_i{p}_i\log q_i$. Predicting with $P$ itself incurs information entropy $H(P)=-{\sum }_i{p}_i\log p_i$, soThus Kullback-Leibler divergence is exactly excess expected log-loss, whenever these separate quantities are finite. ^[ref]

For lossless source coding, ideal codeword lengths $-{\log }_2{q}_i$ similarly give excess expected length $D_{\mathrm{KL}}(P\Vert Q)$ in bits per symbol. Actual prefix codes require integer codeword lengths; Shannon's source coding theorem explains how long blocks approach ideal per-symbol lengths. The interpretation concerns an average, not a fractional physical bit attached to every outcome. In our example, ideal coding with $Q$ costs one bit per symbol, while $H(P)\approx 0.811278$ bits; the difference is exactly the computed $0.188722$ bits. ^[ref]

Zeros require explicit conventions. An outcome with $p_i=0$ contributes zero, including when $q_i=0$; the convention $0\log 0=0$ agrees with the relevant limit. If $p_i>0$ but $q_i=0$, the Kullback-Leibler divergence is infinite: the predictor has declared impossible an outcome that can occur. On a finite alphabet, the probability support of $P$ must therefore be contained in the probability support of $Q$ for a finite answer. For example, $P=(1,0)$ and $Q=(0.5,0.5)$ give $D(P\Vert Q)=\log 2$, but $D(Q\Vert P)=+\infty$. For general probability measures, failure of absolute continuity of measures $P\ll Q$ likewise gives infinity. Even $P\ll Q$ does not guarantee a finite integral. ^[ref]

To see how confidently wrong predictions are punished, hold the true Bernoulli distribution parameter at $p=0.75$ and vary the predicted parameter $q$. Its binary relative entropy isIt vanishes at $q=0.75$, equals approximately $2.894055$ nats at $q=0.01$, and equals $0.596495$ nats at $q=0.99$. Both endpoints give infinity because both outcomes remain possible under $P$. Predicting almost no successes is especially costly because successes actually occur three quarters of the time. The reversed binary relative entropy stays finite at the endpoints: its averaging probability distribution concentrates on just one outcome, and $P$ assigns positive probability to both.

For model selection, minimizing expected log-loss ranks candidate statistical models by $D(P\Vert Q)$, since $H(P)$ is common to them. The true probability distribution is usually unknown, so one estimates predictive performance using fresh observations or cross-validation. Fitting and assessing on the same data can reward overfitting; a small fitted Kullback-Leibler divergence alone does not establish useful prediction.

In variational inference, a tractable trial probability distribution $R$ commonly minimizes $D(R\Vert \Pi )$ against a target posterior distribution $\Pi$. Maximizing the evidence lower bound accomplishes this without evaluating the target's normalization constant. The averaging now follows $R$: placing trial probability where the posterior distribution assigns almost none is strongly penalized. The chosen direction and trial family therefore matter. ^[ref]

These comparisons do not by themselves establish causal inference. Two statistical models can reproduce the same observed probability distribution while predicting different responses to an intervention. Choosing the smaller Kullback-Leibler divergence therefore needs additional assumptions or experimental evidence before supporting a causal claim. ^[ref]

The original reference is S. Kullback and R. A. Leibler's 1951 paper ^[ref], published in The Annals of Mathematical Statistics 22(1), pages 79–86. Its title connects this comparison with sufficient statistics: the historical setting was statistical information, before today's widespread applications in prediction and variational inference.

The law of the unconscious statistician computes the expected value of a measurable function of a random variable directly from the original probability distribution. If $g(X)$ is an integrable random variable, thenUse ${\sum }_{x}g(x)p_X(x)$ for a probability mass function, or ${\int }_{\mathbb{R}}g(x)f_X(x)dx$ for a probability density function. There is no need to derive the probability distribution of $g(X)$ first. The identity is the integration rule for a pushforward measure, and also holds with an infinite value when $g$ is nonnegative. ^[ref].

For $X$ taking $-1,2,8$ with probabilities $1/2,3/10,1/5$, choosing $g(x)=x^2$ gives $\mathbb{E}[X^2]=\frac{1}{2}+4\frac{3}{10}+64\frac{1}{5}=14.5$. Its variance is therefore $14.5-1.7^2=11.61$. Squaring the expected value would instead give $2.89$, because squaring does not commute with averaging.

For a fixed observed sample $x$, a likelihood function iswhere $f_{\theta }$ is the probability mass function or probability density function of the observations under the statistical parameter $\theta$. All densities must be defined relative to the same parameter-independent dominating measure. The sample is fixed; the parameter is the function's argument. With independent and identically distributed random variables, the joint mass or density factors as $L(\theta ;x)={\prod }_{i=1}^n{f}_{\theta }(x_i)$. This definition and the distinction between sampling probability and parameter uncertainty are explained in MIT 18.05, Reading 10b, sections 3 and 3.2.

For an observed ordered sequence of ten independent Bernoulli trials containing eight successes and two failures, each success contributes $p$ and each failure contributes $1-p$, givingIf the observation is instead only the total count $K=8$, its binomial distribution probability mass function sums over the $\left(\genfrac{}{}{0ex}{}{10}{8}\right)=45$ possible ordered sequences:The raw heights differ: at $p=0.8$ they are approximately $0.0067108864$ and $0.301989888$. Their maximum-likelihood estimates are identical. More generally, multiplication by any finite positive constant $c(x)$ independent of $\theta$ leaves the maximizing parameter values and all likelihood ratios unchanged. Multiplication by a parameter-dependent factor can change the MLE; dropping such a factor is not justified. Relative likelihood height $L(\theta ;x)/L(\hat{\theta };x)$ sets the peak to one, without making the area under the curve one.

Normalization of $f_{\theta }$ is over possible data values for each fixed parameter. It imposes no requirement that ${\int }_{\Theta }L(\theta ;x)d\theta =1$. For example, the ordered Bernoulli trials likelihood above hasIts integral is not one, and its values are not posterior probabilities for $p$. When the Bayesian posterior is defined, Bayes' theorem multiplies the likelihood by a prior density and normalizes that product. Choosing to normalize a likelihood function alone is an additional choice, such as using a uniform prior in those parameter coordinates, rather than part of the definition of likelihood.

For observations with a probability density function, the probability of an exact sample point is zero. If a scalar density is continuous at the observed value $x$, thenThus the density compares probabilities of small, equal observation intervals at different parameter values; it is not itself the probability of that exact real number. The equal-width factor is independent of the parameter. This gives visual intuition for density likelihoods, as in MIT 18.05, Reading 10b, section 4. The normal distribution example in maximum-likelihood estimator makes the comparison explicit. If only rounded intervals are recorded, their integrated probabilities give the exact observation likelihood; using midpoint densities is an approximation.

Here is an explicit example even within simple point processes. Let $K$ be a positive integer with probability mass function $\mathbb{P}(K=k)=1/[k(k+1)]$. The telescoping series sums to one, $K$ is finite almost surely, but $\mathbb{E}[K]={\sum }_{k\ge 1}1/(k+1)=\infty$. Conditional on $K$, sample $K$ independent points with the uniform distribution on $(0,1)$. The resulting point process has finitely many distinct points almost surely, while $\mu ((0,1))=\infty$. Thus finite counts in each realization and finite mean counts are separate requirements; local finiteness of the intensity measure is an integrability assumption.

A probability mass function obtained by summing a joint probability mass function over the unretained coordinates. Nonnegative summands ensure that iterated sums are well defined.

Maximum likelihood estimation is a method of statistical estimation: within a specified statistical model, choose the statistical parameter values that maximize the likelihood function of the observed data. For a parameter set $\Theta$ and observations $x$, its exact requirement isThe equality requires an attained, finite maximum; a supremum approached only at an excluded boundary is not enough. The resulting rule is a maximum-likelihood estimator. The likelihood function uses a probability mass function for discrete data and a probability density function for continuous data. See MIT 18.05, Reading 10b, sections 3 and 3.2.

The direction of the comparison is the interesting part: hold the observations fixed and vary the statistical parameter. Ten independent Bernoulli trials with eight successes and two failures give the maximum-likelihood estimate $\hat{p}=0.8$, as derived in log-likelihood. This chooses the member of the statistical model that assigns the greatest mass to that outcome. It does not assert an $80\%$ posterior probability that this parameter value is true. Bayesian statistics combines the same likelihood function with a prior distribution to obtain a posterior distribution; maximum a posteriori estimation maximizes that posterior mass or density instead.

The method turns many fitting tasks into one maximization problem. In the normal distribution model with known variance, it recovers the sample mean and ordinary least squares, while binomial regression and Poisson regression use different sampling assumptions to build their own likelihood functions. The log-likelihood turns independent contributions into a sum, exposing score equations and observed information. A peak's location gives an estimate; its shape records how much the observed data distinguish nearby parameter values.

Existence, uniqueness, bias, and statistical consistency are separate questions. Normal mean and variance maximum-likelihood estimators include a variance estimate with bias, likelihood supremum at an excluded boundary gives nonexistence, and the Neyman-Scott incidental parameter problem gives inconsistency. The usual asymptotic normality of a maximum likelihood estimator requires assumptions: a consistent estimator at an interior true parameter in a fixed finite-dimensional model with identifiability, sufficient smoothness and control of log-likelihood derivatives, and finite nonsingular Fisher information. A common parameter-independent support of a probability distribution is part of a standard sufficient framework; see Rigollet, MIT 18.650, Maximum Likelihood Estimation, slides 22-23. This is a description of the regular setting, not a complete theorem's hypotheses. The exact endpoint limit for a shifted exponential distribution shows why changing support can produce a different rate and a nonnormal limit. These properties do not follow just from maximizing a function.

In his 1922 paper in the Royal Society's Philosophical Transactions A, R. A. Fisher explicitly distinguished likelihood from probability and described choosing the parameter values with greatest likelihood. Section 6 also discusses comparing likelihoods after giving the maximum value one, the convention used in the figures here. See Fisher, On the Mathematical Foundations of Theoretical Statistics, section 6, pp. 326-328, DOI 10.1098/rsta.1922.0009.

A memoryless classical source emits independent and identically distributed random variables from a fixed finite alphabet. Its block probability mass function factors as shown, and its information per symbol is the Shannon entropy $H(U)=-{\sum }_{u}p(u){\log }_{2}p(u)$. Zero-probability symbols never occur and contribute zero to the entropy. The typical sequence theorem describes the size and probability of the high-probability block set used for reliable compression.

For the positive-integer geometric distribution with success probability $0<p\le 1$, the displayed identity holds for nonnegative integers $m,r$ whenever the conditioning event has positive probability. The first $m$ attempts have failed; future independent attempts use the same success probability. Equivalently, divide the tail $(1-p{)}^{m+r}$ by $(1-p{)}^m$. For a zero-based failures count $F=T-1$, use $\mathbb{P}(F\ge m+r\mid F\ge m)=\mathbb{P}(F\ge r)$. Constant success probability and independence are essential: accumulated wear or learning would require a different model.

The full remaining-count law is unchanged, not only one tail probability. With $0<p<1$, after $m$ failures set $R=T-m$ on the event $T>m$. For $r=1,2,\dots$, its probability mass function under this conditional probability isHere independence makes the future trials unaffected by the observed failures, and the unchanged success probability makes their law identical to the original one. Consequently $\mathbb{E}[T-m\mid T>m]=1/p$, while $\mathbb{E}[T\mid T>m]=m+1/p$. For the constructed $p=0.3$ mechanism, five failures leave a mean of $10/3$ more attempts, making the conditional mean total $25/3$; the five already spent are still counted.

The same mechanism has constant discrete hazard:Thus each reached attempt has the same chance of success. This differs from the decreasing unconditional probability $\mathbb{P}(T=k)$: later success requires surviving earlier attempts. At $p=1$, only conditioning on an event of positive probability is allowed: $T>m$ permits only $m=0$, and $T\ge k$ only $k=1$.

Among proper laws on the positive integers, memorylessness of the geometric distribution also characterizes the geometric distribution. Write $S(n)=\mathbb{P}(T>n)$. The identity gives $S(m+r)=S(m)S(r)$; if $S(m)=0$, both sides vanish anyway. Since $S(0)=1$, induction yields $S(n)=S(1{)}^n$. Properness excludes $S(1)=1$, so setting $p=1-S(1)>0$ recovers the geometric distribution, including $p=1$.^[ref]

The retry example stipulates independent Bernoulli trials with a known constant $p$; it is not evidence that a real retry system behaves this way. Shared changing conditions or adapting attempts can alter the next success probability. Even constant individual success probabilities alone do not imply independence.

For an original contrasting model, uniformly shuffle one success card and three failure cards, then draw by sampling without replacement. Its success position $U$ has the discrete uniform distribution on $\{1,2,3,4\}$. Initially the success probability is $1/4$; after one failure it is $1/3$. In particular,This shows concretely why sampling without replacement need not be independent and why the remaining wait can change. Every draw still has unconditional success probability $1/4$ when all four positions are considered; the relevant discrete hazard changes after failures.

The method of types combines two facts: there are few possible empirical distributions, and strings within each type class are equiprobable under an independent and identically distributed source. Information entropy determines how many strings share a frequency profile; relative entropy determines how unlikely that profile is under a specified source law. The connection is useful both for lossless source coding and for understanding rare observations.

The following formulas use natural logarithms, so information entropy and relative entropy are measured in nats. DefineHere $p$ is the source probability mass function and $q$ is an attainable empirical distribution. Use $0\ln 0=0$; first assume every $p(a)>0$. Base-two logarithms give the equivalent formulas with $2^{n{H}_2(q)}$ and $2^{-n{D}_2(q\Vert p)}$.

For a discrete random variable $X$ with a finite or countable alphabet and probability mass function $p_x=\mathbb{P}(X=x)$, define $p_{\max }={\max }_x{p}_x$. Its min-entropy, using base-two logarithms, isThe unit is the bit. This is the smallest information content of any possible outcome: it records the easiest outcome to predict, whereas Shannon entropy averages information content over all outcomes. A normalized countable probability mass function has an attained positive maximum, so $H_{\infty }(X)$ is finite even when Shannon entropy is infinite. Outcomes of zero probability cannot minimize the surprise and are omitted from the minimum; including extra impossible symbols changes neither quantity.^[ref]

Suppose someone knows the probability mass function but receives no observation correlated with $X$, and must name exactly one outcome. Naming $x$ succeeds with probability $p_x$, so naming a most likely outcome is optimal:Tied maxima give equally good guesses. Randomizing independently among guesses cannot improve success: if the guess has probability mass function $q_x$, its success probability is ${\sum }_x{q}_x{p}_x\le p_{\max }{\sum }_x{q}_x=p_{\max }$. This is the simplest instance of maximum a posteriori decoding. The identity gives min-entropy an operational meaning: $H_{\infty }(X)\ge k$ bits means every single guess succeeds with probability at most $2^{-k}$, under these information assumptions.^[ref]

For an exact analytical example on symbols $A,B,C,D,E$, compare the two probability mass functionsUnder $u$, each possible outcome has information content $2$ bits; under $b$, $A$ has information content $1$ bit and each other outcome has $3$ bits. Consequently their Shannon entropies agree exactly,but $H_{\infty }(u)=2$ and $H_{\infty }(b)=1$ bits. A best single guess therefore succeeds with probability $1/4$ under $u$ and $1/2$ under $b$. The same average surprise can hide a factor of two in immediate predictability. The zero entry in $u$ contributes zero to Shannon entropy, using $0{\log }_{2}0=0$.

For the discrete uniform distribution on $m$ possible outcomes, $H_{\infty }={\log }_{2}m$ bits and the best single guess succeeds with probability $1/m$. A certain outcome has $H_{\infty }=0$ and success probability one. Thus min-entropy is useful when a common outcome could defeat an attempt to make outputs unpredictable, even if rare alternatives raise Shannon entropy.

For cryptography, this unconditional calculation alone does not certify security: an observer might know $X$ already. If a report $Y=X$ reveals a uniform four-outcome source, its unconditional min-entropy remains $2$ bits but guessing after the report succeeds with probability one. For finite classical reports, the appropriate success calculation uses conditional probabilities and maximum a posteriori decoding:The conditional entropy $H(X\mid Y)$ instead averages remaining surprise. A security argument must specify the observer's information and the protocol's required guarantee; neither unconditional min-entropy nor Shannon entropy supplies those assumptions.

For independent and identically distributed random variables with probability mass function $p$ on a finite alphabet, evaluate their realized strings under a fixed product model $q_n(x^n)={\prod }_{i}q(x_i)$. If $q$ is positive on the probability support of $p$, the strong law of large numbers givesalmost surely, using cross-entropy equals entropy plus relative entropy. If $q$ vanishes at an outcome of positive $p$ probability, that outcome eventually appears almost surely, making all subsequent product likelihoods zero and the limit $-\infty$. This matches infinite cross-entropy.

Suppose a binary prefix code has integer codeword lengths ${\ell }_i=-{\log }_2{q}_i$. Under a source with probability mass function $p_i$, its average codeword length is exactly cross-entropy, ${\sum }_i{p}_i{\ell }_i=H(P,Q)$. The Kraft inequality is satisfied because ${\sum }_i{2}^{-{\ell }_i}={\sum }_i{q}_i=1$. This exact interpretation requires suitable dyadic rational probabilities; a general $-{\log }_2{q}_i$ is not an integer codeword length. For positive $q_i$, a prefix code with lengths $\lceil -{\log }_2{q}_i\rceil$ exists by the Kraft inequality and has average length between $H(P,Q)$ and $H(P,Q)+1$. The ceiling function causes the one-symbol rounding overhead. The cross-entropy term describes mismatch overhead; those are distinct costs.

A mixture of Poisson distributions over a nonnegative random intensity. With exposure $n>0$ and prior density $\pi (\theta )$, its probability mass function is $p_n(s)={\int }_0^{\infty }{e}^{-n\theta }(n\theta {)}^s\pi (\theta )d\theta /s!$. A gamma mixing distribution gives a negative binomial distribution. Counts sharing the same latent intensity are conditionally independent, but generally dependent after mixing.

A mixed probability distribution combines a discrete probability distribution with a continuous probability distribution. For example, put probability $1/4$ at zero and probability $3/4$ into a uniform distribution on $(0,1)$. Its cumulative distribution function is $0$ for $x<0$, $1/4+3x/4$ for $0\le x<1$, and $1$ for $x\ge 1$. The jump at zero and the continuous increase both contribute; neither an ordinary probability density function alone nor a probability mass function alone describes the whole law.

A mixture model represents a population probability distribution by choosing one mixture component at random and then drawing from its probability distribution. For a finite mixture model whose mixture components have probability density functions $f_j$, the resulting mixture distribution has probability density functionThe mixture weights ensure $f\ge 0$ and $\int f(x)dx={\sum }_j{\pi }_j=1$. For discrete random variables the same formula uses probability mass functions; without either kind of density, write $\mathbb{P}(X\in A)={\sum }_j{\pi }_j{P}_j(A)$ for component distributions $P_j$. A zero mixture weight makes that mixture component inactive.^[ref]

The mixture components need not occupy disjoint bins of $x$. Two normal distributions both have positive probability density functions everywhere, so the same value can plausibly come from either mixture component. The Two-Gaussian mixture example makes this overlap visible, and its mixture responsibilities quantify the remaining uncertainty about the latent variable. This is useful when a single normal distribution cannot represent the population's shape, but a fitted mixture model alone does not establish that its mixture components are real biological or social groups.

A mode is a value at which a probability mass function or specified probability density function attains its largest value. There may be more than one. For the normal distribution the unique mode equals the mean; the Maxwell speed distribution has mode $\sqrt{2k_{B}T/m}$ and a larger mean speed. For a continuous probability distribution, a mode maximizes density rather than the probability of an exact value, which is zero. Density modes can change under a nonlinear change of variables, as the distinction between Maxwell-Boltzmann velocity distribution and Maxwell speed distribution illustrates.

For $N\sim \mathrm{Pois}(\lambda )$ with $\lambda \ge 0$, insert its probability mass function into the expected value:The exponential series shows finiteness for every real $t$. Differentiating this moment-generating function gives mean and variance both $\lambda$. Taking the logarithm gives the cumulant-generating function $\lambda (e^t-1)$, so every positive-order cumulant is also $\lambda$.

For independent and identically distributed observations from a probability mass function $p$ on a finite alphabet, expected relative entropy of the empirical distribution from $p$ is nonincreasing in sample size. The leave-one-out identity for empirical distributions and convexity of relative entropy in its first argument giveEvery deletion sample has the law of $n$ observations from $p$. Taking expected values proves the assertion without requiring independence between deletion samples. Zero-probability symbols are absent almost surely. Since $\mathbb{E}{\hat{p}}_n=p$, the identity $\mathbb{E}D({\hat{p}}_n\Vert p)=H(p)-\mathbb{E}H({\hat{p}}_n)$ also shows that expected empirical information entropy increases towards the source information entropy. Convergence follows from the strong law of large numbers for empirical frequencies and boundedness and continuity of information entropy on a finite probability simplex.

If $(N_1,\dots ,N_k)$ has a multinomial distribution with $n$ trials, group its categories into disjoint nonempty sets $G_1,\dots ,G_m$ covering all categories. The grouped counts in the displayed equation have the multinomial distributionEach original trial belongs to exactly one group, its group probability is the sum of its category probabilities, and grouping preserves independence between trials. This gives the rule without summing the original probability mass function over every possible internal count pattern.

For the four-trial example with probabilities $(1/2,3/10,1/5)$, merge the last two categories. Their total $M=N_2+N_3$ has the binomial distribution $\mathrm{Bin}(4,1/2)$, so $\mathbb{E}M=2$ and $\mathrm{Var}(M)=1$. The same variance follows from the variance of a sum:Ignoring the negative covariance would give $1.48$, although the two counts share a fixed total. Multinomial aggregation lets the resolution of category labels change while retaining a consistent model for the counts.