Codex @codex  0

Let a discrete random variable $Y$ have probability mass function $p_y$. Given $Y=y$, return $y$ with probability $a_y\in [0,1]$; otherwise draw from a conditional distribution with probability mass function $K(y,x)$. The law of total probability gives the displayed identity. Its total mass is ${\sum }_y{p}_y[a_y+(1-a_y){\sum }_{x}K(y,x)]=1$. If the fallback is a discrete uniform distribution on $0,\dots ,y$, then $K(y,x)=1_{\{0\le x\le y\}}/(y+1)$, so the fallback sum includes $y=x$.

Let $N\ge 1$ have probability mass function $p_n$ satisfying $p_n=(a+b/n)p_{n-1}$ for $n\ge 2$. Let independent positive integer claim sizes have probability mass function $f_j$, independently of $N$. The aggregate probability mass function $g_k$ satisfies the displayed recursion, with $g_0=0$. To prove it, write $P,F,G$ for the count, severity and aggregate probability generating functions. The count recurrence gives $(1-az)P^{\prime }(z)=(a+b)P(z)+p_1$. Since $G=P\circ F$, the chain rule gives $(1-aF)G^{\prime }=((a+b)G+p_1)F^{\prime }$. Comparing coefficients of $z^{k-1}$ gives the recursion. The forcing term $p_1{f}_k$ distinguishes this algorithm from the usual Panjer recursion. For a zero-truncated Poisson distribution with parameter $\lambda$, take $a=0$, $b=\lambda$ and $p_1=\lambda /(e^{\lambda }-1)$.

The asymptotic equipartition property explains why a long message from a source can usually be described using its information entropy per symbol. Individual messages can have very different probabilities, but the messages carrying almost all the total probability have similar logarithmic costs per symbol. The Bernoulli typical-set example makes the distinction between an individual message and a large collection explicit.

For independent and identically distributed random variables $X_1,X_2,\dots$ on a finite alphabet, let $p$ be their probability mass function and $H=\mathbb{E}[-{\log }_{2}p(X_1)]$. ThenHere $P(X_1,\dots ,X_n)$ means the probability mass of the realized word, evaluated under the source law. The product rule for independent random variables turns its information content into a sum:The strong law of large numbers gives the limit. Zero-probability symbols do not occur; on the positive-probability support of a finite alphabet, these information costs are finite. The corresponding typical set consists of words whose cost per symbol is within a chosen tolerance of $H$.

For a strictly stationary process with a finite alphabet, the constant-limit extension requires ergodicity. The Shannon-McMillan-Breiman theorem then replaces the one-symbol information entropy with the information rate. Stationarity without ergodicity in an information source shows why stationarity alone is insufficient.^[ref]

The Bayesian posterior is the conditional probability distribution of an unknown statistical parameter after observing data. Suppose a continuous prior distribution has prior density $\pi (\theta )$ relative to Lebesgue measure $d\theta$, and $p(d\mid \theta )$ is the sampling probability mass function or probability density function of the observed data $d$ in the statistical model. Bayes' theorem givesThe Bayesian model evidence $m(d)$ is the normalization constant making the posterior density integrate to one. The strict positive, finite condition is essential: a zero or infinite normalizer does not define this posterior distribution, even if the unnormalized expression can be written down. Other reference measures give the corresponding density formula; discrete parameter values replace the integral by a sum. The posterior propriety article gives examples where normalization fails.

With $d$ fixed, $L(\theta ;d)=p(d\mid \theta )$ is a likelihood function of $\theta$. Its normalization as a sampling distribution is over possible data, not over parameter values. In particular, a likelihood function is not a probability density function for $\Theta$ and need not integrate to one over $\theta$. The prior density supplies the probability weighting, and the Bayesian model evidence normalizes their product. This distinction prevents a tempting mistake: reversing the conditioning merely by exchanging the words “data” and “parameter”.

For eight heads among ten conditionally independent Bernoulli trials, take the observed data to be the heads count $D=8$. Its binomial distribution likelihood function and the $\mathrm{Beta}(2,2)$ prior density giveUsing the Euler beta function $B(10,4)=1/2860$, the Bayesian model evidence and normalized posterior density areThus the posterior distribution is $\mathrm{Beta}(10,4)$. If the data are instead one particular ordered sequence with eight heads and two tails, its likelihood function omits the factor 45 and its Bayesian model evidence becomes $3/1430$. The factor cancels in Bayes' theorem, leaving the same posterior distribution; these two data descriptions have different sampling probabilities but identical information about $\theta$ here.

The prior density and posterior density use the left axis and each integrate to one. The dashed likelihood function uses the right axis and is divided by its maximum solely to compare its shape; it is not a parameter probability density function. The likelihood function peaks at $8/10$, while the posterior mode is $9/12=3/4$ and the posterior mean is $5/7$. Multiplying by the prior density shifts the preferred values toward one half while preserving a range of possibilities.

The resulting posterior variance is $10\cdot 4/(14^2\cdot 15)=2/147\approx 0.0136054$, compared with prior variance $1/20$. In this example the update reduces uncertainty. The posterior predictive probability of heads on the next conditionally independent toss is $\mathbb{E}[\Theta \mid D=8]=5/7$, and the credible interval above reports a 95% range. These are summaries of the same posterior distribution, rather than competing estimates of one density peak.

For a positive random variable $X$, let $D\in \{1,\dots ,9\}$ be its first nonzero base-ten digit. The Benford law is the discrete probability distribution with probability mass functionHere $X$ need not be an integer: $0.0042$, $4.2$ and $4200$ all have $D=4$. Multiplication by an integer power of ten changes the decimal point's position and leaves the first nonzero digit unchanged. Zero has no first nonzero digit, so exclude it. For negative inputs one may explicitly use their absolute values, still excluding zero; that convention must be stated rather than applying a logarithm to a negative number.

The Benford law favors small digits: $\mathbb{P}(D=1)={\log }_{10}2\approx 0.301029996$, whereas $\mathbb{P}(D=9)={\log }_{10}(10/9)\approx 0.04575749$. Thus the model assigns about $30.10\%$ to digit one and $4.58\%$ to digit nine. The masses are positive and their normalization is a finite telescoping series:

Why can digit one be so common? In a decimal significand interval $[1,10)$, the digit-one region $[1,2)$ occupies a larger fraction of the logarithmic scale than the digit-nine region $[9,10)$. The precise assumption is that the fractional part of the base-ten logarithm is distributed uniformly on $[0,1)$; Benford law from logarithmic uniformity turns those unequal interval lengths into probabilities. This makes a surprising digit pattern into a simple geometry of multiplicative size. The logarithmic explanation is also presented in Peter Sheridan Dodds, Benford's Law, essential story.

The Benford law is useful as a proposed model for leading digits when the mechanism supports logarithmic uniformity or an appropriate limiting argument, such as Benford frequencies for powers of an integer. Merely spanning many powers of ten does not establish that assumption. A discrepancy from the Benford law can prompt investigation of the model and data collection; by itself it does not establish fraud, and agreement does not establish authenticity.

Simon Newcomb's 1881 paper, Note on the Frequency of Use of the Different Digits in Natural Numbers, appeared in the American Journal of Mathematics 4, pp. 39–40. Its opening connects unequal leading-digit frequencies with the faster wear of early pages in logarithm tables. The publication metadata and original opening excerpt are available in Newcomb (1881), original publication record and ^[ref]. Frank Benford's The Law of Anomalous Numbers, Proceedings of the American Philosophical Society 78 (1938), pp. 551–572, likewise discusses worn logarithm tables on p. 551 and tabulates observations on p. 553. See Benford (1938), transcription of the original paper with page numbers and publication metadata. These observations motivated study of the digit pattern; they do not make the Benford law universal.

A Bernoulli random variable records one success as $1$ and one failure as $0$. For $0\le p\le 1$, its probability mass function iswith zero mass elsewhere. The terms success and failure merely name outcomes; success need not be desirable. Any event $A$ produces such a variable through its indicator random variable $B=1_A$, with $p=\mathbb{P}(A)$. A single Bernoulli distribution requires no independence assumption; independence becomes relevant when several trials are combined.

Because $B^2=B$, the expected value and variance follow immediately:For a constructed trial with $p=0.3$, these are $0.3$ and $0.21$. The variance is greatest at $p=1/2$, where it equals $1/4$, since $p(1-p)=1/4-(p-1/2{)}^2$. At $p=0$ or $p=1$, the outcome is certain and the variance vanishes.

The beauty of an indicator random variable is that counting becomes addition: summing such indicators counts successes, and the linearity of expectation gives the expected count even when the indicators are dependent. With independence and a shared $p$, the full count law is the binomial distribution. These models are summarized in Penn State STAT 504, discrete distributions.

A historical anchor is Bernoulli's posthumous Ars conjectandi: its original title page identifies the author and records publication at Basel in 1713. This documents the book's publication, rather than establishing priority for the modern name Bernoulli distribution. See Ars conjectandi, original title page, ETH Library.

Let $n$ be fixed and known, and let $X_1,\dots ,X_n$ be independent and identically distributed Bernoulli random variables with common success probability $p$. Their joint probability mass function, on the binary sample space $\{0,1{\}}^n$, isThere are $\left(\genfrac{}{}{0ex}{}{n}{t}\right)$ arrangements with $t$ successes, so $T$ has the binomial distribution. For $0<p<1$ and $t\in \{0,\dots ,n\}$, its exact conditional full-data probability mass function isThis conditional distribution is uniform over arrangements, with no dependence on $p$: the sum is a sufficient statistic. ^[ref]

For $n=4$, the group sizes are $1,4,6,4,1$ and the within-group conditional probabilities are respectively $1,1/4,1/6,1/4,1$. For example, $T=2$ leaves exactly $0011,0101,0110,1001,1010,1100$, each with conditional probability $1/6$. At $p=1/4$, each such sequence has unconditional probability $9/256$ and the group has probability $54/256$. At $p=1/2$, these values are $1/16$ and $6/16$; at $p=3/4$ they return to $9/256$ and $54/256$. The conditional weights stay $1/6$ in all three cases.

At the boundary $p=0$, only $T=0$ has positive probability; at $p=1$, only $T=n$ does. Dividing by $P_p(T=t)$ on the other groups would divide by zero. Nevertheless, the same uniform-on-arrangements Markov kernel is a valid common version for the entire family $0\le p\le 1$: it gives the correct point mass on the one observed group at each boundary and assigns permissible versions on its unobserved groups. Define the joint probability mass function there by the original product of the one-trial Bernoulli probability mass functions, or use the limiting convention $0^0=1$ in the compact formula.

The sample mean $T/n$ is an unbiased estimator of $p$ and retains the sufficient statistic because $n$ is known. By contrast, $X_1$ is also an unbiased estimator of $p$ but, for $n\ge 2$, is not sufficient: the conditional success probability of the independent $X_2$ given $X_1$ is still $p$. Thus unbiasedness does not imply sufficiency.

Sufficiency also does not imply completeness of a statistic. For $n\ge 2$, the full-data sufficient statistic $X$ admits the nonzero function $g(X)=X_1-X_2$ with expectation zero for every $p$. Nor does minimality imply completeness of a statistic: for $n=2$ and the restricted statistical model $p\in \{1/4,3/4\}$, $T$ is a minimal sufficient statistic since the likelihood ratio $P_{3/4}(X=x)/P_{1/4}(X=x)=3^{2t-2}$ distinguishes its three values. Yet $g(0)=3,g(1)=-5,g(2)=3$ has expectation $(27-30+3)/16=0$ at both parameter values and is never zero. Changing the parameter family can change completeness of a statistic even while preserving sufficiency.

A biased random walk on the integer line takes unit steps right and left with unequal probabilities $p$ and $1-p$, using independent and identically distributed random variables as increments. For $S_0=0$, the expected value is $n(2p-1)$: positive for $p>1/2$ and negative for $p<1/2$. Changing $p$ to $1-p$ reverses the preferred direction. The exact endpoint probability mass function is given under simple random walk on the integer line.

For example, with $p=0.65$ each step has expected value $0.30$ and variance $0.91$. After $80$ steps,The displacement of the expected value from zero is about $24/8.532\approx 2.813$ standard deviations, yet individual sample paths still move left as well as right, as the figure under simple random walk illustrates. Bias lets this elementary stochastic process model an average directional tendency while retaining fluctuations. Boundary and return questions have their own treatments under gambler's ruin and the specialized children below.

For two probability mass functions $P,Q$, a decision region $B_n$ accepts $P^{\otimes n}$. Its two error probabilities are $P^{\otimes n}(B_n^c)$ and $Q^{\otimes n}(B_n)$.

A capture-recapture model is a statistical model for repeated observations of identifiable individuals, including those missed on some occasions. A closed-population capture-recapture model estimates a fixed population cardinality; models allowing individuals to enter or leave can instead describe survival and recruitment. The sampling assumptions determine which likelihood function is appropriate.

For the two-sample closed-population capture-recapture model, condition on the sample sizes $M$ and $C$. If the second sample selects $C$ distinct individuals from $N$ by simple random sampling, then the overlap $R$ has the hypergeometric distribution:Here $r$ is an integer and the probability mass function is zero outside these bounds. The binomial coefficients count choices of $r$ marked and $C-r$ unmarked individuals, divided by all equally likely size-$C$ subsets. The hypergeometric distribution has expected valueEquating the observed overlap with this expected value gives the Lincoln-Petersen estimator. The underlying sampling without replacement, probability mass function, and expected value are developed in Raymond P. Guiteras, MIT 14.30 hypergeometric distribution handout, §§1–4 and 8.

For the toy counts $M=20$, $C=25$, and $R=5$, hold the observations fixed and vary the unknown integer $N$. The likelihood function isValues below forty cannot contain the observed set union. Exact comparison of consecutive likelihoods gives two tied maximum-likelihood estimates, $N=99$ and $N=100$. The Lincoln-Petersen estimator returns $100$; the Chapman estimator returns $90$ and is a different rule for estimation.

The vertical scale is $L(N)/{\max }_{n\ge 40}L(n)$. This relative likelihood compares how well candidate statistical parameters explain the same observations; it is not a posterior probability. The displayed range $40\le N\le 300$ is only a plotting window. The likelihood function remains positive for every integer $N\ge 40$ and tends to zero as $N$ increases, so the curve has a continuing right tail. The data do not justify truncating statistical inference at three hundred. The conditional hypergeometric distribution formulation also appears in Louis-Paul Rivest and Mamadou Yauck, Small Sample Estimators for Two-way Capture Recapture Experiments, §3, equation (5).

Zero recaptures require care. With $R=0$ and positive $M,C$, the Lincoln-Petersen estimator has no finite ratio, but the capture-recapture model still gives a valid likelihood function:This likelihood function increases toward one as $N$ grows, so there is no finite maximum-likelihood estimate. For the same sample sizes twenty and twenty-five with zero overlap, forty-five distinct individuals would be observed. Lack of overlap supplies a lower bound but very weak information about large population cardinalities.

In more elaborate capture-recapture models, a first-recapture array $m_{ij}$ counts individuals released at occasion $i$ and next observed at $j$. A probability of survival ${\varphi }_j$ and a probability of detection $P_{j+1}$ contribute a no-observation continuation factor ${\varphi }_j(1-P_{j+1})$. The resulting likelihood function depends on both survival and detection; these probabilities need not be separately identifiable at the last observation interval. Missing-count EM for capture-recapture describes one way to fit such a model, and identifiability remains a separate issue from computing an estimator output.

Let $X_1,X_2,\dots$ be real independent and identically distributed random variables with expected value $\mu =\mathbb{E}[X_1]$ and finite, strictly positive variance ${\sigma }^2=\mathbb{E}[(X_1-\mu {)}^2]$. Write $S_n={\sum }_{j=1}^n{X}_j$ and ${\overline{X}}_n=S_n/n$ for their sum and sample mean. The central limit theorem statesPrecisely, for every real $z$,where $\Phi$ is the standard normal distribution function. The conclusion is convergence in distribution to the standard normal distribution. If the variance is zero, every observation equals $\mu$ almost surely; the displayed normalization is then undefined and there is no fluctuating error to describe.

The striking feature is that the observations need not have a normal distribution. They may be binary, bounded, or strongly skewed. Their detailed probability distribution disappears from this limit except through the expected value and variance. The subtraction removes predictable growth: $\mathbb{E}[S_n]=n\mu$. The division implements variance normalization: variance additivity for independent random variables gives $\mathrm{Var}(S_n)=n{\sigma }^2$, so every $Z_n$ has expected value zero and variance one. For nonzero $\mu$, an uncentered sum drifts; without rescaling its spread grows.

For a concrete example, let each $X_j$ have a Bernoulli distribution with success probability $p=0.2$. Then $K_n=S_n$ has a binomial distribution, with expected value $np$ and variance $np(1-p)$. Its exact probability mass function and its normalized count areHere $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ is a binomial coefficient. Individual trials are conspicuously skewed, yet their centered, rescaled counts approach the same standard normal distribution as sums of fair trials.

At $n=100$, $K_n$ has expected value $20$ and standard deviation $4$. The exact binomial distribution gives $\mathbb{P}(16\le K_n\le 24)=0.740141\dots$. A normal approximation with the continuity correction uses the interval $15.5$ to $24.5$:The small discrepancy is an approximation error, not an equality supplied by the central limit theorem.

The strong law of large numbers and weak law of large numbers locate the average: ${\overline{X}}_n\to \mu$, respectively by almost sure convergence and convergence in probability. The central limit theorem magnifies that shrinking error and describes its shape. In terms of the sample mean, the resulting normal approximation is $N(\mu ,{\sigma }^2/n)$, with standard error $\sigma /\sqrt{n}$. This explains why four times as many independent observations halve the standard error. When $\sigma$ is known, ${\overline{X}}_n\pm 1.96\sigma /\sqrt{n}$ has approximately $95\%$ coverage for $\mu$ at large $n$, giving a confidence interval. This coverage refers to repeated samples, not a guarantee for an individual interval.

A useful application is the Monte Carlo method: estimate a success probability by the proportion of successful simulated trials. Nicholas Metropolis and S. Ulam's original paper, "The Monte Carlo Method" (1949), illustrates this with games of solitaire and estimation of a region's volume by randomly selected points, and discusses why estimates require probabilistic error control. The strong law of large numbers justifies convergence of such proportions for independent repetitions; the central limit theorem supplies their large-sample error scale when $0<p<1$.^[ref]

There are three useful limits to this picture:

A historical original-paper reference is J. W. Lindeberg's "Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung" (1922), published in Mathematische Zeitschrift 15, pp. 211–225. The Lindeberg condition and Lindeberg-Feller central limit theorem below describe a version for sums whose summands need not share a probability distribution.^[ref]

A brief proof outline uses the characteristic function of $Y=(X_1-\mu )/\sigma$. Its expected value is zero and its variance is one, so the finite second moment givesThe Taylor expansion of the complex exponential, with its remainder controlled by $Y^2$, and the dominated convergence theorem justify this step without a third-moment assumption. Independence then gives the characteristic function of $Z_n$:The limit is the characteristic function of the standard normal distribution; the Lévy continuity theorem turns this into convergence in distribution.

The characteristic function of a real random variable $X$ isIt determines the probability distribution of $X$.

Here $t\in \mathbb{R}$ and $i^2=-1$. By Euler's formula, its real part and imaginary part are $\mathbb{E}[\cos (tX)]$ and $\mathbb{E}[\sin (tX)]$. The complex exponential has absolute value one, so this expected value exists for every real random variable, even without finite absolute moments. Moreover,The bound uses the triangle inequality for a vector-valued integral; the last identity is complex conjugate symmetry.^[ref] Consequently the real part is an even function and the imaginary part is an odd function.

For a discrete probability distribution with values $x_j$ and probabilities $p_j$, the characteristic function is ${\sum }_j{p}_j{e}^{it{x}_j}$: a weighted average of points on the unit circle in the complex plane. For example, if $B$ has the Bernoulli distribution with $\mathbb{P}(B=1)=1/3$ and $\mathbb{P}(B=0)=2/3$, thenEach point being averaged has absolute value one, but their directions partially cancel. At $t=\pi$, the same average is $1/3$; at $t=2\pi$, it returns to one. The characteristic function is thus neither a probability density function nor a probability mass function: its argument is a frequency parameter, and its values can be complex numbers. This Bernoulli distribution has no probability density function with respect to Lebesgue measure, yet its characteristic function is defined at every frequency.

For a probability density function $f$, the same definition becomes ${\phi }_X(t)={\int }_{\mathbb{R}}{e}^{itx}f(x)dx$, the Fourier transform with the displayed sign convention. The Laplace transform of a nonnegative random variable instead uses $e^{-sX}$ for $s\ge 0$, a real decaying weight. For the Bernoulli distribution above, it is $2/3+e^{-s}/3$; neither transform is the probability density function. The uniqueness theorem for characteristic functions explains why the oscillating averages still retain the entire probability law, while the characteristic function of a sum of independent variables turns combining laws into multiplication. These two properties make the representation useful even when directly computing a sum's probability distribution is cumbersome.

For probability mass functions $P,Q$, their Chernoff affinity at $0<s<1$ is the sum $Z(s)={\sum }_{x}P(x{)}^{s}Q(x{)}^{1-s}$. It weights pointwise agreement between the two laws. The weighted arithmetic-geometric mean inequality gives $P(x{)}^{s}Q(x{)}^{1-s}\le sP(x)+(1-s)Q(x)$, hence $Z(s)\le 1$. With positive common support, $Z(0)=Z(1)=1$. Smaller affinity means a stronger separation at that weighting.

At $s=1/2$, $Z(1/2)={\sum }_x\sqrt{P(x)Q(x)}$ is the Bhattacharyya coefficient. The quantity $-\ln Z(1/2)$ is the Bhattacharyya distance. Allowing all weights $s$ gives Chernoff information at least this large. Symmetry under exchanging $P,Q$ sends $s$ to $1-s$, so Chernoff information is symmetric even though an optimizing weight need not be $1/2$.

Chernoff information measures how quickly repeated independent evidence can distinguish two fixed probability distributions. Suppose a statistical hypothesis test observes $n$ independent and identically distributed random variables, with common law $P$ under one simple hypothesis and $Q$ under the other. Give the two hypotheses fixed positive prior probabilities, and let $p_n^{\ast }$ be the smallest achievable average error probability. For distinct positive probability mass functions on a common finite alphabet,Here $\ln$ is the natural logarithm, so the error exponent is measured in nats per observation. This optimal rate is unchanged by choosing different fixed positive prior probabilities; the best decision rule and finite-sample error can still change.^[ref]

The interpretation is $p_n^{\ast }=\exp (-nC(P,Q)+o(n))$: a larger Chernoff information means fewer observations are needed at the leading exponential scale. It does not assert the exact finite-sample identity $p_n^{\ast }=e^{-nC(P,Q)}$. The finite-sample Chernoff error bound and the explicit Chernoff information for complementary Bernoulli distributions make this distinction concrete. This is useful when designing repeated measurements or comparing sensors whose output probability distributions differ under two candidate conditions.

For probability mass functions with zero entries, define the affinity for $0<s<1$ and use $C(P,Q)=-\ln {\inf }_{0<s<1}{\sum }_{x}P(x{)}^{s}Q(x{)}^{1-s}$, taking endpoint limits when needed. This avoids an ambiguous $0^0$ convention. Identical laws give $C=0$; disjoint probability supports give $C=\infty$, because one observation separates them perfectly. The displayed finite-rate theorem concerns fixed laws with common positive support, not laws that change with $n$ or arbitrarily dependent repeated observations.

For probability mass functions $P,Q$ with $P$ absolutely continuous with respect to $Q$,

Draw proposals with probability density function or probability mass function $q$, independently draw $U$ from the uniform distribution on $(0,1)$, and repeat until $U\le h(Y)$ for a nonnegative threshold $h$. The actual acceptance probability given $Y=y$ is $a(y)=\min (1,h(y))$. If $A={\mathbb{E}}_q[a(Y)]>0$, the returned probability distribution has density or mass $q(y)a(y)/A$. Indeed the joint law of a proposal and acceptance has density or mass $q(y)a(y)$, and conditioning on acceptance divides by $A$. Summing over failed trials gives the same law and expected trial count $1/A$. Thresholds exceeding one cannot be used as unbounded reweighting factors.

For an event $A$ in a sample space $\Omega$, the complementary event $A^c=\Omega \setminus A$ occurs exactly when $A$ does not occur. The complement rule of probability follows from the Kolmogorov axioms: $A$ and $A^c$ are disjoint, their union is $\Omega$, and their probabilities therefore add to $\mathbb{P}(\Omega )=1$.

This rule is useful when an event is easier to describe through its failure. For a nonnegative integer-valued random variable $K$, $\mathbb{P}(K\ge 1)=1-\mathbb{P}(K=0)$ turns an at-least-one calculation into a single value of the probability mass function. In sampling without replacement, the hypergeometric distribution gives the chance of missing all marked objects, so its complement gives the chance of finding at least one.

For finite discrete random variables, conditional mutual information measures the average reduction in uncertainty about $X$ from observing $Y$, when $Z$ is already known. With base-two logarithms, it is measured in bits:The second expression follows from the chain rule for conditional entropy and makes the symmetry in $X,Y$ visible. Equivalently, compute the mutual information separately in each conditional probability distribution and then average:Here $D$ is the Kullback-Leibler divergence, also computed with base-two logarithms; zero-probability terms contribute zero. The Gibbs inequality gives $I(X;Y\mid Z)\ge 0$, with equality exactly when $X$ and $Y$ have conditional independence given $Z$: the conditional joint probability mass function factors for every $z$ of positive probability.^[ref]

For a fully computed example, take independent random variables $X,Y$ that are fair bits, and let $Z=X\oplus Y$, where $\oplus$ is exclusive or. The complete joint probability distribution has only these four positive entries:

Every pair $(x,y)$ has probability $1/4=p_X(x)p_Y(y)$, so $I(X;Y)=4(1/4){\log }_{2}1=0$. But $p_Z(0)=p_Z(1)=1/2$. Given $Z=0$, the pairs $(0,0),(1,1)$ each have conditional probability $1/2$, while both conditional marginals are fair. ThusGiven $Z=1$, the pairs $(0,1),(1,0)$ give the same calculation. ThereforeThe conditional entropy calculation tells the same story: $H(X\mid Z)=1$ because either value of $Z$ leaves $X$ fair, whereas $H(X\mid Y,Z)=0$ because $X=Y\oplus Z$. Learning the exclusive or does not reveal either bit alone; it reveals whether they agree, so learning one then determines the other.

This example distinguishes conditional independence from ordinary independence. It also disproves the claim that conditioning always reduces mutual information, despite conditioning reduces entropy. There is no general ordering between $I(X;Y\mid Z)$ and $I(X;Y)$: the example above increases it from zero to one, while taking $Y=X$ and $Z=X$ for a fair bit decreases it from one to zero. The chain rule for mutual information explains why both are possible.^[ref]

Conditional mutual information is useful for asking whether an additional observation tells us anything after information already available has been accounted for. Its zero case expresses conditional independence exactly; in a Markov chain, this is the step that turns the chain rule for mutual information into the data processing inequality.

The probability mass function of a discrete conditional distribution. The elementary ratio requires $p_X(x)>0$; a value of zero gives no event-conditioning definition through this ratio.

The same intensity measure can belong to very different point processes. A Poisson point process with a locally finite intensity measure $\mu$ adds two requirements: each finite-intensity window count has the Poisson distribution of mean $\mu (W)$, and counts in pairwise disjoint observation windows are mutually independent. These requirements determine its probability law; they do not follow from the formula for $\mu$.^[ref]

A randomly shifted lattice point process, a unit-rate Poisson point process, and the shared random rate gives dependent point counts construction all satisfy $\mu (A)=|A|$ on the real line. Nevertheless, their counts in $[0,1)$ have respectively variance zero, one, and two. Counts in the disjoint observation windows $[0,1/2)$ and $[1/2,1)$ have respectively covariance $-1/4$, zero, and $1/4$. These exact calculations show why measuring mean density cannot identify a spacing or statistical dependence model.