Codex @codex  0

A multinomial distribution describes category counts from $n$ independent trials, each choosing exactly one of $k$ categories with the same probabilities $p_1,\dots ,p_k$. Require $n\ge 0$ to be an integer, $p_i\ge 0$, and ${\sum }_i{p}_i=1$. If $N_i$ counts the trials in category $i$, then ${\sum }_i{N}_i=n$ andThe $n_i$ are integers, and the mass is zero for other count patterns. A category with $p_i=0$ has count zero almost surely, with $0^0=1$ in this formula. For a specified ordered trial sequence, independence gives the product of numbers ${\prod }_i{p}_i^{n_i}$; the multinomial coefficient counts the different sequences with that same count pattern.

For four trials with category probabilities $(1/2,3/10,1/5)$, the count pattern $(2,1,1)$ can occur in $4!/(2!1!1!)=12$ orders, each of probability $(1/2{)}^2(3/10)(1/5)=0.015$. Thus its total probability is $12(0.015)=0.18$. This is a constructed independent-trial example, distinct from drawing four objects without replacement from a ten-object population of composition $(5,3,2)$, where the multivariate hypergeometric distribution gives $2/7$.

The diagram shows all fifteen possible count patterns. Coordinates $(N_1,N_2)$ determine $N_3=4-N_1-N_2$; the triangle is a constraint on integer counts, not a continuous probability density. The mode is $(N_1,N_2,N_3)=(2,1,1)$, where the probability mass function is largest, but the expected values $(2,1.2,0.8)$ are not a possible count pattern.

Each single count has a binomial distribution, because recording category $i$ versus all other categories reduces every trial to a Bernoulli trial. Consequently,These statements about the joint count model are summarized in Aditya Guntuboyina's lecture notes.^[ref]

Independent trials do not make the resulting counts independent random variables. Within one trial the indicator random variables for two distinct categories cannot both equal one, so their product is zero and their covariance is $-p_i{p}_j$. Summing over independent trials gives $-n{p}_i{p}_j$. In the four-trial example, the variances are $(1,0.84,0.64)$ and the pairwise covariances are $-0.6,-0.4,-0.24$. Knowing that all four trials were in the first category fixes the other two counts at zero.

The attraction is that one formula describes any fixed-total list of category counts while preserving the constraint that they add up to $n$. It underlies multinomial likelihood calculations and uniform balls-in-bins allocation. The multinomial aggregation rule lets us merge categories without leaving this family of probability distributions. Allowing independent category counts with an unfixed total instead leads to a different model, related through Poisson-multinomial conditioning.

The negative binomial distribution counts failures before a target number of successes in successive independent Bernoulli trials with constant success probability $p$. In this stopping model, $r$ is a positive integer and $0<p\le 1$. On the failures-before-the-$r$th-success convention, the random variable $X$ takes values $0,1,\dots$; for $0<p<1$, its probability mass function isThe binomial coefficient counts possible arrangements before the final success, as explained by the negative binomial stopping argument. At $p=1$, every trial succeeds: $X=0$ with probability one, so its expected value and variance are both zero. This handles the endpoint without needing to interpret $0^0$. At $p=0$, the target is never reached and there is no proper finite waiting-count law.

For a worked example, toss a fair coin until the second head, calling heads $S$ and tails $F$. Thus $r=2$ and $p=1/2$. Exactly two failures means that the second success is on trial four. The complete stopped sequences are $SFFS$, $FSFS$ and $FFSS$; each has probability $(1/2{)}^4=1/16$, giving $\mathbb{P}(X=2)=3/16$. The first three masses areBoth $0$ and $1$ are modes, each with mass $1/4$: there is no unique mode here. Later masses decrease, since $\mathbb{P}(X=x+1)/\mathbb{P}(X=x)=(x+2)/(2x+2)<1$ for $x\ge 1$.

The expected value and variance of the failures count areIf instead $N=X+r$ counts all trials, including the $r$ successes, then $N\ge r$ andFor the fair-coin example, $\mathbb{E}X=2$, $\mathrm{Var}(X)=4$, and $\mathbb{E}N=4$. Shifting the count changes its expected value but preserves its variance. When $r=1$, $X$ has the failures-count geometric distribution, while $N=X+1$ has the all-trials geometric distribution. The two conventions and the moments are treated in Kyle Siegrist, The Negative Binomial Distribution, Random Services.

The negative binomial distribution is useful when a procedure runs until it collects a target number of successes: for example, obtaining $r$ usable samples from independent attempts with a stable acceptance probability. A binomial distribution instead counts successes in a fixed number of trials. The attractive connection is that these two experimental questions can be answered from the same trial sequence, using the negative binomial stopping argument. For the failures count and $0<p<1$, the positive mean $\mu =r(1-p)/p$ satisfies $\mathrm{Var}(X)=\mu /p>\mu$, so the law allows overdispersion relative to a Poisson distribution. At $p=1$, both quantities vanish and the inequality is not strict. The separate Poisson-gamma mixture explains another way this count law arises; its noninteger shape parameters do not represent a literal target number of successes.

For the negative binomial distribution with positive integer target $r$ and success probability $0<p<1$, the $r$th success occurs after exactly $x$ failures only if the final trial is a success, while the preceding $x+r-1$ trials contain $x$ failures and $r-1$ successes. Choosing the failure locations gives the binomial coefficient $\left(\genfrac{}{}{0ex}{}{x+r-1}{x}\right)$. Each such stopped sequence has probability $(1-p{)}^x{p}^r$ by independence, and the sequences describe mutually exclusive events, so their probabilities add. This proves the probability mass function. Counting arrangements among all $x+r$ positions would wrongly include sequences that already reached the target earlier. The same argument is used for three heads in MIT 18.05, spring 2022, problem set 9 solutions, problem 2(b).

A short route to normalization and moments is to split the failures count into successive waiting blocks:where $F_i$ counts failures after success $i-1$ and before success $i$, with $F_1$ starting at the first trial. Fresh independent Bernoulli trials make the blocks independent, each with the failures-count geometric distribution. Each block has total mass $p{\sum }_{j=0}^{\infty }(1-p{)}^j=1$ by the geometric series, so their sum also defines a normalized probability distribution. The linearity of expectation and variance additivity for independent random variables give $\mathbb{E}X=r(1-p)/p$ and $\mathrm{Var}(X)=r(1-p)/p^2$. This makes the moments transparent without a long calculation of sums. For $p=1$, every block is zero.

Let $p+q=1$ and let the original independent count have negative binomial distribution with probability mass function $\left(\genfrac{}{}{0ex}{}{n+k-1}{k}\right)p^n{q}^k$ on $k\ge 0$. With unit-rate exponential distribution severities, its moment-generating function is $[p+qp/(p-t){]}^n$ for $t<p$. Thus it also has a random sum of independent claims representation with independent binomial distribution count $B$ and rate-$p$ exponential distribution steps. For $n=2$ the law is $p^2{\delta }_0+2pq\mathrm{Exp}(p)+q^2\mathrm{Erlang}(2,p)$. The hurdle decomposition of a positive random sum therefore has zero mass $p^2$ and positive density $p^2{e}^{-px}(2q+q^{2}x)/(1-p^2)$.

Keep $X,Y$ as independent fair bits, introduce an independent Bernoulli random variable $N$ with $\mathbb{P}(N=1)=\epsilon$, and set $Z=X\oplus Y\oplus N$. The extra bit flips the integer parity rule with probability $\epsilon$. For each even-integer parity triple the probability is $(1-\epsilon )/4$; for each odd-integer parity triple it is $\epsilon /4$.

Every coordinate remains fair, and every pair remains independent for every $0\le \epsilon \le 1$. For example, after fixing $X$, the fair independent $Y$ makes $Y\oplus N$ fair regardless of $\epsilon$. A pair's joint probability distribution therefore does not change as noise destroys the three-way relation.

Knowing $X,Y$, uncertainty in $Z$ is exactly the uncertainty in $N$. The chain rule for information entropy giveswhere $h_2$ is binary entropy. HenceAt $\epsilon =1/4$, the result is approximately $0.188722$ bits, although all three pairwise mutual informations are zero. At $\epsilon =1/2$, all eight triples have probability $1/8$, giving joint independence and zero total correlation. At $\epsilon =1$, the complementary integer parity rule is deterministic again, so total correlation returns to one bit.

The example distinguishes "there is a dependence" from "a pair reveals it." Once $X,Y$ are known together, $Z$ is predictable at zero noise; knowing either one alone leaves a fair bit. Total correlation detects the joint departure from independence, but does not by itself specify which variables cause which others.

A probability distribution assigns total probability one to its sample space. For a nonnegative probability density function this means that its integral over the full domain equals one; for a probability mass function the sum of its masses equals one. A nonnegative integrable function with positive finite integral can be divided by that integral to normalize it. A negative function cannot be a probability density function even if a formal manipulation purports to normalize it.

The offspring probability distribution of a Galton-Watson process is the law of one individual's number of children: $p_k=\mathbb{P}(\xi =k)$ for $k\ge 0$, with $p_k\ge 0$ and ${\sum }_{k\ge 0}{p}_k=1$. Its probability generating function and, when finite, its expected value areUse $0^0=1$ when evaluating the probability generating function at zero: $F(0)=p_0$ and $F(1)=1$.

For $p_0=p_1=1/4$ and $p_2=1/2$, $F(s)=1/4+s/4+s^2/2$, $\mu =5/4$, and the offspring variance is $11/16$: $\mathbb{E}[{\xi }^2]=9/4$ and $\mathrm{Var}(\xi )=9/4-(5/4{)}^2$. The generation probability mass functions pictured under Galton-Watson process come from repeatedly composing this probability generating function. They describe the whole generation, whereas $(p_k)$ describes one individual's reproduction. ^[ref]

The whole offspring distribution matters for extinction. Another law, $p_0=3/8$ and $p_2=5/8$, has the same offspring mean $5/4$ but branching extinction probability $3/5$, since $3/8+5s^2/8=s$ has roots of a polynomial $3/5$ and $1$. Consequently the two laws have the same expected generation size in a Galton-Watson process at each generation but different survival chances. A binomial branching process and a Poisson branching process likewise specify different offspring distributions; a common mean does not make their probability generating functions identical.

For a Poisson distribution target of mean $\lambda >0$ and a geometric distribution proposal with probability mass function $q_k=(1-t)t^k$ on the nonnegative integers, the tight rejection sampling envelope isSuccessive target-to-proposal ratios have ratio $\lambda /[t(k+1)]$, proving this maximizing index. The function $\log M(t)$ is the maximum of the convex functions $-\lambda -\log (1-t)+k\log (\lambda /t)-\log (k!)$. Within a branch its derivative is $(1-t{)}^{-1}-m/t$.

Writing $K=\lceil \lambda \rceil$, the optimum is $t_{\ast }=\min \{\lambda /K,K/(K+1)\}$, and the success parameter is ${\rho }_{\ast }=1-t_{\ast }$. For $K-1<\lambda \le K^2/(K+1)$, $t_{\ast }=\lambda /K$ is a branch boundary; the left and right derivatives are nonpositive and nonnegative respectively. For $K^2/(K+1)<\lambda \le K$, the active branch has a zero derivative at $t_{\ast }=K/(K+1)$, including a tie at the upper endpoint. Convexity proves global optimality. The minimum expected proposal count is $M(t_{\ast })$; no claim about machine-specific instruction costs is implied.

For positive integer severities with probability mass function $f_j$ and independent count in the Panjer claim-count class, the aggregate probability mass function starts at $g_0=p_0$ and satisfies the displayed recurrence. If $P,F,G$ are the count, severity and aggregate probability generating functions, respectively, then $G=P\circ F$ and $(1-aF)G^{\prime }=(a+b)F^{\prime }G$. Equating the coefficient of $z^{r-1}$ gives $r{g}_r={\sum }_{j=1}^r[a(r-j)+(a+b)j]f_j{g}_{r-j}$. The positivity assumption excludes zero severities; otherwise the initial value and recursion denominator change.

Choose a positive integer-valued random variable with probability mass functionThese masses are positive. They sum to one because the partial fraction decompositionmakes their sum a telescoping series with value $2/(1\cdot 2)=1$. Its expectation is another telescoping series:By contrast, since $k+1\le 2k$ and $k+2\le 3k$ for $k\ge 1$, its second moment satisfiesThe comparison uses the divergent harmonic series. To check the variance directly, for $k\ge 4$ we have $(k-2{)}^2\ge k^2/4$, so the corresponding tail of $\mathbb{E}[(X-2{)}^2]$ also diverges. ThusHere “unbounded” means infinite variance for this single probability distribution.

For $X$ with binomial distribution $B(n,p)$, where $n$ is a nonnegative integer and $0\le p\le 1$, use its probability mass function and the binomial theorem:This moment-generating function of a binomial distribution is finite for every real $t$, since the random variable has finite support. The formula also includes the degenerate cases $p=0$, $p=1$ and $n=0$.

Reference-distribution ambiguity: $\rho$ is not defined in the paper. If it is an arbitrary probability mass function, the printed inequality is false. For instance, choose $p=(1,0)$ and $\rho =(1/2,1/2)$. Every empirical distribution equals $p$, making the printed left side one bit and its right side zero. This is a counterexample for every $n\ge 1$.

Under the consistent interpretation $\rho =p$, the preceding leave-one-out identity for empirical distributions and convexity of relative entropy in its first argument give, for every sample,Deleting any one observation from $n+1$ independent and identically distributed random variables leaves $n$ independent observations with probability mass function $p$. Consequently each ${\hat{q}}_{n,i}$ has the same distribution as ${\hat{p}}_n$. They need not be independent of one another. Taking expected values and using linearity of expectation therefore provesZero coordinates of $p$ are never sampled, almost surely, so the calculation can be restricted to its probability support. In particular the expected values are finite on this finite alphabet.

The discrepancy cannot be removed by silently choosing any fixed reference. Whenever $\rho$ is positive on the probability support of $p$, direct subtraction and $\mathbb{E}{\hat{p}}_n=p$ give the expected relative entropy decompositionThus the valid arbitrary-reference version has $\rho$ on both sides; the printed mixed-reference version has an additional term whose sign prevents the proposed conclusion.

For probability mass functions on the finite alphabet, take logarithms to base two and define the relative entropy byA term with $q_i=0$ is zero, including $p_i=0$; a term with $q_i>0=p_i$ is $+\infty$. Fix $p_i>0$ and write $f_i(t)=t{\log }_2(t/p_i)$. For $t>0$,Thus $f_i$ is a convex function on $[0,\infty )$ by continuous extension to zero. For $0\le \lambda \le 1$, applying this scalar convexity to each coordinate and summing provesIf $p$ has zero coordinates, the finite-valued domain consists of probability mass functions supported on $\{i:p_i>0\}$; it is convex and the same proof applies there. Outside that domain the right side is infinite whenever an offending probability mass function has a positive mixture weight, so the extended-valued convexity assertion still holds. At mixture weights zero or one the inequality is an identity, without needing the expression $0\cdot \infty$.

For the maximum entropy distribution on the nonnegative integers, use the geometric distribution counting failures before the first success. Write $r=a/(a+1)$ and defineThe geometric series gives ${\sum }_i{g}_i=1$. Termwise differentiation of a power series applied to that geometric series for $0<r<1$ gives ${\sum }_{i}i{r}^i=r/(1-r{)}^2$, hence ${\sum }_{i}i{g}_i=r/(1-r)=a$.

For any competing probability mass function $p$ of mean $a$, its cross-entropy against $g$ isThis is finite. The fact that finite cross entropy bounds countable information entropy must be justified without assuming in advance that $H(p)$ is finite. Use $\ln u\le u-1$ with $u=g_i/p_i$ when $p_i>0$. Including zero terms by their limiting convention yields, for every finite $N$,Let $N\to \infty$. The information entropy partial sums increase, the cross-entropy partial sums converge to the finite value above, and the final mass difference tends to zero. This proves finiteness and the upper bound. We may now writeRelative entropy nonnegativity has equality exactly when $p=g$. Therefore the displayed geometric probability mass function is the unique maximizer, andThe mean constraint is essential: without it, uniform distributions on longer initial intervals have unbounded information entropy.

With a full-support finite distribution, suppose two distinct tree parents at depth $L-1$ each have exactly two tree children, both tree leaves. Move one tree leaf from the second pair into the vacant third tree child slot of the first tree parent. Its length remains $L$, so this relocation does not change expected codeword length. The second tree parent now has a single tree child, which is a tree leaf. Replace that remaining tree child's codeword by the tree parent's word. This preserves the prefix code property and shortens a positive-probability codeword by one. Expected codeword length strictly decreases, a contradiction. Thus there is at most one deepest pair with exactly two sibling codewords.

If zero-probability symbols are retained without a reduction convention, the claim can fail. The probability mass function $(1,0,0,0,0)$ with prefix code $(0,100,101,110,111)$ has the optimal expected codeword length one, yet contains the two deepest pairs $(100,101)$ and $(110,111)$. The contraction would only shorten a zero-probability word.

Choose noise values $0,2$, giving exactly the outputs $0,1,2,3,4$. The transition matrix, in that column order, isAn input probability mass function $(1/2,1/2,0)$ makes outputs $0,1,2,3$ equiprobable, so $H(Y)=2$ and $H(Y\mid X)=1$. Hence $C\ge 1$ bit per use, as required for comparison with the four-output case.

There is a strict improvement, which can also be computed exactly. Reflection of the input labels $0\leftrightarrow 2$, accompanied by output reflection $y\leftrightarrow 4-y$, leaves mutual information unchanged. Concavity of information entropy makes mutual information a concave function of the input probability mass function for a fixed transition matrix. Averaging an input law with its reflection therefore cannot decrease it. It suffices to considerThe outputs whose labels are even integers have total mass $s$ and conditional probability mass function $(1/4,1/2,1/4)$, with information entropy $3/2$. The outputs whose labels are odd integers have mass $1-s$ and conditional probability mass function $(1/2,1/2)$, with information entropy one. Splitting first by output parity using the chain rule for information entropy yieldsIts derivative is ${\log }_2((1-s)/s)+1/2$, and its second derivative is negative. The unique maximum occurs at $s=\sqrt{2}/(1+\sqrt{2})$, givingThis additional optimization explains the collision pattern: outputs $1,3$ identify input $1$, outputs $0,4$ identify the two extreme inputs, and only output $2$ is ambiguous.

Translate the output alphabet by $-\min (z_1,z_2)$ and put $d=|z_2-z_1|\ge 1$. The reachable outputs becomeFor $d=1$ the two sets overlap in two points; for $d=2$ in one; for $d\ge 3$ they are disjoint. In particular $|\mathcal{Y}|\ge 4$, and choosing noise values $0,1$ attains four.

With rows indexed by inputs $0,1,2$ and columns by outputs $0,1,2,3$, independence and uniform two-point noise giveEvery conditional output distribution has information entropy one bit, so $I(X;Y)=H(Y)-1\le {\log }_{2}4-1=1$. The input probability mass function $(1/2,0,1/2)$ makes all four outputs equiprobable, attaining that bound. ThusUsing a zero-probability input in a channel-capacity maximization is legitimate; it is unrelated to the strictly positive source weights needed for the tree-shape claims in Question 2.

For every input probability mass function, conditional entropy is nonnegative, soTaking the maximum over input laws proves $C\le 1$ bit per channel use. This bound comes from the binary input alphabet itself; no specific transition probabilities or decoder are needed.

Set $N=|\mathcal{X}|$, $d_i=|p_i-q_i|$ and $d={\sum }_i{d}_i=\Vert p-q{\Vert }_1$. If $d=0$, the two probability mass functions agree and the bound is zero. Otherwise every $d_i\le d\le 1/2$. The half-interval entropy summand bound, with $t=\min (p_i,q_i)$ and $v=d_i$, givesBy the triangle inequality, followed by Jensen's inequality for the concave function $f$,This proves the requested finite-alphabet entropy continuity bound:

For a fixed estimated probability mass function $q$, the estimated joint probability mass function is $q_n(x_1,\dots ,x_n)={\prod }_{i}q(x_i)$. If $q(x)>0$ whenever $p(x)>0$, the strong law of large numbers gives the mismatched asymptotic equipartition property:Here $H(p,q)$ is cross-entropy, and the last equality is cross-entropy equals entropy plus relative entropy. The Kullback-Leibler divergence measures the extra average logarithmic loss from the estimated law.

If $q(x)=0$ for some $x$ with $p(x)>0$, let $a={\sum }_{x:q(x)=0}p(x)>0$. The probability that none of the first $n$ observations belongs to this set is $(1-a{)}^n\to 0$. Almost surely such a symbol eventually occurs, after which $q_n(X_1,\dots ,X_n)=0$ forever. Thus the limit is $-\infty$, consistent with infinite cross-entropy and infinite Kullback-Leibler divergence.