Multinomial distribution ($\mathrm{Mult}(n;p_1,\dots ,p_k)$)

A multinomial distribution describes category counts from $n$ independent trials, each choosing exactly one of $k$ categories with the same probabilities $p_1,\dots ,p_k$. Require $n\ge 0$ to be an integer, $p_i\ge 0$, and ${\sum }_i{p}_i=1$. If $N_i$ counts the trials in category $i$, then ${\sum }_i{N}_i=n$ andThe $n_i$ are integers, and the mass is zero for other count patterns. A category with $p_i=0$ has count zero almost surely, with $0^0=1$ in this formula. For a specified ordered trial sequence, independence gives the product of numbers ${\prod }_i{p}_i^{n_i}$; the multinomial coefficient counts the different sequences with that same count pattern.

For four trials with category probabilities $(1/2,3/10,1/5)$, the count pattern $(2,1,1)$ can occur in $4!/(2!1!1!)=12$ orders, each of probability $(1/2{)}^2(3/10)(1/5)=0.015$. Thus its total probability is $12(0.015)=0.18$. This is a constructed independent-trial example, distinct from drawing four objects without replacement from a ten-object population of composition $(5,3,2)$, where the multivariate hypergeometric distribution gives $2/7$.

The diagram shows all fifteen possible count patterns. Coordinates $(N_1,N_2)$ determine $N_3=4-N_1-N_2$; the triangle is a constraint on integer counts, not a continuous probability density. The mode is $(N_1,N_2,N_3)=(2,1,1)$, where the probability mass function is largest, but the expected values $(2,1.2,0.8)$ are not a possible count pattern.

Each single count has a binomial distribution, because recording category $i$ versus all other categories reduces every trial to a Bernoulli trial. Consequently,These statements about the joint count model are summarized in Aditya Guntuboyina's lecture notes.^[ref]

Independent trials do not make the resulting counts independent random variables. Within one trial the indicator random variables for two distinct categories cannot both equal one, so their product is zero and their covariance is $-p_i{p}_j$. Summing over independent trials gives $-n{p}_i{p}_j$. In the four-trial example, the variances are $(1,0.84,0.64)$ and the pairwise covariances are $-0.6,-0.4,-0.24$. Knowing that all four trials were in the first category fixes the other two counts at zero.

The attraction is that one formula describes any fixed-total list of category counts while preserving the constraint that they add up to $n$. It underlies multinomial likelihood calculations and uniform balls-in-bins allocation. The multinomial aggregation rule lets us merge categories without leaving this family of probability distributions. Allowing independent category counts with an unfixed total instead leads to a different model, related through Poisson-multinomial conditioning.

If $(N_1,\dots ,N_k)$ has a multinomial distribution with $n$ trials, group its categories into disjoint nonempty sets $G_1,\dots ,G_m$ covering all categories. The grouped counts in the displayed equation have the multinomial distributionEach original trial belongs to exactly one group, its group probability is the sum of its category probabilities, and grouping preserves independence between trials. This gives the rule without summing the original probability mass function over every possible internal count pattern.

For the four-trial example with probabilities $(1/2,3/10,1/5)$, merge the last two categories. Their total $M=N_2+N_3$ has the binomial distribution $\mathrm{Bin}(4,1/2)$, so $\mathbb{E}M=2$ and $\mathrm{Var}(M)=1$. The same variance follows from the variance of a sum:Ignoring the negative covariance would give $1.48$, although the two counts share a fixed total. Multinomial aggregation lets the resolution of category labels change while retaining a consistent model for the counts.

For cell means ${\mu }_j$ with ${\sum }_j{\mu }_j=n$, the independent Poisson distribution likelihood and the multinomial distribution likelihood with probabilities ${\mu }_j/n$ differ by a parameter-independent factor. Thus a maximum-likelihood estimator for the unrestricted Poisson model whose fitted total is $n$ is also an estimator for the constrained multinomial model. Equality of named parameter estimates presumes identifiability and uniqueness; without these, the conclusion concerns the sets of maximizers. A free intercept in a log-linear Poisson model enforces the fitted-total equation.

Counts of several categories before a fixed number of stopping-category outcomes have a negative multinomial distribution. For one stopping outcome of probability $r=1-\alpha -\beta$, the two-count mass function is $r\left(\genfrac{}{}{0ex}{}{x+y}{x}\right){\alpha }^x{\beta }^y$. Its probability generating function is $r/(1-\alpha s-\beta t)$ and its means are $\alpha /r$ and $\beta /r$.

For $n$ independent one-stop negative multinomial distribution observations, the natural estimates are $\hat{\alpha }=S_X/(n+S_X+S_Y)$ and $\hat{\beta }=S_Y/(n+S_X+S_Y)$. Their sum is $g((S_X+S_Y)/n)$ with strictly concave $g(z)=z/(1+z)$, so Jensen's inequality makes its expectation strictly below $\alpha +\beta$. Conditional category proportions split this bias between both estimates. With zero observed counts of a category, that estimate lies at zero; in a parameter space requiring strict positivity the likelihood supremum is not attained.

The vector of category counts is the sum of independent categorical indicator vectors. Their means are $p$ and covariance is $\mathrm{diag}p-p{p}^T$, so the multivariate central limit theorem gives the displayed limit. At uniform probabilities on $k$ categories, Pearson-standardized deviations have covariance equal to the orthogonal projection onto the $(k-1)$-dimensional subspace of zero-sum vectors. Their squared norm has an asymptotic chi-squared distribution with $k-1$ degrees of freedom.

Place $n$ labelled balls independently into $m$ labelled bins, each with probability $1/m$. The vector of bin counts has a multinomial distribution, and each count separately has the binomial distribution $\mathrm{Bin}(n,1/m)$. Distinct bin counts have covariance $-n/m^2$, so they are dependent for $n>0$ and $m>1$. Counts of bins of a specified size are sums of indicator variables; their expectations follow from the linearity of expectation, without any independence between bins.

In repeated uniform balls-in-bins allocation, remove each ball that is alone in its bin and independently reallocate all remaining balls in the next round. If $B_j$ balls remain, the conditional expectation of the next count is $B_j[1-(1-1/n{)}^{B_j-1}]$ for $B_j\ge 2$, and zero for $B_j\le 1$. The surviving-ball indicator random variables need not be independent. For two balls and two bins, both remain with probability $2^{-j}$ after $j$ rounds, so $\mathbb{E}{B}_j=2^{1-j}$.

For $n$ balls in $n$ uniformly and independently chosen boxes, put $p_n=(1-1/n{)}^n$. The empty-box indicators in uniform allocation have negative pairwise covariance, so $\mathrm{var}(K_n)\le n{p}_n(1-p_n)\le n/4$. Chebyshev's inequality givesSince $p_n\to e^{-1}$, this proves the displayed convergence in probability. The centring and the negative covariance are essential to this short proof; an expectation limit alone would not establish concentration.

Place $n\ge 1$ balls independently and uniformly into $m\ge 2$ boxes. If $I_i$ indicates that box $i$ is empty, then $\mathbb{E}{I}_i=p$, $\mathrm{var}(I_i)=p(1-p)$, and, for $i\ne j$,Indeed all balls avoid one specified box with probability $p$ and two specified boxes with probability $q$. The indicator random variable identities give the expectations, variances, and covariance. Finally $1-2/m<(1-1/m{)}^2$ proves strict negativity. Thus the number $K$ of empty boxes has $\mathbb{E}K=mp$ and $\mathrm{var}(K)=mp(1-p)+m(m-1)(q-p^2)$ by the variance of a sum.

In a uniform balls-in-bins allocation with $m\to \infty$ and $n/m\to \lambda \in (0,\infty )$, a fixed bin count has convergence in distribution to a Poisson distribution variable of mean $\lambda$ by the Poisson limit theorem. If $N_j$ counts bins containing exactly $j$ balls, symmetry and the linearity of expectation give $\mathbb{E}[N_j]/m=\mathbb{P}(B_1=j)\to e^{-\lambda }{\lambda }^j/j!$ for each fixed $j$. In particular the expected empty fraction tends to $e^{-\lambda }$, and the expected fraction with at least two balls tends to $1-(1+\lambda )e^{-\lambda }$. These are statements about expectations; convergence of the random fractions requires an additional argument.

For category counts $y_1,\dots ,y_k$ with fixed total $n$ and probabilities $p_1,\dots ,p_k$, the multinomial likelihood isIts maximum-likelihood fitted probabilities are ${\hat{p}}_j=y_j/n$.

For counts with a multinomial distribution, this is twice the log likelihood ratio of the saturated cell-probability estimate to fixed probabilities $p_i$. Zero-count terms have limiting value zero. Simulation under the specified probabilities gives a direct reference distribution when asymptotic approximations are unsuitable.

A categorical distribution describes one trial with one of $k$ labelled outcomes and probabilities $p_1,\dots ,p_k$ summing to one. The labels need not be numbers or have an order. It is the single-trial building block of the multinomial distribution: $n$ independent draws with the same categorical distribution produce multinomial category counts.

For a trial choosing labels $A,B,C$ with probabilities $1/2,3/10,1/5$, the category-indicator random vector is $(1,0,0)$, $(0,1,0)$, or $(0,0,1)$. Its expected value is $(1/2,3/10,1/5)$; its components are dependent because exactly one is one. A numerical encoding of $A,B,C$ is a separate modelling choice: the category labels themselves do not define an arithmetic mean.