Past exam of the mathematics course of the University of Cambridge / 2023 / iii / Paper 224 / 4

Put $p_i=\mathbb{P}(X_i=1)$ and $\lambda ={\sum }_i{p}_i$. A relative-entropy form of the Poisson approximation bound for dependent Bernoulli variables isThe last two terms form the total correlation; they vanish when the Bernoulli variables are independent.

Here $D_e$ and $H_e$ use natural logarithms.

To prove the bound, let $Q_i$ be the Poisson distribution with mean $p_i$ and let $Q={⨂}_i{Q}_i$. Expanding the Kullback-Leibler divergence against this product law givesThe supplied one-dimensional estimate bounds the first sum by ${\sum }_i{p}_i^2$. Under the addition map, $P_{X_1^n}$ becomes $P_{S_n}$, while the sum of independent Poisson random variables under $Q$ has the Poisson distribution with mean $\lambda$. The data processing inequality for relative entropy proves the displayed result.

If a bound directly in the paper's unhalved total-variation norm is desired, Pinsker's inequality also gives

LetHere $S_n$ has the binomial distribution with parameters $(n,\lambda /n)$. Part a, now with independent coordinates, givesBy Pinsker's inequality, the probability mass functions therefore converge in total variation, and in particular $S_n$ converges in distribution to $Z\sim \mathrm{Poisson}(\lambda )$.

The joint probability of the observed row depends only on $S_n$:Taking logarithms in any fixed base and choosing $c_n=\log n$ giveswhereThe convergence lemma supplied in the question now yieldsThis sparse triangular array therefore has a random limiting normalized self-information rather than the constant limit in the usual asymptotic equipartition property.