Past exam of the information theory course of the University of Oxford / 2021 / 1 / b / ii

Index correction: the printed definition only permits $i\le n$, although the required average includes $i=n+1$. Extend the same deletion formula to $1\le i\le n+1$. Without that extension the final summand is undefined; there is no new sample or different estimator to supply.

For a fixed outcome of the sample and a fixed symbol $x$, put $A_j=1_{\{X_j=x\}}$. The leave-one-out empirical distribution then satisfiesDividing by $n+1$ gives ${\hat{p}}_{n+1}=(n+1{)}^{-1}{\sum }_{i=1}^{n+1}{\hat{q}}_{n,i}$. Each of the $n+1$ indicator functions was counted exactly $n$ times before division by $n$. This leave-one-out identity for empirical distributions is deterministic; independence plays no part in this part.