Chi-squared divergence (${\chi }^2(P\Vert Q)$)

For probability mass functions $P,Q$ with $P$ absolutely continuous with respect to $Q$,

If $r=dQ/dP$, independence gives $E_{P^{\otimes n}}{\prod }_{i}r(X_i{)}^2=(E_P{r}^2{)}^n$. Since $E_{P}r=1$, this proves the displayed identity. The Cauchy-Schwarz inequality also gives $\Vert Q^{\otimes n}-P^{\otimes n}{\Vert }_1^2\le {\chi }^2(Q^{\otimes n}\Vert P^{\otimes n})$. Thus perturbations of size ${\chi }^2(Q\Vert P)=O(1/n)$ have a uniformly bounded joint total variation distance, a useful input to a metric squared-loss two-point bound.

If $L_j=d{P}_j/dQ$ and the nonnegative mixture weights sum to one, the likelihood ratio of the mixture model is $L={\sum }_j{w}_j{L}_j$. Expanding ${\mathbb{E}}_Q{L}^2-1$ proves the displayed chi-squared divergence identity. For independent random variables sampled from a fixed component, one-observation cross moments multiply across observations. The component is drawn once for the whole sample; redrawing it per observation gives a different mixture model.