Past exam of the mathematics course of the University of Cambridge / 2017 / ii / Paper 3 / 5J / Solution

Write $\ell (\beta )$ for the log-likelihood and $U=\nabla \ell$ for the score function. Newton method uses the Observed Fisher information $M=-{\nabla }^2\ell$; Fisher scoring uses the Fisher information $M={\mathbb{E}}_{\beta }[-{\nabla }^2\ell ]$.

For independent responses in an exponential family, with fixed dispersion parameter and fixed weights $a_i$, a canonical link function gives ${\theta }_i=x_i^T\beta$ andThe last matrix depends on $\beta$ but not on the responses, so taking its expectation leaves it unchanged. Hence the two updates are identical, wherever this Fisher information matrix is invertible. Equivalently both solve the same iteratively reweighted least squares problem. This statement concerns the regression coefficients at a fixed dispersion parameter; a joint update of an unknown dispersion parameter need not have identical observed and expected full Fisher information matrices.