Past exam of the mathematics course of the University of Cambridge / 2018 / iii / Paper 216 / 4

In factor analysis, the independent multivariate normal distributions of $Z_i$ and ${\xi }_i$ imply that their linear combination is also multivariate normal. Its expected value is zero and its covariance matrix isThereforeThe noise variance is assumed positive. The factor contribution is positive semidefinite and has rank at most $k$, while the noise makes the marginal covariance nonsingular.

For one factor, write $\lambda \in {\mathbb{R}}^p$ for the loading vector and $v={\sigma }^2>0$. Let ${\lambda }_t$ be its current value and $s=v+\Vert {\lambda }_t{\Vert }^2$. The E-step of the expectation-maximization algorithm uses the conditional multivariate normal distribution to obtainThus $\mathbb{E}[Z_i^2\mid Y_i]=u+m_i^2$. The terms of the expected complete-data log-likelihood that depend on a candidate $\lambda$ areThis is a strictly concave quadratic in $\lambda$, since ${\sum }_i(u+m_i^2)>0$. Its unique maximizer isFor an explicit expression involving only the current loading and data, put $S=n^{-1}{\sum }_i{Y}_i{Y}_i^T$. Substitution gives the EM update for a single-factor Gaussian modelNo empirical centering is applied because the model fixes the mean to zero. If initialized at zero, this update remains at zero; monotonicity alone does not guarantee that such a stationary point is a global maximum. This part is present in the PDF but missing from the TeX transcription.

Fix the observed $Y$, let $L(\theta )=p_{\theta }(Y)$, and let $q_0(z)=p_{{\theta }_0}(z\mid Y)$. The E-step formsFor finite expected log-likelihoods and $L({\theta }_0)>0$, the Jensen inequality givesThe first inequality allows additional mass outside the old complete-data support; it is equality when the relevant supports coincide. Ratios are taken on the support of $q_0$. If a candidate assigns zero density on positive $q_0$-mass, its $Q$ is minus infinity and it cannot be an improving finite M-step.

An M-step of the expectation-maximization algorithm chooses ${\theta }_1$ with $Q({\theta }_1\mid {\theta }_0)\ge Q({\theta }_0\mid {\theta }_0)$, soThis EM likelihood monotonicity also holds for a generalized M-step that merely increases $Q$, rather than maximizing it exactly. It is a statement about the observed-data likelihood function.