= Solution
<Factor analysis> models a large set of correlated measurements using a smaller collection of shared <latent factors> and variable-specific variation. A classical <orthogonal factor model> is
$$
\boxed{X=\mu+LF+\epsilon,\quad\mathbb EF=0,\quad\operatorname{Cov}(F)=I_q,\quad\operatorname{Cov}(\epsilon)=\Psi=\operatorname{diag}(\psi_1,\ldots,\psi_p),\quad\operatorname{Cov}(F,\epsilon)=0.}
$$
Take $q<p$, centered errors, and positive specific <variances> for the inverse-covariance and <likelihood> formulas below. The <factor loadings> form the $p\times q$ <matrix> $L$, and the model <covariance> is
$$
\Sigma=LL^T+\Psi.
$$
Thus for $i\ne k$, $\operatorname{Cov}(X_i,X_k)=\sum_jL_{ij}L_{kj}$; the common <latent factors> explain off-diagonal <correlation>. The <communality> of variable $i$ is $h_i^2=\sum_jL_{ij}^2$, and $\operatorname{Var}(X_i)=h_i^2+\psi_i$. For standardized variables this is $1=h_i^2+\psi_i$. The specific <variance> represents measurement noise or genuinely unshared variation; it is not another common <latent factor>.
This differs from <principal component analysis>. A <principal component> maximizes projected total <variance> and is a determined linear score after choosing its direction; a factor model attempts to explain shared <covariance> while reserving diagonal specific <variance>. Low-dimensional <PCA> truncation can help initialize a factor fit, but it does not justify setting all specific <variances> to zero. <Latent factors> are unobserved, not directly observed scores. When the <latent factors> and the errors are jointly Gaussian, their <conditional mean> is
$$
\mathbb E(F\mid X=x)=L^T\Sigma^{-1}(x-\mu),
$$
which supplies one factor-score estimate, with remaining uncertainty. This follows from the block conditional normal formula in Question 1; it does not make the <latent factor> uniquely recoverable from one observation.
For the Gaussian <likelihood>, use the empirical covariance $S_0=n^{-1}\sum_i(x_i-\bar x)(x_i-\bar x)^T$. Fit the factor model by minimizing $\log|LL^T+\Psi|+\operatorname{tr}(S_0(LL^T+\Psi)^{-1})$ subject to admissible specific <variances>. Other approaches estimate <communalities> and extract <latent factors> from the reduced <correlation matrix>. Choose $q$ using substantive plausibility, a <scree plot>, fitted residual <correlations> and model-fit assessment. Under local <identifiability>, a <likelihood> goodness-of-fit count for the <covariance> model is
$$
\frac{p(p+1)}2-\left[pq+p-\frac{q(q-1)}2\right],
$$
where the subtracted rotational dimension must be removed from the loading parameter count. This count alone does not establish <identifiability>, and boundary fits with zero specific <variance> need special care.
<Factor rotation> addresses interpretability and nonuniqueness. For every <orthogonal matrix> $Q$, replace $(L,F)$ by $(LQ,Q^TF)$; then $(LQ)(LQ)^T=LL^T$, so the fit is unchanged. The absolute orientation of unrestrained <latent factors> is therefore not identified by the <covariance>. <Varimax rotation> chooses orthogonal axes favoring a simple pattern of large and small squared loadings. Oblique rotation can permit correlated <latent factors>, provided their <covariance> is transformed consistently. Sign changes and permutations are further harmless relabelings; an interpretation must state its chosen convention.
\Image[/past-exam-of-the-mathematics-course-of-the-university-of-cambridge/2005/iii/paper-46-factor-rotation.png]
{title=Two-factor loading directions, an equivalent rotated representation and the covariance scree plot}
The sketch uses four standardized hypothetical measurements sharing two orthogonal <latent factors>. Rotating the factor coordinates changes the loading descriptions while preserving every implied <covariance> and <communality>; the scree plot belongs to that same constructed <covariance matrix>. \b[<Factor analysis> separates shared variation from specific <variance>, and its <latent factors> require an identifying or interpretive orientation.]
Back to article page
