= High explained variance need not preserve class information under PCA
Let $W,S$ be independent balanced variables with values $-1,1$, and let $X=(aW,bS)^T$ with $a>b>0$ and class $Y=S$. Then $E[X]=0$ and $\operatorname{Cov}(X)=\operatorname{diag}(a^2,b^2)$. The first <principal component> is the first coordinate and explains $a^2/(a^2+b^2)$ of total <variance>. Yet it is independent of $Y$, so even the best <classifier> using only that coordinate has accuracy $1/2$. The second coordinate determines $Y$ exactly. Taking $a/b$ arbitrarily large makes the retained <variance> approach one without improving class accuracy. Adding a constant baseline to each coordinate preserves the centered result and permits positive neural rates.
Back to article page
