Square-power rigidity of a finite nonzero spectrum

Let $S$ be a finite multiset of nonzero complex numbers such that taking the $m^2$th powers permutes it for every positive integer $m$. Taking $m=2$ and iterating the resulting permutation shows that each $z\in S$ obeys $z^{4^r}=z$ for some $r\ge 1$, hence is a root of unity. Let $d$ be a common multiple of their orders. The $d^2$th powers are all one, and the assumed permutation then forces every original element to be one. For a unitary matrix, the spectral theorem for normal operators further forces the matrix itself to be the identity.