 Solution

Apply the Abelian group criterion for a product of two prime squares with the smaller and larger primes in their correct order. For $p<11$, all such groups are Abelian exactly when $p\nmid 11^2-1=120$. Of the primes $2,3,5,7$, only $7$ qualifies. For $p>11$, the criterion is $11\nmid p^2-1$, equivalently $p\not\equiv \pm 1(\mathrm{mod}11)$. Therefore the exact answer isEquivalently, among the stipulated primes $p\ne 11$, exclude $2,3,5$ and the residue classes $\pm 1$ modulo $11$.

For completeness, the necessity in that criterion supplies actual counterexamples at every excluded prime. If $\ell <q$ and $\ell \mid q^2-1$, the multiplicative group of a finite field is cyclic, so choose $\alpha \in {\mathbb{F}}_{q^2}^{\times }$ of order $\ell$. Let a generator of $C_{{\ell }^2}$ act on the additive group of ${\mathbb{F}}_{q^2}$ by multiplication by $\alpha$. This defines a nontrivial group action, hence a nonabelian semidirect product of order ${\ell }^2{q}^2$. Use $(\ell ,q)=(p,11)$ for $p=2,3,5$, and $(\ell ,q)=(11,p)$ for $p\equiv \pm 1(\mathrm{mod}11)$ with $p>11$. Thus no excluded prime can be restored to the answer.