 Solution

Let ${\rho }_3(P)=P\otimes P\otimes P$ act on the alternating coefficient tensor $\phi$, and setThe coefficient convention here is the one in the source's hint; using the usual covariant pullback gives the same stabilizer subgroup, since an invertible transformation preserves a differential form if and only if its inverse does. The identity matrix fixes $\phi$. If $P,Q$ fix it, then ${\rho }_3(PQ)\phi ={\rho }_3(P){\rho }_3(Q)\phi =\phi$. Multiplying ${\rho }_3(P)\phi =\phi$ by ${\rho }_3(P^{-1})$ shows that $P^{-1}$ fixes it as well. Thus $H$ is a subgroup of $SO(7)$. Its defining equations are polynomial, so it is closed and hence a Matrix Lie group.

For $P(t)=I+tT+O(t^2)\in H$, differentiating $P^{T}P=I$ gives $T^T=-T$. Differentiating the three factors in ${\rho }_3(P(t))\phi =\phi$ givesThe skew-symmetric matrix condition records the ambient special orthogonal group; it must be included when differentiating that group's definition.

These linear conditions are sufficient as well as necessary. On the tensor product, the infinitesimal action isIts three terms commute, so ${\rho }_3(e^{tT})=e^{tD(T)}$. If $D(T)\phi =0$, every positive power of $D(T)$ annihilates $\phi$, giving $e^{tD(T)}\phi =\phi$. Also $T\in \mathfrak{so}(7)$ implies $e^{tT}\in SO(7)$. Therefore $e^{tT}\in H$, proving the asserted Lie algebra description. For the given positive three-form this is the compact G2 Lie algebra ${\mathfrak{g}}_2$.