Solution (source code)

= Solution

Define the invariants by the <characteristic polynomial>
$$
\det(tI-B)=t^3-I_Bt^2+II_Bt-III_B.
$$
Because $B=FF^T$ has positive <eigenvalues> $\lambda_j^2$, its invariants are the <elementary symmetric functions> in those <eigenvalues>:
$$
I_B=\lambda_1^2+\lambda_2^2+\lambda_3^2,\qquad
II_B=\lambda_1^2\lambda_2^2+\lambda_1^2\lambda_3^2+\lambda_2^2\lambda_3^2,\qquad
III_B=\lambda_1^2\lambda_2^2\lambda_3^2.
$$
The <Frobenius norm> gives $|F|^2=\operatorname{tr}(F^TF)=\sum_j\lambda_j^2$. Write $J=\det F>0$. Since $\operatorname{cof}F=JF^{-T}$, its <singular values> are $J/\lambda_j$, so
$$
|\operatorname{cof}F|^2=J^2\sum_j\lambda_j^{-2}
=\lambda_2^2\lambda_3^2+\lambda_1^2\lambda_3^2+\lambda_1^2\lambda_2^2.
$$
Finally $\det B=\det F\det F^T=J^2$. Therefore
$$
\boxed{I_B=|F|^2,\qquad II_B=|\operatorname{cof}F|^2,\qquad III_B=(\det F)^2.}
$$
These identities also give $I_B=\operatorname{tr}B$ and $II_B=((\operatorname{tr}B)^2-\operatorname{tr}(B^2))/2$.