Solution (source code)

= Solution

Expanding the <determinant> along the first row gives
$$
\det M=-a^2-b^2+3b+3.
$$
Under the <matrix trace> condition $a^2=3$, this simplifies to $\det M=b(3-b)$. A square <matrix> is invertible exactly when its <determinant> is nonzero. Thus
$$
\boxed{a=\pm\sqrt3,\qquad b\in\mathbb R\setminus\{0,3\}.}
$$