 Solution

With $J=\mathrm{diag}(1,-1)$ these matrices obey $M^{†}JM=J$ and $\det M=1$, identifying the group as SU(1,1). Products preserve the displayed conjugate-entry pattern and determinant; the identity belongs, andalso belongs. Multiplication and inversion are smooth. The real constraint $|\alpha {|}^2-|\beta {|}^2=1$ has nonzero gradient on its level set, so the matrices form a smooth real three-dimensional manifold. Associativity is inherited from matrix multiplication. These facts establish a Lie group, which is noncompact because $|\beta |$ is unbounded.