 Solution

Use metric signature $(+---)$ and four-velocities $U_i^a={\gamma }_{i}c(1,{\mathbf{\beta }}_i)$, with ${\gamma }_i=(1-{\beta }_i^2{)}^{-1/2}$. Their Lorentz invariant inner product isIn the rest frame of either particle this same scalar is the relative Lorentz factor ${\gamma }_{\mathrm{rel}}$. HenceLet $d={\mathbf{\beta }}_1\cdot {\mathbf{\beta }}_2$. The numerator after combining terms isUse $|{\mathbf{\beta }}_1\times {\mathbf{\beta }}_2{|}^2={\beta }_1^2{\beta }_2^2-d^2$ to obtainThe two rest frames agree on the relative speed, although their velocity vectors are expressed in different spatial frames. Checks include equal velocities giving zero, collinear velocities giving $|{\beta }_1-{\beta }_2|/|1-{\beta }_1{\beta }_2|$, and low speeds reducing to the Galilean relative speed. For timelike particles the four-velocity inner product also ensures $0\le {\beta }_{\mathrm{rel}}<1$.