Solution (source code)

= Solution

Let $q=(m_C^2-m_B^2)/(2m_B)$ and $p=(m_B^2-m_A^2)/(2m_B)$. In the B frame these are the energies of $t$ and $s$. If $\theta$ is their relative angle,
$$
m_{st}^2=2pq(1-\cos\theta),\qquad
\boxed{(m_{st}^2)_{\min}=0,\quad
(m_{st}^2)_{\max}=\frac{(m_C^2-m_B^2)(m_B^2-m_A^2)}{m_B^2}.}
$$
The minimum has parallel massless particles; the maximum has antiparallel ones. The isotropic decay allows both limiting directions. The bounds are <Lorentz invariant> even though the B frame makes their calculation simple.