Solution (source code)

= Solution

Choose the $M$ ball's initial <velocity> as $+v$; the $3M$ ball then has <velocity> $-v$. Both have <Lorentz factor> $\gamma=(1-v^2/c^2)^{-1/2}$. Their total <relativistic energy> and signed <relativistic momentum> are
$$
E_i=4\gamma Mc^2,\qquad p_i=\gamma Mv-3\gamma Mv=-2\gamma Mv.
$$
The laboratory frame is not the <centre-of-momentum frame>, despite the equal and opposite <speeds>. <Four-momentum conservation> and the <relativistic energy-momentum relation> give the <invariant mass> $M_f$ of the combined incoming system:
$$
M_f^2c^4=E_i^2-c^2p_i^2
=\gamma^2M^2c^4(16-4\beta^2),\qquad \beta=\frac vc.
$$
The <relativistic collision mass threshold> for a final object of <rest mass> $5M$ is $M_f\geq5M$. At the threshold,
$$
\frac{16-4\beta^2}{1-\beta^2}=25
\quad\Longrightarrow\quad 21\beta^2=9,
\qquad
\boxed{v_{\min}=c\sqrt{\frac37}\simeq0.65465\,c.}
$$
At equality, the outgoing object has laboratory <velocity> $V=c^2p_i/E_i=-v/2$, directed towards the initially heavier ball's motion. Its nonzero final <kinetic energy> is why setting $4\gamma Mc^2=5Mc^2$ would give the wrong threshold.

For the literal <rest mass of a coalescing relativistic collision>, if there is exactly one outgoing object of exactly fixed <rest mass> $5M$ and no radiation or other products, <four-momentum conservation> selects the equality above, rather than allowing every larger $v$. Speeds above this threshold require a larger final <invariant mass> or additional outgoing <energy> carriers.