Solution (source code)

= Solution

Assume $\gamma>0$ and zero initial cavity amplitude. Taking <Laplace transforms> gives
$$
(s+\gamma/2)\widetilde a=-\sqrt\gamma\,\widetilde b_0,
\qquad \widetilde b_1=\sqrt\gamma\,\widetilde a+\widetilde b_0.
$$
Eliminating the mode yields the <passive cavity input-output transfer function>
$$
\boxed{\widetilde b_1=G(s)\widetilde b_0,\qquad
G(s)=1-\frac\gamma{s+\gamma/2}=\frac{s-\gamma/2}{s+\gamma/2}.}
$$
Its unique pole is $s=-\gamma/2$, so the cavity's internal mode decays and its causal input-output response is stable. A nonzero initial amplitude would add $\sqrt\gamma\,a(0)/(s+\gamma/2)$ to the output, a decaying transient.

For a precise <Nyquist stability criterion> interpretation of this open cavity, rewrite the mode equation as an integrator with negative state feedback. Its return ratio is $L(s)=(\gamma/2)/s$, and its characteristic equation is $1+L(s)=0$. The frequency locus $L(i\omega)=-i\gamma/(2\omega)$ lies on the imaginary axis. The Nyquist indentation excluding the pole at the origin maps to a large semicircle in the right half-plane; the whole contour has no winding around $-1$. There are no right-half-plane open-loop poles within that indented contour, so there are no unstable characteristic zeros. There is also no zero-frequency characteristic root, since the physical characteristic polynomial is $s+\gamma/2$. This recovers asymptotic stability.

Do not use the plant $G$ as a return ratio without specifying an additional feedback connection. Indeed
$$
G(i\omega)=\frac{\omega^2-(\gamma/2)^2+i\gamma\omega}{\omega^2+(\gamma/2)^2},\qquad |G(i\omega)|=1,
$$
and its locus passes through $-1$ at $\omega=0$. That would indicate a marginal mode for a new unity-negative-feedback loop with characteristic factor $1+G$, but not instability of the original cavity. Part (e) supplies the actual return ratio $\alpha G$ for the beam-splitter network.