 Solution

In the usual three-dimensional isotropic setting, the transition is expected to be discontinuous, by the Brazovskii fluctuation-induced first-order transition mechanism.

If $\bar{\kappa }<0$, write $q_{\ast }^2=-\bar{\kappa }/(2\gamma )$ and $r=\bar{a}-{\bar{\kappa }}^2/(4\gamma )$. Then $J(q)=r+\gamma (q^2-q_{\ast }^2{)}^2$. A continuous finite-wavevector transition would require $r\to 0$ with $q_{\ast }>0$. But the soft-mode shell integral behaves asThe positive fluctuation corrections in the self-consistency equations diverge in that limit. In particular $\bar{\kappa }=\kappa +2B{I}_0$ cannot stay at a finite negative value while $I_0$ diverges. Thus the isotropic disordered state cannot lose stability through a zero-mass mode on a fixed nonzero shell in this approximation.

Ordering is instead expected through a free-energy crossing with a finite-amplitude smectic phase, giving a first-order phase transition. This is a reasoned fluctuation prediction for the finite-wavevector transition; the disordered Gaussian variational approximation alone does not locate the coexistence point or calculate the amplitude jump. A zero-wavevector instability or a lower-dimensional setting would require a separate analysis.