Solution (source code)

= Solution

Take $\gamma>0$ so that the short-wavelength quadratic energy is stable. For $B=0$, a <Fourier mode> with wavevector magnitude $q$ has inverse <Gaussian field theory> covariance
$$
G(q)=a+\kappa q^2+\gamma q^4
=a-\frac{\kappa^2}{4\gamma}+\gamma\left(q^2+\frac\kappa{2\gamma}\right)^2.
$$
For $\kappa<0$, its minimum is at
$$
\boxed{q_0=\sqrt{-\frac\kappa{2\gamma}},\qquad a_c=\frac{\kappa^2}{4\gamma}.}
$$
The zero-average constraint excludes the uniform mode but allows $\phi=A\cos(q_0z)$. At $a=a_c$ the modes on the <nonzero-wavevector soft-mode sphere> become soft; below it they have negative quadratic energy, so the homogeneous state loses stability. A modulation along any chosen direction is the simplest <smectic phase> pattern. The quadratic instability selects its wavelength; with $B=0$, a nonlinear stabilizing interaction would still be needed to determine its amplitude.