Solution (source code)

= Solution

The integrand is even and decays at both infinity and zero, since $a,b>0$. On $x>0$ set $u=\sqrt a\,x-\sqrt b/x$, a strictly increasing bijection onto the whole real line. Then
$$
ax^2+b/x^2=u^2+2\sqrt{ab},\qquad
x=\frac{u+\sqrt{u^2+4\sqrt{ab}}}{2\sqrt a},\qquad
\frac{dx}{du}=\frac1{2\sqrt a}\left(1+\frac u{\sqrt{u^2+4\sqrt{ab}}}\right).
$$
Using evenness for the negative half-axis gives
$$
I(a,b)=\frac{e^{-\sqrt{ab}}}{\sqrt a}\int_{-\infty}^{\infty}e^{-u^2/2}\left(1+\frac u{\sqrt{u^2+4\sqrt{ab}}}\right)du.
$$
The second summand is an integrable <odd function>, so its integral vanishes. The remaining <Gaussian integral> gives
$$
\boxed{I(a,b)=\sqrt{\frac{2\pi}a}\,e^{-\sqrt{ab}}.}
$$
For the second integral use $x=1/t$ separately on each half-axis. The factor $x^{-2}dx$ becomes $-dt$, and reversing the transformed limits yields $J(a,b)=I(b,a)$. Thus
$$
\boxed{J(a,b)=\sqrt{\frac{2\pi}b}\,e^{-\sqrt{ab}}.}
$$
These identities are instances of the <Gaussian integral with an inverse-square term>. Splitting at zero before either <change of variables> avoids treating a discontinuous transformation as a single bijection.