 Solution

Take $U=Y$ and $V=X{Y}^2$. The inverse is $y=u$, $x=v/u^2$, with absolute Jacobian $u^{-2}$. The original unit-square support becomes $0<u<1$ and $0<v<u^2$, so the joint density is $u^{-2}$ there. For a fixed $0<v<1$, the admissible range is $\sqrt{v}<u<1$. ThereforeIt is nonnegative and integrates to ${\int }_0^1(v^{-1/2}-1)dv=1$, as required.