Solution (source code)

= Solution

For a fixed $x\in[0,1]$, the allowed $y$ interval is $[0,\sqrt x]$. Integrating the <joint probability density> over it gives the <marginal density>
$$
\boxed{f_X(x)=\begin{cases}\displaystyle\int_0^{\sqrt x}6xy\,dy=3x^2,&0\le x\le1,\\0,&\text{otherwise}.\end{cases}}
$$
For a fixed $y\in[0,1]$, the inequality $y\le\sqrt x$ is equivalent to $x\ge y^2$. The other <marginal density> is therefore
$$
\boxed{f_Y(y)=\begin{cases}\displaystyle\int_{y^2}^1 6xy\,dx=3y(1-y^4),&0\le y\le1,\\0,&\text{otherwise}.\end{cases}}
$$
As a normalization check, $\int_0^1 3x^2\,dx=1$ and $\int_0^1 3y(1-y^4)\,dy=1$.