 Solution

The square roots simplify to the absolute values $|a-b|$ and $|a+b|$. Put $s=\sin (\pi \sqrt{a^2+b^2})$. A nontrivial solution of the homogeneous linear system exists exactly when its coefficient matrix is singular:The second factor vanishes precisely when $(a-b{)}^2=(a+b{)}^2$, or $ab=0$, giving both coordinate axes. The first factor vanishes when $\sqrt{a^2+b^2}$ is a nonnegative integer. Hence the complete locus isThese are the two axes together with concentric circles of positive integer radii. The zero-radius case is the origin, already included on the axes; at that point both equations vanish and any nonzero pair $(x,y)$ works.