Solution (source code)

= Solution

A homogeneous <linear system> has a nonzero solution exactly when its coefficient <matrix> is singular. Expansion of the <determinant> gives
$$
\det M=5\alpha^2+3\alpha-8=(\alpha-1)(5\alpha+8).
$$
Thus the allowed values are $\boxed{\alpha=1\text{ or }-8/5}$. Solving the first two equations and checking the third gives
$$
\boxed{\alpha=1:\ x=\frac z5,\ y=-\frac{7z}5;\qquad \alpha=-\frac85:\ x=-5z,\ y=\frac{5z}2.}
$$
In each case $z\ne0$ gives a nonzero solution and spans the one-dimensional <null space>; $z=0$ gives only the zero vector.