Solution (source code)

= Solution

For the two sine modes, <orthogonality> gives
$$
\int_0^1\widetilde\psi^2dx=\frac{1+\alpha^2}{2},\qquad\int_0^1\widetilde\psi'^2dx=\frac{\pi^2}{2}(1+4\alpha^2).
$$
The diagonal potential integrals vanish, while the mixed one is
$$
\int_0^1\sin(\pi x)\sin(2\pi x)\cos(\pi x)dx=\frac14.
$$
Thus $\int_0^1\cos(\pi x)\widetilde\psi^2dx=\alpha/2$, giving the <two-sine variational bound for a Dirichlet Mathieu equation>:
$$
\boxed{R(\alpha)=\frac{\pi^2+\alpha+4\pi^2\alpha^2}{1+\alpha^2}.}
$$
Its <derivative> vanishes when $1+6\pi^2\alpha-\alpha^2=0$. Hence the two stationary parameters are $3\pi^2\pm\sqrt{9\pi^4+1}$. To identify the minimum, use the normalized sine basis: the restricted <self-adjoint operator> has real <symmetric matrix>
$$
H_{\mathrm{trial}}=\begin{pmatrix}\pi^2&1/2\\1/2&4\pi^2\end{pmatrix}.
$$
Its smaller <eigenvalue> is the minimum of the <Rayleigh quotient>, attained at the negative stationary parameter. Therefore
$$
\boxed{\alpha_{\mathrm{opt}}=3\pi^2-\sqrt{9\pi^4+1},\qquad\lambda_0\leq\frac{5\pi^2-\sqrt{9\pi^4+1}}2\approx9.86116.}
$$
The estimate improves on the single-sine value $\pi^2$. The <Rayleigh-Ritz variational principle> remains valid here even though $q(x)=\cos(\pi x)$ changes sign: on this finite interval the <Sturm-Liouville operator> is self-adjoint and bounded below, which is sufficient for the upper bound.