Solution (source code)

= Solution

Since $X_1$ is the sum of two independent centered Gaussian innovations,
$$
\boxed{X_1\sim N(0,\sigma^2(1+\theta^2)).}
$$
For any nonzero $a\in\mathbb R^n$,
$$
a^T\Sigma a
=\operatorname{Var}\left(\sum_{t=1}^na_tX_t\right).
$$
Expanding each $X_t=\varepsilon_t+\theta\varepsilon_{t-1}$ expresses this as $\sigma^2$ times a sum of squared innovation coefficients. If all coefficients vanished, the coefficient of the latest innovation gives $a_n=0$, and backward induction gives every $a_t=0$, a contradiction. Hence $a^T\Sigma a>0$ and the covariance matrix is positive definite.