Past exam of the mathematics course of the University of Cambridge / 2012 / iii / Paper 38 (Time Series and Monte Carlo Inference)

Use the one-sided convention: integrating the one-sided spectral density over $[0,\pi ]$ gives the variance. A standard sufficient hypothesis is absolute summability of the autocovariance sequence:For a real weakly stationary process, extend the sequence by ${\gamma }_{-k}={\gamma }_k$. The Fourier pair isAbsolute summability makes the series uniformly convergent and continuous; cosine orthogonality gives the inverse relation. Nonnegativity follows from the spectral measure of a stationary time series, or from the nonnegative Fejér approximations obtained by taking variances of finite Fourier sums.

For precision, absolute summability is sufficient, not necessary. The exact general condition is that the spectral measure be absolutely continuous with respect to Lebesgue measure. For example, a density constant on $[0,\pi /2]$ and zero elsewhere has ${\gamma }_k$ proportional to $\sin (k\pi /2)/k$, which is not absolutely summable. Thus the displayed absolutely convergent Fourier formula answers the usual covariance-summability version; an arbitrary density need only have the integral inverse relation and Fejér-mean Fourier recovery in $L^1$.

The series defining the linear filter of a stationary time series converges in $L^2$, because the norm of a tail is at most $\Vert X_0{\Vert }_2$ times the corresponding tail sum of $|a_s|$. Its mean is the original mean times ${\sum }_s{a}_s$. Its covariance is time independent and equalsUnder the summability assumption in (i),We may therefore interchange all the sums. Setting $h=k-s+u$ in the Fourier series givesThe second factor is the conjugate of the first because the coefficients are real. HenceThis spectral density transformation under a linear filter also holds whenever the original spectral measure has a density, without covariance summability: insert its integral representation into the covariance double sum, justified by absolute summability of the filter coefficients.

In the usual causal stationary convention, a centered autoregressive moving-average model satisfiesSet $\Phi (z)=1-{\sum }_{j=1}^p{\varphi }_j{z}^j$ and $\Theta (z)=1+{\sum }_{j=1}^q{\theta }_j{z}^j$. After removal of common factors, the usual causality condition is that all zeros of $\Phi$ lie outside the closed unit disk. Then $\Theta (z)/\Phi (z)$ has an absolutely summable causal coefficient sequence, giving a linear-filter representation of $X$ in terms of white noise. Invertibility of the moving-average part is a separate issue and is not needed for its covariance spectrum.

White noise has one-sided density ${\sigma }^2/\pi$. Applying the filtering result, or equating the spectra of the two filtered sides of the defining equation, givesThe moving-average sign agrees with the positive $\theta$ convention used later. More generally, the same quotient applies to a stationary two-sided solution when the reduced denominator has no unit-circle zeros; causality selects the past-dependent realization. This is the spectral density of an ARMA process.

For $|\varphi |<1$, the causal representation is $X_t={\sum }_{j\ge 0}{\varphi }^j{\epsilon }_{t-j}$. ThusAlternatively the rational spectrum from (iii) is ${\sigma }^2/\{\pi (1-\varphi z)(1-\varphi z^{-1})\}$ on $|z|=1$. The absolutely convergent geometric sums giveConsequentlyThis is the Poisson-kernel expansion of an AR(1) spectrum, with the same one-sided normalization as (i).

For the causal case $|\varphi |<1$, expandHence the white noise coefficients are ${\psi }_0=1$ and ${\psi }_j=(\varphi +\theta ){\varphi }^{j-1}$ for $j\ge 1$. Orthogonality of white noise gives ${\gamma }_k={\sigma }^2{\sum }_{j\ge 0}{\psi }_j{\psi }_{j+k}$ for $k\ge 0$. Summing the geometric series yields the autocovariance of a causal ARMA(1,1) process:For $\varphi =0$, this means ${\gamma }_1={\sigma }^2\theta$ and ${\gamma }_k=0$ for $k\ge 2$, together with ${\gamma }_0={\sigma }^2(1+{\theta }^2)$. For $\theta =-\varphi$, all positive-lag covariances vanish and ${\gamma }_0={\sigma }^2$, as expected from cancellation of the two filter polynomials.

The autocovariance of an MA(1) process comes directly from shared noise terms:Work with the zero-mean white noise convention and ${\sigma }^2>0$. Put ${\mathcal{H}}_{t-1}=\mathrm{span}(X_1,\dots ,X_{t-1})$. Suppose the preceding finite-past innovations $U_1,\dots ,U_{t-1}$ are mutually orthogonal. Their triangular relation to the observations means that they span the same finite past. For $s\le t-2$, $U_s$ is a linear combination of $X_1,\dots ,X_s$, so $\mathrm{Cov}(X_t,U_s)=0$. Moreover,Projection onto this orthogonal basis therefore has only its last coefficient nonzero:The residual is orthogonal to the entire preceding span, proving the induction. Expanding its variance givesThe initialization is $U_1=X_1$, ${\phi }_1={\gamma }_0$, ${\lambda }_1=0$ and $U_0=0$. Each finite covariance matrix is a positive-definite matrix: the last observation in any nonzero finite linear combination contains a noise term absent from earlier observations. Thus all the projection denominators are positive. These are the finite-sample innovations of an MA(1) process, requiring no Gaussian assumption.

Substituting the covariances into the recursion givesThe denominator has a positive limit when ${\lambda }_{t-1}\to \lambda \in [-1,1]$. Taking limits gives $(\lambda -\theta )(\theta \lambda -1)=0$. The permitted interval selectsFor $\theta =0$, every coefficient is zero; at $\theta =\pm 1$, the roots coincide. This is the limiting MA(1) innovations coefficient. The limiting innovation variance is correspondingly ${\sigma }^2\max (1,{\theta }^2)$.

For the observed numerical case, ${\gamma }_0=2$ and ${\gamma }_1=-1$. The exact innovation calculations areTherefore the predictor of $X_3$ based on the two observations and its error variance areFor $X_4$, the available information is still only ${\mathcal{H}}_2$. Since its covariances with both $X_1$ and $X_2$ are zero, its orthogonal projection onto that span is zero:This is two-step prediction for an MA(1) process. Applying the next one-step recursion would instead use an observed $X_3$, which is not available here.

To generate a chi-squared variable with one degree of freedom by the Marsaglia polar method, repeatedly draw independent $V,W$ uniformly on $(-1,1)$ until $0<S=V^2+W^2<1$. ReturnThe accepted point is uniform on the unit disk; its squared radius is uniform on $(0,1)$ and independent of angle. The radial rescaling gives a standard normal coordinate $Z$, hence $Y\sim {\chi }_1^2$. The same accepted pair also supplies another independent standard normal coordinate, if needed. The proof stages below follow the four printed sampler labels.

Squaring a variable with the standard normal distribution gives the required chi-squared distribution. Its two inverse images at $y>0$ are $\pm \sqrt{y}$, so the change-of-variable formula givesDraw the branch-selection uniform variable independently of the randomness used for the normal draw. Then the pair $(Y,U)$ has the product law: conditioning on $Y$ leaves $U$ uniform. This independence is exactly what justifies interpreting the subsequent threshold as the conditional probability of selecting the smaller root.

Introduce $t=\mu y/(2\lambda )\ge 0$. Solving the quadratic and using its product of roots givesBoth roots are positive, with $x_{-}\le \mu \le x_{+}$; equality occurs only for $y=0$. For computation, the smaller root has the equivalent cancellation-free formThis is stable root evaluation for inverse Gaussian sampling.

Define $h(x)=\lambda (x-\mu {)}^2/({\mu }^{2}x)$. It decreases from infinity to zero on $(0,\mu )$ and increases from zero to infinity on $(\mu ,\infty )$, sinceThe two roots are precisely its inverse branches. This supplies the Jacobians needed for the reciprocal-root inverse Gaussian sampler.

For $0<x<\mu$, only the smaller-root branch can produce a value near $x$. Conditional on $Y=h(x)$, its selection probability is $\mu /(\mu +x)$. The output density on that interval is thereforeUseThe factors cancel to giveThus the selected lower branch has exactly the lower part of the desired inverse Gaussian density, rather than the unweighted density of the smaller root alone.

For $x>\mu$, the corresponding smaller root is ${\mu }^2/x$. The conditional probability of selecting the larger root isIt is the same weight as a function of the actual selected value $x$. HenceCombining the two open intervals proves the output has the Inverse Gaussian distribution $IG(\mu ,\lambda )$. The event $Y=0$ has probability zero, so no atom occurs at $\mu$; the limiting density there is finite. The algorithm always selects one of its positive roots, so these branch densities exhaust its probability distribution.

For the constructed output, its defining quadratic says $h(X)=Y$ identically. Since the preceding branch calculation proves that its law is $IG(\mu ,\lambda )$, any variable with that law has the same pushforward distribution:One can also verify this chi-squared transform of an inverse Gaussian variable directly. The density contributions from the inverse branches areThe final equality follows from $x_{-}{x}_{+}={\mu }^2$. This confirms the transformed law independently of which reciprocal root was selected.

A Gibbs sampler updates one coordinate by drawing from its full conditional distribution, leaving the others unchanged and using their current values. Each coordinate kernel $K_i$ preserves the target joint law $P$: integrate first over the unchanged coordinates, then over the old coordinate, and finally draw the new coordinate from the same conditional distribution. The resulting joint law is again $P$. In the discrete notation, if two states share the remaining coordinates,which is symmetric in $x_i,x_i^{\prime }$. Thus a coordinate update satisfies detailed balance. A systematic full sweep has kernel $K_1\cdots K_k$ and remains invariant because each factor preserves $P$. A fixed-probability random-scan mixture also preserves $P$; the mixture is reversible, whereas a systematic composition need not be.

In the autologistic binary-image model, count each neighboring pair once. Changing one pixel from zero to one raises $n_1$ by one and $n_2$ by the number $z_{ij}$ of neighboring ones. The ratio of its two conditional weights is $\exp (\alpha +\beta z_{ij})$. ThereforeFor single-site Gibbs sampling, select a pixel, compute its current neighbor sum and replace it by a draw from a Bernoulli distribution with this probability. One may use random scans or complete ordered sweeps. The normalizing constant is not needed. For finite $\alpha ,\beta$, every conditional probability lies strictly between zero and one; the finite chain is irreducible and has self-transitions, so its stationary law is unique and iterating converges to it.

For checkerboard Gibbs sampling, no edge joins two black pixels or two white pixels. Conditional on the whites, the black-pixel law factorizes:Draw all black pixels independently with these probabilities, then draw all white pixels independently conditional on the newly drawn blacks. Each color block is a full conditional update, so the two-block sweep preserves the joint target. Pixels of the same color can be updated together because their conditional distribution factorizes.

For the noisy image, the log likelihood contributes $-\frac{1}{2}{\sum }_v(y_v-x_v{)}^2$. Since $x_v^2=x_v$, dropping terms independent of $x$ givesThe Gaussian-noise posterior for an autologistic image therefore has conditional probabilitiesUse this probability in both single-pixel and sequential color-block schemes. The independent observation factors change only the local field, so conditional independence within each color is retained. The same invariance proof then targets the posterior distribution rather than the prior.