Past exam of the mathematics course of the University of Cambridge / 2014 / iii / Paper 35 / 4 / j / Solution

Fit both the normal and Student t random-effect model with comparable proper prior distributions. Compare priors on the same spread measure: a normal distribution scale $\tau$ is a standard deviation, whereas the $t_4$ standard deviation is $\sqrt{2}\psi$.

Use a posterior predictive check: draw study effects and binomial counts from each fitted hierarchy and compare replicated dispersion and extreme study contrasts with the observations. For predicting a new study, generate a new effect from the hierarchy rather than reusing an existing fitted effect. A Leave-one-out cross-validation with entire studies held out can compare integrated predictive probabilities for both arms of each omitted trial, averaging over hyperparameters and its unobserved study effect. Leave-one-study-out influence analysis also reveals whether the difference is driven by a single trial.

Prefer the heavier-tailed hierarchy if it improves the relevant predictive checks and held-out study predictions robustly to reasonable prior choices. The deviance information criterion can supplement the comparison, but its effective parameter count can depend on the latent-variable representation, and six studies give limited information about tail shape. A small numerical criterion difference alone is insufficient evidence.