Past exam of the statistics course of the University of Oxford / 2017 / 3

Let $s={\sum }_i{x}_i$. The Bernoulli distribution likelihood function is ${\theta }^s(1-\theta {)}^{n-s}$. Multiplication by the beta distribution prior density and Bayes' theorem giveRecognizing the normalized beta distribution,This is Beta-binomial conjugacy; the same update applies whether the data are recorded as the individual Bernoulli trials or their success count.

Conditional on $\theta$, the future Bernoulli random variable is independent of the observed statistical sample and has success probability $\theta$. Its posterior predictive probability is therefore the posterior mean of $\theta$. Put $A=a+s$ and $B=b+n-s$. Thenusing the beta-gamma identity and the Gamma function recurrence. HenceThe whole posterior predictive distribution of $X_{n+1}$ is the Bernoulli distribution with this success probability.

For a specified posterior probability $1-\alpha$, an interval $I=({\theta }_1,{\theta }_2)$ is a highest posterior density interval ifand every included point has posterior density at least as large as every excluded point, apart from boundary and null-set choices. Equivalently, $I$ is a level set $\{\theta :\pi (\theta \mid x)>k\}$, with possible choices on its boundary, whose posterior probability is $1-\alpha$.

This selects the most densely supported values of the statistical parameter. When the relevant level set is one interval and has no plateau of positive probability, it is the shortest credible interval of that posterior probability. With several separated peaks, the corresponding highest density region can instead be disconnected, so an interval need not exist. The definition is in the stated parameter coordinate: posterior density ordering can change under a nonlinear change of parameter.

Put $A=a+s>1$ and $B=b+n-s>1$ as in part (a), and let $F_{A,B}$ be the cumulative distribution function of the beta distribution $\mathrm{Beta}(A,B)$. The two endpoint equations areThe first equation fixes the posterior probability, and the second makes the two endpoint posterior densities equal. Their admissible ordering isTo justify sufficiency and uniqueness, the derivative of the logarithm of the posterior density iswhich is positive below $m=(A-1)/(A+B-2)$ and negative above it. Thus the posterior density increases strictly to its unique mode and decreases strictly thereafter, vanishing at both ends. Each density threshold $k$ between zero and the maximum determines exactly one endpoint on each side of $m$. The probability between these endpoints decreases continuously and strictly from one to zero as $k$ increases, so exactly one threshold gives 0.95. This is the beta highest posterior density endpoint characterization.

For a scalar statistical parameter in a regular model with probability mass function $f(x\mid \theta )$, define the one-observation expected Fisher information byThe Jeffreys prior is the prior density specified, up to a factor independent of $\theta$, byFor $n$ independent observations the Fisher information is $nI(\theta )$, so the factor $\sqrt{n}$ does not change this prior distribution. If $\int \sqrt{I(\theta )}d\theta$ is finite, divide by this integral to normalize; otherwise the rule defines an improper prior.

The Jeffreys prior is invariant under smooth one-to-one reparameterization. This means that applying the rule directly in a new parameter $\eta =h(\theta )$ gives the same prior distribution as first constructing the rule for $\theta$ and then transforming its prior density by the change-of-variable formula.

More explicitly, for a differentiable invertible change of parameter with nonzero derivative,Thus ${\pi }_{J,\eta }(\eta )d\eta$ and ${\pi }_{J,\theta }(\theta )d\theta$ represent the same measure, up to the normalization constant. Invariance concerns the measure: the numerical values of the prior densities need not be equal in the two coordinates. The same change of variables then makes the posterior distributions agree. This is the reparameterization invariance of Jeffreys prior.

For one Bernoulli random variable,Its variance is $\theta (1-\theta )$, so the expected Fisher information and Jeffreys prior areThe normalizer is $B(1/2,1/2)=\pi$. ThereforeThis Jeffreys prior for a Bernoulli parameter is proper despite its integrable endpoint singularities. It is the arcsine distribution. It replaces the specified $a,b>1$ prior distribution in part (a) when the Jeffreys prior rule is chosen; it is not one of those restricted priors.

Let $y_{(n)}={\max }_i{y}_i$ and $c=\max (\beta ,y_{(n)})$. For the observed positive data, the uniform distribution likelihood function isMultiplying by the Pareto distribution prior density gives the posterior density kernel ${\theta }^{-(\alpha +n+1)}{1}_{\{\theta >c\}}$. Sincethe normalized posterior distribution isThis is uniform-Pareto conjugacy. The posterior density is strictly decreasing over its support, so the highest posterior density interval starts at $c$, where the admissible density is largest in the limiting sense. Its upper endpoint $u$ satisfiesHence a 95% highest posterior density interval isInclusion of endpoints does not change its posterior probability. Every point inside has higher posterior density than every admissible point to its right, and points below $c$ have density zero. This is the Pareto highest density interval; equal endpoint densities are unnecessary because one endpoint is the support boundary.