Past exam of the mathematics course of the University of Cambridge / 2014 / iii / Paper 35 / 2

For a regular one-parameter sampling distribution, the Fisher information and Jeffreys prior areUnder the usual differentiation and integrability conditions, $I(\theta )=-{\mathbb{E}}_{\theta }[{\partial }_{\theta }^2\log p_Y(Y\mid \theta )]$. This prior distribution transforms as a density under smooth one-to-one reparameterizations, so the rule is coordinate invariant. Its integral need not be finite; posterior propriety must still be established if it is an improper prior.

For a binomial distribution, the score function isIts squared expected value is $I(\theta )=n/[\theta (1-\theta )]$, using the binomial distribution variance $n\theta (1-\theta )$. Hence the Jeffreys prior isthe Beta distribution $\mathrm{Beta}(1/2,1/2)$. The factor $\sqrt{n}$ is independent of $\theta$ and disappears on normalization.

Independent Jeffreys priors and the two independent binomial distribution likelihood factors give, by Beta-binomial conjugacy,The Bayesian posteriors remain independent because each observation factor involves only its own probability. Their posterior means are respectively $0.7$ and $0.3$. Notice that $p_T$ is the probability of saying milk first when tea was first, rather than the probability of a correct tea identification.

In Markov chain Monte Carlo, retain the derived quantity $\delta =p_M-p_T$ at every iteration. Rough BUGS code using the actual observations isSupply milkAnswers=3 and teaAnswers=1. Summarize delta by its posterior mean, empirical quantiles and credible interval; the average of positive estimates $\mathbb{P}(p_M>p_T\mid \mathcal{D})$. Here $\mathbb{E}[\delta \mid \mathcal{D}]=0.4$. Check Markov chain Monte Carlo convergence diagnostics before interpreting the simulation. Since both Bayesian posteriors are independent known Beta distributions, direct independent sampling is an equally valid, simpler way to obtain the same summaries.

Start with the Beta distribution draw in the Beta-binomial exchangeable coupling. Write $s=\alpha +\beta$ andThese are the expected value and variance of $p_M$. The fixed $n$ in this coupling is a dependence parameter chosen by the investigator, not necessarily the observed number of cups. The four numbered stages together address the single printed correlation request.

Conditional on $p_M$, the auxiliary binomial distribution gives $\mathbb{E}[X/n\mid p_M]=p_M$ and $\mathrm{Var}(X/n\mid p_M)=p_M(1-p_M)/n$. Thus large $n$ makes $X/n$ close to $p_M$. The law of total variance also givesThe latent variable encodes the first draw increasingly accurately.

The next Beta distribution has conditional expected value and varianceFor large $n$, its expected value approaches $X/n$, while its variance is at most $1/[4(s+n+1)]$. The final draw therefore stays close to the first. These are prior distributions for the cup probabilities: the auxiliary count is a device for constructing dependence, rather than additional observed tea data.

The law of iterated expectation yieldsOne may obtain $\mathrm{Var}(p_T)=v$ either from its unchanged marginal distribution proved below or directly from the law of total variance: the variance of its conditional mean is $nv/(s+n)$ and its expected conditional variance is $sv/(s+n)$. Hence the correlation coefficient is exactlyAlso $\mathbb{E}[(p_T-p_M{)}^2]=2vs/(s+n)\to 0$. The two probabilities become close while keeping their original beta marginals.

Apply the law of iterated expectation to the auxiliary binomial distribution:This expectation is over the full prior distribution construction, not over a particular observed auxiliary count.

Using the conditional Beta distribution expected value and the law of iterated expectation,Equality of the expected values alone does not prove equality of the marginal distributions; the later exchangeability argument does.

Multiply the Beta distribution density of $p_M$, the conditional binomial distribution mass of $X$, and the conditional Beta distribution density of $p_T$. With $B(a,b)$ the beta function, the full joint density, relative to counting measure in $x$ and Lebesgue measure in the two probabilities, isHere $x=0,\dots ,n$ and both probabilities lie in $(0,1)$. The printed joint expression omits the factor $1/B(\alpha +x,\beta +n-x)$. It is a constant when $x$ is fixed and only the two probabilities vary, but is not a constant for the full joint probability distribution. Keeping it is essential when summing over the latent variable.

Two exchangeable random variables satisfy $(X,Y)\stackrel{d}{=}(Y,X)$: their joint probability distribution is unchanged by swapping the coordinates. Thus for every measurable set $A$,Exchangeability implies identical marginal distributions, but does not imply independence.

For each auxiliary count $x$, the correctly normalized joint expression above is symmetric in $p_M,p_T$. Summing it over $x=0,\dots ,n$ preserves that symmetry, so these are exchangeable random variables. Their marginal distributions are therefore identical. Since $p_M$ was generated from a Beta distribution,The Beta-binomial exchangeable coupling changes the dependence, while leaving both prior distributions unchanged.