Past exam of the statistics course of the University of Oxford / 2017 (A13765W1)

Source: released examination PDF, bundle pages 412–414.

For the independent statistical sample, write the likelihood function and log-likelihood function asThe expected Fisher information in the whole statistical sample isUnder the regularity conditions that permit differentiation under the integral sign, the score function has expected value zero andFor an interior statistical parameter, assuming identifiability and $0<I_1(\theta )<\infty$, the regular asymptotic normality of a maximum likelihood estimator statesThus the large-sample normal approximation is ${\hat{\theta }}_n\dot{\sim }N(\theta ,I_n(\theta {)}^{-1})$. The inverse information is the leading sampling variance approximation; the displayed convergence specifies its precise asymptotic meaning.

For $0<{\sigma }^2<\infty$, the central limit theorem givesIf $g$ is differentiable at $\mu$, the delta method applies to the first-order expansionwhere the remainder notation means that its product with $\sqrt{n}$ tends to zero in probability. HenceThe useful nondegenerate normal approximation requires $g^{\prime }(\mu )\ne 0$. When $g^{\prime }(\mu )=0$, the first-order limit is zero and a higher-order expansion may be needed.

For a Bernoulli distribution with fixed $0<p<1$, the mean and variance are $p$ and $p(1-p)$. The odds transformation $g(x)=x/(1-x)$ has derivative $g^{\prime }(x)=(1-x{)}^{-2}$. Applying the delta method from part (ii),More precisely, $\sqrt{n}\{\overline{X}/(1-\overline{X})-p/(1-p)\}$ converges to a normal distribution of mean zero and variance $p/(1-p{)}^3$, after defining the statistic arbitrarily on $\overline{X}=1$. That exceptional event has probability $p^n\to 0$, so this extension does not change the limit. The unextended ratio is undefined there and has no finite exact variance if interpreted as infinite. At $p=0$ it is zero almost surely; at $p=1$ the denominator is always zero. This is the delta method for Bernoulli odds.

Put $A={\sum }_i{x}_i$, $B={\sum }_i{x}_i^2$ and $M_2=B/n$. The normal distribution model givesand, up to a constant independent of $\mu$, its log-likelihood function isThe score function and its zero equation areFor $B>0$, this quadratic equation has exactly one positive root, giving the maximum-likelihood estimatorThe numerator $B-A\mu -n{\mu }^2$ of the score function is positive between its negative and positive roots and negative above its positive root. Thus the likelihood function increases up to $\hat{\mu }$ and decreases afterwards: this is its unique global maximum on $(0,\infty )$. Also, $\ell (\mu )\to -\infty$ at both ends. The all-zero observation vector has probability zero; on that exceptional vector $L(\mu )\propto {\mu }^{-n}$ and there is no attained maximum.

The second derivative and the relevant moments areTherefore the expected Fisher information isThe support of a probability distribution is the whole real line, independent of the positive statistical parameter. The law of large numbers gives $\overline{X}\to \mu$ and $M_2\to 2{\mu }^2$ in probability, so the displayed maximum-likelihood estimator is a consistent estimator. The regular asymptotic normality of a maximum likelihood estimator now yieldsThus the requested leading asymptotic variance on the unscaled estimator's scale is ${\mu }^2/(3n)$. The normal mean proportional to standard deviation model provides a reusable version of this calculation.

Let $z=z_{1-\alpha /2}$ be the $(1-\alpha /2)$ standard normal quantile. Since $\hat{\mu }$ is a consistent estimator, Slutsky's theorem givesIts estimated standard error is $\hat{\mu }/\sqrt{3n}$, giving the approximate $(1-\alpha )$ Wald confidence intervalFor each fixed $\mu >0$, this procedure's coverage probability tends to $1-\alpha$. Intersecting with the positive parameter space preserves coverage of the true positive value.

Write $S^2=Q/(n-1)$ and $S=\sqrt{S^2}$, with $n\ge 2$. Two exact facts about a normal sample areand independence of the normal sample mean and sample variance holds. A chi-squared distribution with $n-1$ statistical degrees of freedom is a sum of $n-1$ independent ${\chi }_1^2$ variables, each with mean one and variance two. The central limit theorem therefore givesFor $g(x)=\sqrt{x}$, the derivative at ${\mu }^2$ is $1/(2\mu )$ because $\mu >0$. The delta method gives the asymptotic normality of a normal sample standard deviation:The exact independence of $\overline{X}$ and $S$ makes their limiting normal distributions independent. Consequently,ThusThis has the same leading asymptotic variance as $\hat{\mu }$. The weights $1/3$ and $2/3$ are those in the optimal asymptotic combination of a normal sample mean and standard deviation. This does not require $S$ to be an exactly unbiased estimator of $\mu$.

The Neyman-Pearson lemma concerns a simple hypothesis $H_0:\theta ={\theta }_0$ against another simple hypothesis $H_1:\theta ={\theta }_1$. For a continuous statistical sample, put $f_j(x)={\prod }_{i}f(x_i;{\theta }_j)$ for the joint probability density functions, $j=0,1$. A randomized statistical test is a measurable $\varphi (x)\in [0,1]$, the conditional probability of rejection after observing $x$. Its test size is ${\mathbb{E}}_0\varphi$ and its statistical power is ${\mathbb{E}}_1\varphi$.

Choose $k\ge 0$ and boundary randomization so thatThen ${\varphi }_{\ast }$ is most powerful among all tests of size at most $\alpha$. Thus one rejects for sufficiently large alternative-to-null likelihood ratio. If the equality set has zero probability under $H_0$, the test needs no randomization. Continuous observations alone do not guarantee that their likelihood ratio has a continuous probability distribution.

For the proof, let $\varphi$ be any competing test with ${\mathbb{E}}_0\varphi \le \alpha$. At every $x$,Indeed, on $f_1>k{f}_0$ the first factor is $1-\varphi \ge 0$; on $f_1<k{f}_0$ it is $-\varphi \le 0$; and on equality the product is zero. Integration giveswhich proves the claimed optimality. Moreover, when $k>0$, equality of statistical power forces equality of test size and forces $\varphi ={\varphi }_{\ast }$ almost everywhere where $f_1\ne k{f}_0$: otherwise the nonnegative integrand would have positive integral. This proves the usual necessity assertion as well, up to equality sets and sets of zero probability under both hypotheses.

Let $S={\sum }_i{X}_i$. For the exponential distribution with mean $\theta$, the likelihood function is $L(\theta )={\theta }^{-n}{e}^{-S/\theta }$ on positive observations. Fix any $0<{\theta }_1<{\theta }_0$. The alternative-to-null likelihood ratio isSince $1/{\theta }_1-1/{\theta }_0>0$, this is strictly decreasing in $S$. The Neyman-Pearson lemma says to reject for small $S$. Under the null hypothesis, the given sum rule yields $S\sim \mathrm{Gamma}(n,1/{\theta }_0)$, where the second argument is a rate, and henceLet $q_{\alpha }$ be the lower $\alpha$ quantile of this chi-squared distribution. The required critical region isIts test size is exactly $\alpha$ because the null distribution is continuous. Its cutoff is the same for every ${\theta }_1<{\theta }_0$, so it is simultaneously most powerful against every such value: it is a uniformly most powerful test. Equivalently, reject if $S$ is below the lower $\alpha$ quantile of the specified gamma distribution.

Put $A={\sum }_i{X}_i$, $B={\sum }_j{Y}_j$ and $N=n+m$. Independence gives the likelihood functionThe unrestricted maximum-likelihood estimators are $\hat{\theta }=A/n$ and $\hat{\mu }=B/m$. Under the null hypothesis, the common mean $\eta =\theta =\mu$ maximizes ${\eta }^{-N}{e}^{-(A+B)/\eta }$ at $\hat{\eta }=(A+B)/N$. These are maxima because each one-parameter score function has the sign of its observed sum minus its sample size times its proposed mean.

Define the generalized likelihood ratio to be the constrained maximum divided by the unrestricted maximum. The exponential factors cancel, givingSince $T=A/(A+B)$ and $1-T=B/(A+B)$,It has maximum one at $T=n/N$, when the two fitted means agree, and is small toward either endpoint. A likelihood-ratio test rejects for small $\Lambda$, or equivalently large $W=-2\log \Lambda$. This is the likelihood-ratio test for equality of exponential means.

Under the null hypothesis, the full model has two free positive statistical parameters and the constrained model has one. The Wilks theorem therefore givesas both sample sizes tend to infinity, with $n/(n+m)$ tending to a number strictly between zero and one. For the observed $t$, calculateThe large-sample p-value is the upper-tail probabilitywhere $\Phi$ is the cumulative distribution function of the standard normal distribution. The last equality uses the fact that the square of a standard normal random variable has a chi-squared distribution with one statistical degree of freedom. Values of $T$ on either side of $n/(n+m)$ are already included by the transformation to $W$.

Let $s={\sum }_i{x}_i$. The Bernoulli distribution likelihood function is ${\theta }^s(1-\theta {)}^{n-s}$. Multiplication by the beta distribution prior density and Bayes' theorem giveRecognizing the normalized beta distribution,This is Beta-binomial conjugacy; the same update applies whether the data are recorded as the individual Bernoulli trials or their success count.

Conditional on $\theta$, the future Bernoulli random variable is independent of the observed statistical sample and has success probability $\theta$. Its posterior predictive probability is therefore the posterior mean of $\theta$. Put $A=a+s$ and $B=b+n-s$. Thenusing the beta-gamma identity and the Gamma function recurrence. HenceThe whole posterior predictive distribution of $X_{n+1}$ is the Bernoulli distribution with this success probability.

For a specified posterior probability $1-\alpha$, an interval $I=({\theta }_1,{\theta }_2)$ is a highest posterior density interval ifand every included point has posterior density at least as large as every excluded point, apart from boundary and null-set choices. Equivalently, $I$ is a level set $\{\theta :\pi (\theta \mid x)>k\}$, with possible choices on its boundary, whose posterior probability is $1-\alpha$.

This selects the most densely supported values of the statistical parameter. When the relevant level set is one interval and has no plateau of positive probability, it is the shortest credible interval of that posterior probability. With several separated peaks, the corresponding highest density region can instead be disconnected, so an interval need not exist. The definition is in the stated parameter coordinate: posterior density ordering can change under a nonlinear change of parameter.

Put $A=a+s>1$ and $B=b+n-s>1$ as in part (a), and let $F_{A,B}$ be the cumulative distribution function of the beta distribution $\mathrm{Beta}(A,B)$. The two endpoint equations areThe first equation fixes the posterior probability, and the second makes the two endpoint posterior densities equal. Their admissible ordering isTo justify sufficiency and uniqueness, the derivative of the logarithm of the posterior density iswhich is positive below $m=(A-1)/(A+B-2)$ and negative above it. Thus the posterior density increases strictly to its unique mode and decreases strictly thereafter, vanishing at both ends. Each density threshold $k$ between zero and the maximum determines exactly one endpoint on each side of $m$. The probability between these endpoints decreases continuously and strictly from one to zero as $k$ increases, so exactly one threshold gives 0.95. This is the beta highest posterior density endpoint characterization.

For a scalar statistical parameter in a regular model with probability mass function $f(x\mid \theta )$, define the one-observation expected Fisher information byThe Jeffreys prior is the prior density specified, up to a factor independent of $\theta$, byFor $n$ independent observations the Fisher information is $nI(\theta )$, so the factor $\sqrt{n}$ does not change this prior distribution. If $\int \sqrt{I(\theta )}d\theta$ is finite, divide by this integral to normalize; otherwise the rule defines an improper prior.

The Jeffreys prior is invariant under smooth one-to-one reparameterization. This means that applying the rule directly in a new parameter $\eta =h(\theta )$ gives the same prior distribution as first constructing the rule for $\theta$ and then transforming its prior density by the change-of-variable formula.

More explicitly, for a differentiable invertible change of parameter with nonzero derivative,Thus ${\pi }_{J,\eta }(\eta )d\eta$ and ${\pi }_{J,\theta }(\theta )d\theta$ represent the same measure, up to the normalization constant. Invariance concerns the measure: the numerical values of the prior densities need not be equal in the two coordinates. The same change of variables then makes the posterior distributions agree. This is the reparameterization invariance of Jeffreys prior.

For one Bernoulli random variable,Its variance is $\theta (1-\theta )$, so the expected Fisher information and Jeffreys prior areThe normalizer is $B(1/2,1/2)=\pi$. ThereforeThis Jeffreys prior for a Bernoulli parameter is proper despite its integrable endpoint singularities. It is the arcsine distribution. It replaces the specified $a,b>1$ prior distribution in part (a) when the Jeffreys prior rule is chosen; it is not one of those restricted priors.

Let $y_{(n)}={\max }_i{y}_i$ and $c=\max (\beta ,y_{(n)})$. For the observed positive data, the uniform distribution likelihood function isMultiplying by the Pareto distribution prior density gives the posterior density kernel ${\theta }^{-(\alpha +n+1)}{1}_{\{\theta >c\}}$. Sincethe normalized posterior distribution isThis is uniform-Pareto conjugacy. The posterior density is strictly decreasing over its support, so the highest posterior density interval starts at $c$, where the admissible density is largest in the limiting sense. Its upper endpoint $u$ satisfiesHence a 95% highest posterior density interval isInclusion of endpoints does not change its posterior probability. Every point inside has higher posterior density than every admissible point to its right, and points below $c$ have density zero. This is the Pareto highest density interval; equal endpoint densities are unnecessary because one endpoint is the support boundary.