Past exam of the options II course of the University of Oxford / 2004 / O (Statistics)

Let $f_0,f_1$ be densities or probability mass functions relative to a common measure, describing two simple hypotheses. A randomized hypothesis test is a measurable function $0\le \phi \le 1$, whose value is the conditional rejection probability. Fix $0<\alpha <1$. The Neyman-Pearson lemma states that a test ${\phi }_{\ast }$ with ${\mathbb{E}}_0{\phi }_{\ast }=\alpha$ is most powerful among tests with size at most $\alpha$ if it rejects where $f_1>c{f}_0$, accepts where $f_1<c{f}_0$, and randomizes as necessary on equality, for a threshold $c\ge 0$. Sets where $f_0=0<f_1$ are included in rejection. A likelihood-ratio quantile and boundary randomization supply such a test.

To prove optimality, let $\phi$ be any competitor with ${\mathbb{E}}_0\phi \le \alpha$. Pointwise, the threshold rule makesIntegrating and rearranging givesThus no competitor of the permitted size has larger statistical power, proving the lemma. Randomization permits exact size when discrete observations cannot supply it with a deterministic rejection region.

The likelihood ratio for the two binomial probabilities isstrictly increasing in $y$. By the Neyman-Pearson lemma, reject in the highest upper tail. Under the null,Hence reject exactly when $Y\ge 4$, with no randomization required. Its statistical power at the stated alternative is

For any fixed $p>1/4$ below 1, the ratio to the null isIt increases with $y$. Its ordering is therefore the same for every such alternative, while the size is determined only by the null. The very same rule, reject when $Y\ge 4$, is most powerful against each alternative and hence is a uniformly most powerful test. At $p=1$ its power is 1, also optimal.

More generally, for $0<p_0<1$, the binomial likelihood ratio against any $p>p_0$ increases with $y$. Choose an integer cutoff $c$ so thatand reject for $Y>c$, randomize with probabilitywhen $Y=c$, and accept below. This gives exact size $\alpha$ and is a uniformly most powerful test for every positive integer $n$. Exact prescribed size may require this boundary randomization; a deterministic cutoff alone need not attain it. For $p_0=0$, reject all $Y>0$ and reject at $Y=0$ with probability $\alpha$, which is likewise optimal. At $p_0=1$ the stated one-sided alternative is empty. The cutoff generally depends on $p_0,n,\alpha$; there is no claim that one fixed cutoff serves every null parameter.

An estimator $T$ of a parameter or estimand $g(\theta )$ is an unbiased estimator if its expectation exists andUnbiasedness concerns the sampling mean of the estimator over repetitions, rather than equality to the target for any particular observed sample.

Maximum likelihood estimation treats the observations as fixed and chooses admissible parameter values maximizing their joint likelihood function, or equivalently its logarithm. Interior score equations identify candidates, but parameter boundaries and global maximality still need checking; an unattained supremum is not an estimate.

The statistical method of moments equates enough empirical moments to the corresponding theoretical moments to solve for the unknown parameters. For example, for a one-parameter exponential rate, the relation $\mathbb{E}X=1/\lambda$ gives the moment estimate $1/\bar{X}$; maximizing ${\lambda }^n{e}^{-\lambda \sum X_i}$ gives the same estimate. Agreement of these two methods in this example does not imply that they agree in every model, nor that either is unbiased.

In a regular one-parameter model with joint density $f_{\theta }(x)$, write the score function as $U_{\theta }={\partial }_{\theta }\log f_{\theta }(X)$ and the Fisher information as $I_n(\theta )={\mathbb{E}}_{\theta }[U_{\theta }^2]$. Assume a parameter-independent support, differentiability and sufficient domination to interchange parameter differentiation and integration, and $0<I_n(\theta )<\infty$. If $T$ has finite variance and is unbiased for a differentiable target $g(\theta )$, the Cramér-Rao lower bound isIndeed, differentiating total mass gives ${\mathbb{E}}_{\theta }{U}_{\theta }=0$, and differentiating ${\mathbb{E}}_{\theta }T=g(\theta )$ gives ${\mathrm{Cov}}_{\theta }(T,U_{\theta })=g^{\prime }(\theta )$. Cauchy-Schwarz inequality proves the bound. For $n$ independent identically distributed observations, information adds: $I_n=n{I}_1$. In particular for the parameter itself, $g(\theta )=\theta$, the bound is $1/I_n$. The assumptions are material: the same formula cannot simply be imposed on models with parameter-dependent support.

Put $S={\sum }_i{X}_i$. Independence and the exponential sampling model give the gamma distribution with densityFor $n>k$, substitution $u=\lambda s$ yieldsAt $k=1$, $n\ge 2$, this gives $\mathbb{E}({\bar{X}}^{-1})=n\lambda /(n-1)$. Thereforeis an unbiased estimator of the rate. For $n>2$, the same formula with $k=2$ givesFor the source-allowed value $n=2$, the first moment is still finite, but the second moment contains an integral proportional to ${\int }_0^1{s}^{-1}{e}^{-\lambda s}ds$ and diverges. Hence the estimator has infinite variance when $n=2$; the finite formula applies only to $n>2$.

The sample score is $n/\lambda -S$, and its Fisher information is $I_n(\lambda )=n/{\lambda }^2$. The Cramér-Rao lower bound for unbiased rate estimation is thus ${\lambda }^2/n$. For every $n>2$, the actual variance is strictly larger by the factor $n/(n-2)$. The ratio approaches one as $n$ grows, but equality is not attained at finite $n$. At $n=2$ the finite-variance form of the bound is inapplicable to this estimator; its infinite variance provides no efficiency equality.