Past exam of the options II course of the University of Oxford / 2004 / O / O5

An estimator $T$ of a parameter or estimand $g(\theta )$ is an unbiased estimator if its expectation exists andUnbiasedness concerns the sampling mean of the estimator over repetitions, rather than equality to the target for any particular observed sample.

Maximum likelihood estimation treats the observations as fixed and chooses admissible parameter values maximizing their joint likelihood function, or equivalently its logarithm. Interior score equations identify candidates, but parameter boundaries and global maximality still need checking; an unattained supremum is not an estimate.

The statistical method of moments equates enough empirical moments to the corresponding theoretical moments to solve for the unknown parameters. For example, for a one-parameter exponential rate, the relation $\mathbb{E}X=1/\lambda$ gives the moment estimate $1/\bar{X}$; maximizing ${\lambda }^n{e}^{-\lambda \sum X_i}$ gives the same estimate. Agreement of these two methods in this example does not imply that they agree in every model, nor that either is unbiased.

In a regular one-parameter model with joint density $f_{\theta }(x)$, write the score function as $U_{\theta }={\partial }_{\theta }\log f_{\theta }(X)$ and the Fisher information as $I_n(\theta )={\mathbb{E}}_{\theta }[U_{\theta }^2]$. Assume a parameter-independent support, differentiability and sufficient domination to interchange parameter differentiation and integration, and $0<I_n(\theta )<\infty$. If $T$ has finite variance and is unbiased for a differentiable target $g(\theta )$, the Cramér-Rao lower bound isIndeed, differentiating total mass gives ${\mathbb{E}}_{\theta }{U}_{\theta }=0$, and differentiating ${\mathbb{E}}_{\theta }T=g(\theta )$ gives ${\mathrm{Cov}}_{\theta }(T,U_{\theta })=g^{\prime }(\theta )$. Cauchy-Schwarz inequality proves the bound. For $n$ independent identically distributed observations, information adds: $I_n=n{I}_1$. In particular for the parameter itself, $g(\theta )=\theta$, the bound is $1/I_n$. The assumptions are material: the same formula cannot simply be imposed on models with parameter-dependent support.

Put $S={\sum }_i{X}_i$. Independence and the exponential sampling model give the gamma distribution with densityFor $n>k$, substitution $u=\lambda s$ yieldsAt $k=1$, $n\ge 2$, this gives $\mathbb{E}({\bar{X}}^{-1})=n\lambda /(n-1)$. Thereforeis an unbiased estimator of the rate. For $n>2$, the same formula with $k=2$ givesFor the source-allowed value $n=2$, the first moment is still finite, but the second moment contains an integral proportional to ${\int }_0^1{s}^{-1}{e}^{-\lambda s}ds$ and diverges. Hence the estimator has infinite variance when $n=2$; the finite formula applies only to $n>2$.

The sample score is $n/\lambda -S$, and its Fisher information is $I_n(\lambda )=n/{\lambda }^2$. The Cramér-Rao lower bound for unbiased rate estimation is thus ${\lambda }^2/n$. For every $n>2$, the actual variance is strictly larger by the factor $n/(n-2)$. The ratio approaches one as $n$ grows, but equality is not attained at finite $n$. At $n=2$ the finite-variance form of the bound is inapplicable to this estimator; its infinite variance provides no efficiency equality.