Past exam of the options I course of the University of Oxford / 2012 / O (Statistics)

Let ${\ell }_n(\theta )={\sum }_{i=1}^n\log f(X_i;\theta )$ be the log-likelihood. Under the usual regularity conditions, the expected Fisher information in the whole sample isThe equality of the two expressions uses mean-zero score and permitted differentiation of the normalization identity. Independent observations make the information additive. The maximum-likelihood estimator has asymptotic normality:The factor $n$ is included in $I_n$; it must not be inserted a second time into the variance.

Let $z={\Phi }^{-1}(1-\alpha /2)$ for the standard normal cumulative distribution function. Consistency of the maximum-likelihood estimator and Slutsky's theorem give the asymptotic standardizationIts probability of lying between $-z$ and $z$ tends to $1-\alpha$. Solving those inequalities for the unknown parameter yields the Wald confidence intervalThus the estimated standard error is $I_n(\hat{\theta }{)}^{-1/2}$. Its coverage is approximate and relies on the large-sample regularity assumptions.

For the stated geometric distribution, write $S={\sum }_i{X}_i$. ThenSince $\mathbb{E}{X}_i=1/\theta$, we have $\mathbb{E}(S-n)=n(1-\theta )/\theta$. Taking expectation gives the sample Fisher informationThe support begins at one, so this expectation uses the trials-until-first-success convention of the geometric law.

For the Bernoulli distribution, with $S={\sum }_i{X}_i$,Using $\mathbb{E}S=n\theta$, the expected Fisher information is

For the same sample size $n$, the geometric distribution provides more Fisher information:The corresponding asymptotic variances of the maximum-likelihood estimators areHence choose the geometric sample for more precise inference at equal numbers of observations. Its asymptotic variance is $\theta$ times as large, and its standard error is $\sqrt{\theta }$ times as large. This illustrates Fisher information per observation and per trial. It is a per-observation comparison: one geometric observation records a waiting time containing, on average, $1/\theta$ Bernoulli trials. A constraint on the total number of underlying trials would be a different sampling-budget comparison.

For the full-column-rank design matrix, the normal-error maximum-likelihood estimator is the least-squares estimatorIndeed maximizing the Gaussian likelihood over $\theta$ minimizes $\Vert Y-X\theta {\Vert }^2$, whose normal equations are $X^{T}X\hat{\theta }=X^{T}Y$; rank $p$ makes $X^{T}X$ invertible. Substitution of the model givesSince $\mathbb{E}ϵ=0$, it follows that $\boxed{\mathbb{E}\hat{\theta }=\theta }$, proving unbiasedness.

Independence and common error variance give $\mathrm{Var}(ϵ)={\sigma }^2{I}_n$. For the fixed matrix $A=(X^{T}X{)}^{-1}{X}^T$, the covariance transformation rule givesThus the covariance is determined by the error variance and the inverse Gram matrix of the design columns.

The design columns are $1$, $x$ and $z$. Their pairwise inner products are zero by the three supplied sums. Put $S_{xx}={\sum }_i{x}_i^2$ and $S_{zz}={\sum }_i{z}_i^2$, positive under the full-rank assumption. ThenApplying part (a),The off-diagonal covariances vanish because the design columns are orthogonal. With Gaussian errors, the coefficient estimators are also jointly normal and therefore independent.

The future mean response is $E_{\ast }=\alpha +\beta x_{\ast }+\gamma z_{\ast }$. By invariance of maximum likelihood estimation, its estimator is the fitted meanIts sampling variance is the quadratic form of the coefficient covariance matrix with $c=(1,x_{\ast },z_{\ast }{)}^T$:This estimator is unbiased because each fitted coefficient is unbiased. The future error ${ϵ}_{\ast }$ does not enter the estimator of the mean, so its independent variance is not part of this expression.