Past exam of the Probability, statistics and options II course of the University of Oxford / 2009 / O (Statistics)

Minimizing $\Vert Y-X\theta {\Vert }^2$ gives the normal equations $X^{T}X\hat{\theta }=X^{T}Y$. Since $X$ has full column rank, $X^{T}X$ is positive definite and invertible, and the objective has a unique minimum. Set $A=(X^{T}X{)}^{-1}{X}^T$; then $\hat{\theta }=AY$ and $AX=I_p$. The model gives $\mathbb{E}[Y]=X\theta$ and $\mathrm{Var}(Y)={\sigma }^2{I}_n$, so $\mathbb{E}[\hat{\theta }]=\theta$ and $\mathrm{Var}(\hat{\theta })={\sigma }^{2}A{A}^T={\sigma }^2(X^{T}X{)}^{-1}$. For the alternative estimator the corresponding general formulas are $\mathbb{E}[\tilde{\theta }]=DX\theta$ and $\mathrm{Var}(\tilde{\theta })={\sigma }^{2}D{D}^T$.

Unbiasedness as an estimator of the parameter means $DX\theta =\theta$ for every $\theta \in {\mathbb{R}}^p$, rather than just one particular value. Thus $(DX-I_p)\theta =0$ for every vector, forcing $\boxed{DX=I_p}$. Conversely that identity plainly suffices for unbiasedness.

Put $D^{\ast }=D-A$. By (i), $D^{\ast }X=0$. Thus $D^{\ast }{A}^T=D^{\ast }X(X^{T}X{)}^{-1}=0$, and its transpose vanishes too. Expanding $D{D}^T=(D^{\ast }+A)(D^{\ast }+A{)}^T$ yieldsThe covariance difference is positive semidefinite. In particular its $i$th diagonal entry is ${\sigma }^2{\sum }_j(D_{ij}^{\ast }{)}^2\ge 0$, proving the requested componentwise variance inequalities. Prefer $\boxed{{\hat{\theta }}_1}$ within the class of linear unbiased estimators, since it has the least variance and the same expectation. A tie is possible if the first row of $D^{\ast }$ is zero. This is the Gauss-Markov theorem conclusion; no normal-distribution assumption on the errors is required.

The multinomial likelihood is a parameter-independent factor times ${\prod }_i{\pi }_i^{n_i}$. Maximize its logarithm subject to $\sum {\pi }_i=1$: the multiplier equations give $n_i/{\pi }_i=c$, and summation makes $c=n$. Thus ${\hat{\pi }}_i=n_i/n$. Positivity of all counts makes this the unique interior maximum by strict concavity.

The likelihood ratio compares the fitted null likelihood with this saturated likelihood. Taking minus twice its logarithm gives $\Lambda (\hat{\theta })=2{\sum }_i{n}_i\log [n_i/(n{\pi }_i(\hat{\theta }))]$. Under regularity and large expected counts, compare it to ${\chi }_{k-2}^2$ for one fitted parameter: the saturated probabilities have $k-1$ free coordinates and the null has one. If the parameter is fixed at ${\theta }_0$, use $\Lambda ({\theta }_0)$ and ${\chi }_{k-1}^2$. Reject for a value above the chosen upper quantile.

For the ungrouped logarithmic-series observations write $n=1188$ and $S={\sum }_r{X}_r=6968$. The log likelihood isSetting its derivative to zero givesBisection of this monotone mean equation, or Newton's method with a valid bracket, obtains ${\hat{\theta }}_{\mathrm{full}}\approx 0.9442280$.

The supplied table instead uses ten grouped cells, with the last probability ${\pi }_{\ge 10}(\theta )=1-{\sum }_{i=1}^9{\pi }_i(\theta )$. Its likelihood is determined by the cell counts and is not the ungrouped likelihood just differentiated. From its listed values, the minimum deviance occurs near $\boxed{{\hat{\theta }}_{\mathrm{grouped}}\approx 0.93}$; direct minimization of this grouped deviance gives about $0.9333714$. The supplied two forms of the estimate therefore concern different likelihoods. They must not be identified: the full-data score equation gives about $0.94423$, while the grouped table points near $0.93$. The following requested tests use the table's grouped likelihood, with ten cells. The source values and totals are preserved unchanged in TeX.

For the specified parameter there is no fitted parameter, so the null reference has $10-1=9$ degrees of freedom. The given deviance is $15.59$, below the supplied five-percent critical value $16.92$. Hence do not reject the fixed log-series model at level five percent. The parameter is fixed in this test, so subtracting an additional degree of freedom would be inappropriate.

With one parameter fitted to the grouped cells, the reference degrees of freedom are $10-1-1=8$. The table already gives a deviance of $8.29$ at $0.93$, so the minimized deviance is no larger than that; its direct grouped minimum is about $7.76$. Either is below the supplied critical value $15.51$. Therefore do not reject the fitted grouped log-series model at level five percent. The estimate being profiled in this test is the grouped-likelihood estimate described above, not the different full-data score estimate.