Past exam of the mathematics course of the University of Cambridge / 2007 / iii / Paper 44 (Statistical Theory)

A useful full statement is the rank-sum form of Cochran's theorem. Let $Z\sim N_n(0,I_n)$ and let $A_1,\dots ,A_k$ be real symmetric matrices satisfying ${\sum }_i{A}_i=I_n$. Put $r_i=\mathrm{rank}(A_i)$ and $Q_i=Z^T{A}_{i}Z$. Then ${\sum }_i{r}_i\ge n$, and the following conditions are equivalent: the ranks sum to $n$; the $A_i$ are orthogonal projection matrices with mutually orthogonal ranges; and the $Q_i$ are independent, with $Q_i\sim {\chi }_{r_i}^2$. A rank-zero form is identically zero. In particular, the usual projection version of Cochran's theorem partitions the squared length of a standard normal vector into independent chi-squared distributions.

Here is a proof which also explains the rank condition. Let $L_i=\mathrm{im}{A}_i$. Since every vector satisfies $x={\sum }_i{A}_{i}x$, the spaces $L_i$ span ${\mathbb{R}}^n$. Therefore their dimensions sum to at least $n$. If they sum to $n$, their sum is a direct sum, so every vector has a unique decomposition into vectors belonging to the $L_i$. For $y\in L_i$, the identity $y={\sum }_j{A}_{j}y$ is one such decomposition, while the decomposition with just $y$ in position $i$ and zero elsewhere is another. Uniqueness gives $A_{j}y={\delta }_{ij}y$. ThusA symmetric matrix with this idempotence is an orthogonal projection matrix. Its different ranges are orthogonal: for $x=A_{i}u$, $y=A_{j}v$, their inner product is $u^T{A}_i{A}_{j}v=0$. Choose an orthonormal basis for each range and concatenate them. The resulting change-of-basis matrix is an orthogonal matrix. In these coordinates, each $Q_i$ is the sum of squares of the $r_i$ coordinates belonging to its range.

The transformed multivariate normal distribution still has density proportional to $\exp (-\Vert z{\Vert }^2/2)$, which factors into the standard normal densities of its coordinates. The coordinates are independent, so the disjoint groups of squared coordinates give independent chi-squared distributions of the asserted degrees of freedom. This proves the forward implication. Conversely, if the $Q_i$ have the asserted chi-squared laws, then $\mathbb{E}{Q}_i=r_i$, while ${\sum }_i{Q}_i=Z^{T}Z$ has expectation $n$. Taking expectations yields ${\sum }_i{r}_i=n$. This completes the equivalence and the proof of Cochran's theorem.

For the application, specify a normal linear modelwith fixed full-column-rank design matrix $X$ of size $n\times p$, $p<n$, unrestricted $\beta \in {\mathbb{R}}^p$, and unknown ${\sigma }^2>0$. Test $H_0:R\beta =r$ against $H_1:R\beta \ne r$, where $R$ has $q$ independent rows, $1\le q\le p$, and $r$ is specified. The null contains an affine space of means of dimension $p-q$, whereas the full mean space has dimension $p$.

Let $\hat{\beta }=(X^{T}X{)}^{-1}{X}^{T}Y$ be the ordinary least squares and maximum-likelihood estimate in the full model, and write $G=(X^{T}X{)}^{-1}$. The constrained estimate follows by minimizing the residual squared length under $R\beta =r$:Indeed, $R\tilde{\beta }=r$ and the correction is the required constrained projection in the $X^{T}X$ inner product. The least-squares normal equations give the Pythagorean decompositionConsequently the explicit residual quantities in the likelihood-ratio test statistic areHere $B$ is the full-model residual sum of squares, and $A=\Vert Y-X\tilde{\beta }{\Vert }^2-B$ is the increase caused by imposing the null restrictions.

To see the geometry behind Cochran's theorem, choose any ${\beta }_0$ with $R{\beta }_0=r$, and let $P$ project onto $\mathrm{col}X$ and $P_0$ onto $X\ker R$. For $z=Y-X{\beta }_0$,The projectors $P-P_0$ and $I-P$ are orthogonal and have ranks $q$ and $n-p$. Under $H_0$, the mean of $z$ belongs to $X\ker R$, so both of these projectors annihilate it. Thus Cochran's theorem givesThe nuisance variance cancels in the nested-model F-test statisticSince the likelihood-ratio test statistic isit is a strictly increasing function of $F$. Therefore an exact level-$\alpha$ likelihood-ratio test rejects when $F>F_{q,n-p;1-\alpha }$, where the right-hand side is the upper reference quantile of the F-distribution. This calibration is exact under the normal linear model, rather than an asymptotic chi-squared replacement.

Write $K(t)=\log \mathbb{E}[e^{tY}]$ for the cumulant-generating function in the question's $(\varphi ,\lambda )$ parametrization. Multiplying the density by $e^{ty}$ replaces $\varphi$ in the decaying part of its exponent by $\varphi -2t$. For $t<\varphi /2$, the remaining integral is the normalizing integral for an Inverse Gaussian distribution with parameters $(\varphi -2t,\lambda )$. Comparing the two normalizing constants givesIn particular $K^{\prime }(0)=\sqrt{\lambda /\varphi }$, so the mean parameter used in the more usual Inverse Gaussian distribution notation is $\mu =\sqrt{\lambda /\varphi }$; it must not be confused with the present parameter $\varphi$.

First compute the cumulant-generating function of the sample average, as requested. Independence givesThis is exactly the cumulant-generating function of an Inverse Gaussian distribution with parameters $(n\varphi ,n\lambda )$. The uniqueness theorem for moment-generating functions therefore identifies that law. Scaling $S_n=n{\overline{Y}}_n$ and including the density Jacobian $1/n$ givesEquivalently, the inverse Gaussian sum closure is $S_n\sim IG(\varphi ,n^2\lambda )$ in the parametrization of this question. Its mean is $n\sqrt{\lambda /\varphi }$ and its variance is $n\sqrt{\lambda }/{\varphi }^{3/2}$, in agreement with addition of independent means and variances.

For a general continuous iid sample with cumulant-generating function $K$, finite near zero, set $x=s/n$ and choose a real saddle $\hat{t}$ in the interior of the finite moment domain satisfying $K^{\prime }(\hat{t})=x$. With $K^{\prime \prime }(\hat{t})>0$, the saddlepoint density approximation for the sum isUnder the usual smooth nonlattice conditions, at a fixed interior $x$ the true density has this leading expression times $1+O(n^{-1})$. This is a relative density approximation; the leading expression is not automatically an exactly normalized probability density in every model.

One way to understand the statistical saddlepoint approximation is through exponential tilting. The tilted one-observation density is $f_t(y)=e^{ty-K(t)}f(y)$, with mean $K^{\prime }(t)$ and variance $K^{\prime \prime }(t)$. Undoing this tilt for the sum multiplies its tilted density at $s$ by $e^{nK(t)-ts}$. At $t=\hat{t}$, $s$ is exactly the tilted mean of the sum. A local version of the central limit theorem gives tilted density there approximately $[2\pi n{K}^{\prime \prime }(\hat{t}){]}^{-1/2}$. Multiplying these two factors gives the displayed saddlepoint density approximation. Higher local expansion terms produce the stated relative error order.

For the Inverse Gaussian distribution, differentiation givesThus for every $x>0$ the saddle exists, withAt this saddle,Substitution into the saddlepoint density approximation gives precisely the exact density already found for $S_n$. ThereforeThis exact saddlepoint density of an inverse Gaussian sum is stronger than a relative $O(n^{-1})$ claim: the leading formula is exact and needs no renormalization.

For the final distribution, both tails decay polynomially. Any positive exponential tilt makes the positive tail nonintegrable, and any negative tilt makes the negative tail nonintegrable. Its moment-generating function is therefore infinite away from zero, leaving no open interval on which to form the cumulant-generating function and solve a saddle equation. Here polynomial tails prevent exponential tilting, so the conventional saddlepoint density approximation is unavailable. A finite mean and variance, and even an ordinary central limit theorem, do not supply the missing exponential moments.

For a regular two-parameter model, write $\psi$ for the parameter of interest and $\lambda$ for the nuisance parameter. If $u_{\psi },u_{\lambda }$ are its score functions, the parameters are orthogonal statistical parameters when the cross entry of the expected Fisher information matrix vanishes:This condition may hold at a particular parameter value or throughout the parameter domain. It concerns the information geometry, not exact independence of finite-sample estimators. Under regular asymptotic efficiency, a diagonal information matrix gives zero first-order asymptotic covariance.

An interest-respecting reparametrization leaves $\psi$ unchanged, or replaces it by a one-to-one function of $\psi$ alone, while changing the nuisance coordinate by a smooth invertible transformation which may depend on both old parameters. Thus it preserves the level sets of the inferential target. Without loss of generality, keep $\psi$ itself and write the old nuisance as $\lambda =g(\psi ,\eta )$, where $\eta$ is the new nuisance coordinate and $g_{\eta }\ne 0$.

The chain rule for the log-likelihood gives new score functionsTheir expected cross product isSetting this to zero, with $I_{\lambda \lambda }>0$, gives the orthogonalization equation for two statistical parametersSolve this ordinary differential equation along curves of constant $\eta$, using $\eta$ to label distinct initial conditions. Smooth local solutions with nonzero transverse derivative produce an invertible interest-respecting reparametrization. This informal construction identifies precisely which derivative is held at fixed new nuisance coordinate.

For the given positive-shape generalized Pareto distribution, its one-observation log-likelihood isDefine $L=\log (1+\psi Y/\sigma )$ and $D=e^{-L}$. The density transformation $y=\sigma (e^L-1)/\psi$, $dy/dL=\sigma e^L/\psi$, givesThus $L$ has an exponential distribution of mean $\psi$. Direct differentiation of the original log-likelihood, holding the other original parameter fixed, givesThe following exponential integrals determine their expected products:For example, $\mathbb{E}{e}^{-kL}=(1+k\psi {)}^{-1}$ follows by integrating the exponential density, and differentiating this integral in $k$ gives $\mathbb{E}(L{e}^{-kL})=\psi /(1+k\psi {)}^2$. These formulas also give $\mathbb{E}{u}_{\psi }=\mathbb{E}{u}_{\sigma }=0$.

For clarity, the nuisance information and cross information can be calculated explicitly. Put $a=1+\psi$. ThenUsing $\mathbb{E}[L-a+aD]=0$, the cross product isThe same moments give $I_{\psi \psi }={\psi }^{-4}\mathbb{E}[L-a+aD{]}^2=2/[(1+\psi )(1+2\psi )]$. Therefore the Fisher information matrix for one observation isIn particular, $I_{\psi \sigma }>0$ for every admissible pair, proving the original shape and scale parameters are not orthogonal. No finite-mean assumption on $Y$ was needed: the transformed logarithm $L$ and the bounded variable $D$ have the required integrable score products even when the Pareto mean is infinite.

The orthogonalization equation for two statistical parameters now becomesIntegration along a constant-$\eta$ curve gives $\sigma (1+\psi )=\eta$. Hence a global interest-respecting reparametrization isIt is smoothly invertible on the entire positive parameter domain, with inverse $\sigma =\eta /(1+\psi )$. The Fisher information matrix transforms as $J^{T}IJ$, whereSubstitution givesThis verifies the orthogonal shape and scale for a generalized Pareto distribution directly, as well as by the differential equation. For an iid sample, every information entry is multiplied by the sample size, so the same parameter orthogonality remains valid.

Assume $n\ge 2$, so the residual sum of squares is positive almost surely. Write $\overline{Y}=n^{-1}{\sum }_i{Y}_i$ and $S={\sum }_i(Y_i-\overline{Y}{)}^2$. The log-likelihood isThe decomposition ${\sum }_i(Y_i-\mu {)}^2=S+n(\overline{Y}-\mu {)}^2$ shows that, at every fixed positive variance, its constrained maximum-likelihood estimate of the mean is $\overline{Y}$. Substituting this mean and differentiating with respect to the variance givesThe denominator $n$ is required by maximum likelihood estimation; $S/(n-1)$ is the unbiased variance estimate, which is a different estimator.

Under the null variance ${\sigma }_0^2$, maximize over the same mean $\overline{Y}$. Subtracting the null maximum log-likelihood from the unrestricted maximum gives the likelihood-ratio test statisticBy Cochran's theorem, the residual sum of squares in a normal sample satisfies $U=S/{\sigma }_0^2\sim {\chi }_{n-1}^2$ under the null. Set $V=(U-n)/n$, so $U/n=1+V>0$. ThenThe normal mean is a nuisance parameter, but this null law does not depend on it.

A Bartlett correction removes the first-order discrepancy between the null mean of a regular likelihood-ratio test statistic and the degrees of freedom of its limiting chi-squared distribution. In the convention used here, if the limiting degrees of freedom are $d$ andthen $b$ is the Bartlett correction coefficient, also called the Bartlett correction factor, and the Bartlett-corrected likelihood-ratio statistic is $W_B=W/(1+b/n)$. Its null mean is $d+O(n^{-2})$. The finite-$n$ divisor is $1+b/n$, rather than $b$ itself. There is one restriction on the variance here, so $d=1$.

Expand the logarithm around $V=0$ and use the permitted termwise asymptotic integration:Using the supplied raw moments of the chi-squared distribution and expanding $(U-n{)}^r$ givesFor instance, $\mathbb{E}U=n-1$ and $\mathbb{E}{U}^2=(n-1)(n+1)$ give the first formula after subtracting $2n\mathbb{E}U-n^2$. The same binomial expansion, using the raw third and fourth moments, gives the other two formulas.

We must justify why higher moments cannot change the $n^{-1}$ coefficient. Represent $U={\sum }_{i=1}^{n-1}{Z}_i^2$ for independent standard normal variables, and put $T_i=Z_i^2-1$. Each $T_i$ is centered and has all absolute moments finite. With $H={\sum }_{i=1}^{n-1}{T}_i$, one has $V=(H-1)/n$. The supplied centered-sum bounds say, for each fixed integer $j\ge 2$,Also $\mathbb{E}H=0$ and $\mathbb{E}{H}^0=1$. Therefore, for every fixed $r\ge 2$, expanding $(H-1{)}^r$ givesThese are bounds on the absolute values of signed moments; the odd-moment estimate must not be misread as a bound on $\mathbb{E}|H{|}^j$. In particular, each fixed Taylor term of degree $r\ge 5$ has $n\mathbb{E}{V}^r=O(n^{-2})$ or smaller. Combined with the question's permission to integrate the asymptotic expansion term by term, this controls the higher-order contribution at the required order. It does not require a globally convergent Taylor series for every realized $V$.

Substituting the first three moment formulas now givesThus the Bartlett correction for a normal variance with unknown mean isIts null mean is $1+O(n^{-2})$, and the usual reference law is ${\chi }_1^2$. Keeping the estimated mean's loss of one residual degree of freedom, and using the maximum-likelihood variance divisor $n$, are both essential to the coefficient $11/6$.