Past exam of the mathematics course of the University of Cambridge / 2008 / iii / Paper 42 (Statistical Theory)

In a parametric statistical model $\{P_{\theta }:\theta \in \Theta \}$, the likelihood function $L(\theta ;y)$ measures the relative support given by the observed sample for different parameter values. Maximum likelihood estimation supplies a point estimate; the sampling distributions of suitable statistics calibrate significance tests and confidence intervals. A sufficient statistic retains all parameter-dependent information in the likelihood function, so one first seeks a reduction to such a statistic rather than discarding information arbitrarily.

The characteristic conditioning principle in Fisherian conditional inference is to calibrate inference conditional on the observed value of an ancillary statistic. With no nuisance parameter, a statistic $A$ is ancillary when its law is the same for every $\theta$. Its observed value can nevertheless describe the experimental configuration or precision. Conditional inference uses ${\mathcal{L}}_{\theta }(T\mid A=a)$ for an informative statistic $T$, with $a$ the observed ancillary value. Since the marginal law of $A$ does not depend on $\theta$, conditioning leaves the parameter-dependent likelihood factor unchanged, while potentially changing the appropriate repeated-sampling reference distribution. Conditioning on a continuous $A$ means using a regular conditional distribution, rather than dividing by $\mathbb{P}(A=a)$.

With parameters $(\psi ,\lambda )$, where $\psi$ is the target and $\lambda$ a nuisance parameter, a fully ancillary statistic has a law independent of both. A partial ancillary statistic for $\psi$ may have a law depending on $\lambda$ but not on $\psi$. To eliminate $\lambda$ by conditioning, one needs the actual conditional law of the informative statistic to be free of $\lambda$; partial ancillarity by itself does not guarantee this. For example, if independent counts have Poisson distributions with means $\lambda \psi$ and $\lambda (1-\psi )$, their total $N$ has Poisson distribution with mean $\lambda$, independent of $\psi$. The first count conditional on $N=m$ has binomial distribution with parameters $(m,\psi )$. Conditioning removes the nuisance intensity $\lambda$ exactly. More generally, nuisance-free pivotal quantities or conditional likelihoods can be used where available; there is no universal conditioning device that removes every nuisance parameter.

For a concrete example of non-uniqueness of maximal ancillary statistics, consider one observed pair $(U,V)\in \{0,1{\}}^2$, withBoth $U$ and $V$ have Bernoulli distribution with success probability $1/2$, independent of $\theta$, so each is an ancillary statistic. Their generated partitions are different and incomparable. Moreover, each is maximal: refining either two-point cell singles out a point whose probability is $\theta /2$ or $(1-\theta )/2$, which is parameter-dependent. Their joint statistic is not ancillary, since $P_{\theta }(U=V)=\theta$. Thus there is no common finer ancillary partition that reconciles the two choices. The conditional laws differ as labelled experiments: $P_{\theta }(V=U\mid U)=\theta$ and $P_{\theta }(U=V\mid V)=\theta$, but they condition on different observed information. This illustrates why “condition on a maximal ancillary” need not specify a unique procedure.

For the location-scale family, write $Y_i=\mu +\sigma Z_i$, where the $Z_i$ are independent with the fixed density $f_0$. Let $s_Z^2=n^{-1}{\sum }_i(Z_i-\bar{Z}{)}^2$. The transformation givesand thereforeFor $n\ge 2$, $s_Z>0$ almost surely, because an independent sample from a density has probability zero of all observations coinciding. The right side depends only on a sample from the fixed base law, so the entire vector $A$ has a law independent of $(\mu ,\sigma )$: $\boxed{A\text{ is ancillary}.}$ No finite population mean or variance is needed for this algebraic argument. With one observation the displayed standardization is undefined, so at least two observations are implicitly required.

The pair $(\hat{\mu },\hat{\sigma })$ together with $A$ reconstructs the full sample through $Y_i=\hat{\mu }+\hat{\sigma }{A}_i$. Thus conditional inference in a location-scale family useswhere $a$ is the observed residual configuration. Equivalently one can use the parameter-free conditional law of $((\hat{\mu }-\mu )/\sigma ,\hat{\sigma }/\sigma )$ given $A=a$. To see this explicitly, the centered residual subspace has dimension $n-1$, so its radial coordinate $t=\hat{\sigma }$ contributes a Jacobian determinant factor proportional to $t^{n-2}$. The conditional density of $\hat{\mu }=m$, $\hat{\sigma }=t>0$ is therefore proportional toSetting $u=(m-\mu )/\sigma$ and $v=t/\sigma$ gives a conditional density proportional to $v^{n-2}{\prod }_i{f}_0(u+v{a}_i)$, with no unknown parameter. The normalizing factor can depend on $a$, but not on $\mu$ or $\sigma$. These statements concern almost every attainable ancillary value.

Let $M_n={\max }_{1\le j\le n}{W}_j$ for independent and identically distributed random variables with distribution function $F$. Then $P(M_n\le u)=F(u{)}^n$. Membership in the maximum domain of attraction of a nondegenerate distribution function $G$ means that some deterministic $a_n>0$ and $b_n\in \mathbb{R}$ satisfyor equivalently $F(a_{n}x+b_n{)}^n\to G(x)$ at every continuity point of $G$.

Under the stated convergence, put $Z_n=(M_n-b_n)/a_n$, $r_n={\alpha }_n/a_n\to a>0$, and $s_n=({\beta }_n-b_n)/a_n\to b$. The Slutsky theorem says that combining a sequence with convergence in distribution with sequences converging in probability to constants preserves the corresponding sum, product and ratio limits, provided a limiting denominator is nonzero. Applying it here givesThe limiting distribution function is $P((Z-b)/a\le x)=G(ax+b)$, continuous for every real $x$ because $G$ is continuous and $a>0$. Consequently,

For large $t$, write the survival function asA positive function $\ell$ is slowly varying if $\ell (tu)/\ell (t)\to 1$ for every fixed $u>0$. Heresince $\log (tu)=\log t+\log u$ and $\log \log (tu)-\log \log t\to 0$. Thus $\overline{F}$ has regular variation of index $-2$.

The relevant extreme-value criterion is that an infinite right endpoint and a survival function regularly varying with index $-r$, $r>0$, give attraction to the standard Fréchet distribution of shape $r$. Its distribution function is $G_r(x)=e^{-x^{-r}}$ for $x>0$ and zero for $x\le 0$. Here is a direct verification, so no additional theorem is needed to establish the criterion in this example. Choose $a_n\to \infty$ satisfying $n\overline{F}(a_n)\to 1$ and set $b_n=0$. For $x>0$,If $u_n\to 0$ and $n{u}_n\to c$, then $n\log (1-u_n)=-n{u}_n+O(n{u}_n^2)\to -c$. Applying this with $u_n=\overline{F}(a_{n}x)$ gives $F(a_{n}x{)}^n\to e^{-x^{-2}}$. For $x\le 0$, the distribution function is zero because the observations are supported on $[10,\infty )$. ThereforeFor example, elementary asymptotic normalizers arefor sufficiently large $n$, with arbitrary positive definitions at smaller indices. Indeed $\log a_n\sim \frac{1}{2}\log n$ and $\log \log a_n\sim \log \log n$, so $a_n^2\log a_n\log \log a_n\sim Cn$. The precise valid choice of $C$ at the lower endpoint does not change the limiting shape.

The Gumbel distribution for maxima has distribution function $G(x)=\exp (-e^{-x})$ on the whole real line. For $n\ge 2$ setBoth the centering and scale are elementary, and the scale is positive. For any fixed real $x$, the normalized threshold ${\beta }_n+{\alpha }_{n}x$ is positive for all sufficiently large $n$. Writing $L=\log n$, a Taylor expansion givesConsequently,The logarithm calculation in part (i) now givesThus $F$ belongs to the maximum domain of attraction of the standard Gumbel distribution. The scale also has the Gumbel auxiliary function interpretation: the reciprocal hazard function is $2\sqrt{t}$, whose derivative $1/\sqrt{t}$ tends to zero at the infinite right endpoint. This agrees with the Von Mises conditions for extreme values, and at $t={\beta }_n$ gives exactly ${\alpha }_n=2\log n$.

Iterating the recurrence yieldsThus every finite collection of the $X_j$ is a linear transformation of independent standard normal random variables, and has a multivariate Gaussian distribution. The expected values are zero. Independence of the $Y_k$ givesFor $i<j$, the recurrence writes $X_j={\rho }^{j-i}{X}_i$ plus a sum of future innovations independent of $X_i$. Hence $\mathrm{Cov}(X_i,X_j)={\rho }^{j-i}$. The required vector law isThe covariance matrix is positive definite: the lower-triangular transformation from $(Y_1,\dots ,Y_n)$ has nonzero diagonal entries $1,\sqrt{1-{\rho }^2},\dots ,\sqrt{1-{\rho }^2}$. This also identifies a stationary autoregressive process of order one, with marginal density $f=\varphi$.

With the standard normal density as the kernel for density estimation, the two characteristic functions are $\psi (t)=e^{-t^2/2}$ and ${\psi }_K(ht)=e^{-h^2{t}^2/2}$. The Gaussian random variable $X_1-X_{j+1}$ has mean zero and variance $2(1-{\rho }^j)$, soThis expression is real. Substitution in the given integrated covariance expression, followed by the Gaussian integral ${\int }_{\mathbb{R}}{e}^{-a{t}^2}dt=\sqrt{\pi /a}$ for $a>0$, gives

To bound the total Gaussian autoregressive kernel variance correction, apply the mean value theorem to $u\mapsto u^{-1/2}$. For $j\ge 1$, the whole interval between $1+h^2-{\rho }^j$ and $1+h^2$ is bounded below by $1-\rho >0$. ThereforeSumming the geometric series gives the uniform boundIt follows from the supplied variance identity thatThe constant depends on the fixed $\rho$, but not on $h$ or $n$. In particular this applies to the requested shrinking-bandwidth regime; the covariance bound itself does not need $nh\to \infty$.

All observations have the same marginal density $f$, so dependence changes the variance but not the bias of a kernel density estimator: $\mathbb{E}{\hat{f}}_h=K_h\ast f$. Write $R(q)={\int }_{\mathbb{R}}{q}^2$ and ${\mu }_2(K)=\int u^{2}K(u)du$. For a smooth density with $f^{\prime \prime }\in L^2$, the integrated variance of a kernel density estimator in the independent case is $R(K)/(nh)+O(n^{-1})$. A second-order Taylor expansion of the convolution gives $K_h\ast f-f=(h^2{\mu }_2(K)/2)f^{\prime \prime }+o(h^2)$ in the L2 norm. Thus the bias-variance decomposition of mean squared error givesThe first two terms are the asymptotic mean integrated squared error. The additional dependence term is smaller than $n^{-4/5}$, so the leading optimal bandwidth and optimal rate are the same as for independent data:Here both $K$ and $f$ are the standard normal density. Since $f^{\prime \prime }(x)=(x^2-1)\varphi (x)$, direct Gaussian integrals giveConsequently the explicit leading bandwidth isAt this bandwidth the leading integrated variance is four times the leading integrated squared bias, which is also a check on the minimization constant.

Let the summands be independent and identically distributed random variables with mean $\mu$, variance ${\sigma }^2>0$, and standardized third cumulant ${\gamma }_1=\mathbb{E}[(Y_1-\mu {)}^3]/{\sigma }^3$. DefineAn Edgeworth expansion refines the central limit theorem using the higher cumulants of one summand. Some smoothness and moment assumptions are essential: finite variance alone does not supply a density expansion. A convenient sufficient set for the expansions here is a smooth density whose derivatives decay faster than every inverse power, together with $\mathbb{E}{e}^{\delta |Y_1|}<\infty$ for some $\delta >0$. These assumptions give all required moments, Fourier smoothing and the nonlattice characteristic-function condition. Less restrictive sufficient conditions are possible, but this set covers the logistic model below.

Write $\varphi$ and $\Phi$ for the standard normal density and standard normal distribution function. The first-order density Edgeworth expansion isThe remainder is uniform on every fixed compact interval under the stated assumptions. To see the origin of the correction, for $Z=(Y_1-\mu )/\sigma$ the cumulant expansion of its characteristic function givesExponentiating and using Fourier inversion yields the displayed correction, since the inverse transform of $(it{)}^3{e}^{-t^2/2}$ is $-{\varphi }^{(3)}(x)=(x^3-3x)\varphi (x)$. The Probabilists' Hermite polynomial convention is being used: $H_2(x)=x^2-1$ and $H_3(x)=x^3-3x$.

The corresponding distribution function expansion isuniformly on fixed compact intervals. The sign follows from $\frac{d}{dx}[H_2(x)\varphi (x)]=-H_3(x)\varphi (x)$. These are expansions of both the density and distribution function; a uniform density error alone would not justify integrating an error bound over the entire real line without the accompanying tail control.

For a fixed $\alpha \in (0,1)$, write $z=z_{\alpha }={\Phi }^{-1}(\alpha )$. At leading order the proposed quantile expansion gives $\Phi (p_0(z))=\Phi (z)$, and strict monotonicity gives $p_0(z)=z$. Substitute $y_{\alpha }=z+p_1(z)n^{-1/2}+O(n^{-1})$ into the distribution function expansion. A Taylor expansion at $z$ givesSince $\varphi (z)>0$, the coefficient of $n^{-1/2}$ must vanish. Thus the first Cornish-Fisher expansion coefficients are

For the logistic distribution, set $Z_i=Y_i-\theta$. Its density $q(z)=e^z/(1+e^z{)}^2$ satisfies $q(-z)=q(z)$ and has exponentially decreasing tails. Symmetry and integrability give $\mathbb{E}{Z}_i=0$, and the supplied integral givesTherefore $\mathbb{E}{Y}_i=\theta$, $\mathrm{Var}(Y_i)={\pi }^2/3$, and the central limit theorem givesWriting $z_{-}=z_{\alpha /2}$ and $z_{+}=z_{1-\alpha /2}$, the condition $z_{-}\le T_n\le z_{+}$ is equivalent toConsequently the confidence intervalhas limiting coverage $\Phi (z_{+})-\Phi (z_{-})=1-\alpha$. Whether the finite endpoints are included makes no probability difference because the sample mean has a density.

The centered logistic distribution is symmetric, so ${\gamma }_1=0$. Its smooth density and exponentially decreasing derivatives satisfy the conditions stated above; in particular all moments exist and Fourier smoothing is available. The first correction in the Edgeworth expansion vanishes, leaving $F_{T_n}(z)=\Phi (z)+O(n^{-1})$ at each of the two fixed quantiles. Subtracting gives the stronger conclusionThe standardized law is independent of $\theta$, so this coverage-error conclusion holds with the same bound for every location parameter.

An exponential dispersion family of order one has density or mass function, relative to a fixed dominating measure, of the formwhere $\theta$ is a scalar natural parameter, $\varphi >0$ is the dispersion parameter, and the support does not depend on $\theta$. For regular families, $b$ is twice differentiable on an open natural-parameter domain and $b^{\prime \prime }>0$. Known observation weights $w$ replace $\varphi$ by $\varphi /w$. “Order one” refers to the scalar canonical statistic $y$, rather than an arbitrary vector of canonical statistics.

Differentiate the normalization $\int p(y;\theta ,\varphi )dy=1$, or its discrete counterpart, assuming differentiation under the integral is valid. The first derivative gives $\mathbb{E}(Y-b^{\prime }(\theta ))=0$. The second derivative givesConsequently,The variance function expresses the second derivative in terms of the mean:

A generalized linear model specifies independent responses from an exponential dispersion family, a linear predictor ${\eta }_i=x_i^T\beta$, and a differentiable monotone link function relating it to the response mean by $g({\mu }_i)={\eta }_i$. The family, variance function, covariate design, link function and dispersion specification together define the model. The canonical link function identifies the linear predictor with the natural parameter:One advantage is that the likelihood equations become particularly simple. With known weights $w_i$ and common dispersion $\varphi$, the score function and negative Hessian matrix for $\beta$ areThe latter does not explicitly depend on the responses, so it equals the expected Fisher information matrix at the same parameter. With full-rank design and positive variances the likelihood is strictly concave in $\beta$, simplifying optimization. These statements concern the coefficient block at a fixed dispersion value.

Three explicit examples, taking unit weights, are as follows.

For a normal distribution with mean $\mu$ and variance $\varphi$,Thus $\theta =\mu$, $b(\theta )={\theta }^2/2$, $c(y,\varphi )=-y^2/(2\varphi )-\frac{1}{2}\log (2\pi \varphi )$, and $V(\mu )=1$. Its canonical link function is $\boxed{g(\mu )=\mu }$, the identity link.

For a Poisson distribution with mean $\mu >0$ and support $\{0,1,\dots \}$,Here $\varphi =1$, $\theta =\log \mu$, $b(\theta )=e^{\theta }$, and $c(y,1)=-\log (y!)$. Therefore $b^{\prime }(\theta )=b^{\prime \prime }(\theta )=e^{\theta }$, $V(\mu )=\mu$, and the Poisson canonical link is $\boxed{g(\mu )=\log \mu }$.

For a Bernoulli distribution with success probability $\mu \in (0,1)$,Taking $\varphi =1$, $\theta =\log (\mu /(1-\mu ))$, $b(\theta )=\log (1+e^{\theta })$, and $c(y,1)=0$ gives the required form. Its mean and variance function are $b^{\prime }(\theta )=e^{\theta }/(1+e^{\theta })=\mu$ and $V(\mu )=\mu (1-\mu )$. Its canonical link function is $\boxed{g(\mu )=\log (\mu /(1-\mu ))}$, the logit link. The fixed dispersion values in the Poisson and Bernoulli cases are part of those models; an arbitrary continuous dispersion is not being asserted for them.

To test $H_0:{\beta }_j=0$, first fit the unrestricted model and then the restricted model with that component fixed at zero, estimating all nuisance parameters, including dispersion when applicable, in each fit. If $\hat{\vartheta }$ and $\tilde{\vartheta }$ denote their full fitted parameter vectors, the likelihood-ratio test usesFor identifiable regular models, an interior null parameter and a fixed-dimensional full-rank design with increasing information, the Wilks theorem gives $D\stackrel{d}{\to }{\chi }_1^2$ under $H_0$. A level-$\alpha$ test rejects for $D$ greater than the $(1-\alpha )$-quantile of the chi-squared distribution with one degree of freedom.

Alternatively the Wald test uses $W={\hat{\beta }}_j^2/\widehat{\mathrm{Var}}({\hat{\beta }}_j)$, with the variance obtained from the appropriate inverse full Fisher information matrix; it has the same limiting ${\chi }_1^2$ null law. The score test fits only the restricted model. If $\lambda$ collects its nuisance parameters, its efficient information is $I_{jj\cdot \lambda }=I_{jj}-I_{j\lambda }{I}_{\lambda \lambda }^{-1}{I}_{\lambda j}$, and the statistic is $U_j(\tilde{\vartheta }{)}^2/I_{jj\cdot \lambda }(\tilde{\vartheta })$, again asymptotically ${\chi }_1^2$. Accounting for the nuisance block is necessary rather than using the unadjusted diagonal information.

For ordinary homoscedastic fixed-design nonparametric regression, take observations $Y_i=g(x_i)+{ϵ}_i$ at known distinct sites, with independent mean-zero errors of common variance. A degree-$p$ local polynomial estimator at a target $u$ solves the finite-dimensional weighted least squares problemHere $K$ is a nonnegative regression kernel, $h>0$ is the smoothing bandwidth, and sufficient local design rank ensures uniqueness. The estimate of the regression function is $\hat{g}(u)={\hat{\beta }}_0(u)$. Each target has its own locally fitted polynomial; this is not one global polynomial fit. The bandwidth controls the localization and the bias-variance tradeoff.

A natural cubic smoothing spline instead solves the global penalized problemThe Sobolev space here consists of functions with square-integrable weak derivatives through order two. The minimizer is a natural cubic spline with knots at the observation sites and linear tails outside the extreme sites. Equivalently one may perform the minimization on that finite-dimensional spline space: the minimum roughness property of the natural cubic spline interpolant shows that replacing a candidate by the natural spline with the same site values preserves the data loss and decreases the second-derivative penalty. The smoothing parameter $\lambda$ controls curvature; it differs from the local bandwidth $h$.

For the interval-average observation problem, the printed upper limit in the loss is an indexing error: there are only $n-1$ responses and intervals, so a term with $i=n$ would involve the undefined $Y_n$ and $x_{n+1}$. The coherent loss uses the upper limit $n-1$. Put ${\Delta }_i=x_{i+1}-x_i$ and define the linear observation functionalsThe intended criterion on $C^1[a,b]$ is thereforeUnlike the ordinary cubic smoothing spline, this uses interval averages as data and a first-derivative penalty; the resulting spline degree will be two.

Fix any candidate $v\in C^1[a,b]$. By the supplied interpolation property, there is a unique quadratic spline $q\in C^1[a,b]$ with knots $x_1,\dots ,x_n$, constant on both exterior intervals, such that $L_{i}q=L_{i}v$ for every $i$. Put $d=v-q$. ThenOn each interior knot interval $q^{\prime \prime }$ is a constant $c_i$, and on the two exterior intervals $q^{\prime }=0$. Integrate $q^{\prime }{d}^{\prime }$ by parts separately on the intervals. At every knot the boundary contributions cancel because both $q^{\prime }$ and $d$ are continuous; the contributions at $a$ and $b$ vanish because $q^{\prime }$ is zero there. ThusThis orthogonality is the key to the quadratic smoothing spline for interval averages. Expanding the roughness penalty givesThe data-fit terms are equal because the interval averages are equal. Consequently,Equality can occur only if $d^{\prime }=0$ everywhere: the derivative is continuous and its squared integral is zero. Then $d$ is constant, and any of its zero interval averages forces that constant to be zero. Therefore every minimizer must equal its matching quadratic spline $q$.

We conclude that the minimizer is a continuously differentiable quadratic spline with knots at the specified sites and constant tails on both exterior intervals. Existence and uniqueness also follow from the supplied interpolation property. The interval-average map identifies the finite-dimensional space of these quadratic splines with ${\mathbb{R}}^{n-1}$, and the criterion becomes a strictly convex quadratic function of that average vector: its data-fit term is $\Vert Y-y{\Vert }^2$, while its penalty is nonnegative and quadratic. It has a unique minimizer, and the strict equality argument above proves uniqueness in the full function class.