Past exam of the mathematics course of the University of Cambridge / 2006 / iii / Paper 42 (Statistical Theory)

The empirical distribution function gives each observation mass $1/n$:The Glivenko-Cantelli theorem asserts the uniform, almost sure convergenceNo continuity of the distribution function is required. Here is a proof that also handles its jumps. Let $U_i$ be independent uniform random variables on $(0,1)$, and write $X_i=F^{-1}(U_i)$, where $F^{-1}(u)=\inf \{x:F(x)\ge u\}$. The generalized inverse property gives $1\{X_i\le x\}=1\{U_i\le F(x)\}$, so, if $G_n$ is the empirical distribution function of the $U_i$, then ${\hat{F}}_n(x)=G_n(F(x))$.

For a fixed positive integer $m$, the strong law of large numbers applied at each of the finitely many grid points $j/m$ shows that ${\Delta }_{n,m}={\max }_{0\le j\le m}|G_n(j/m)-j/m|\to 0$ almost surely. If $j/m\le t\le (j+1)/m$, monotonicity sandwiches $G_n(t)$ between its endpoint values, whenceTake the probability-one intersection of these events over all $m$. On that event the limit superior is at most $1/m$ for every $m$, hence is zero. Therefore ${\sup }_x|G_n(F(x))-F(x)|\to 0$. The coupled sequence has the same joint law as any independent sample from $F$, which proves the asserted almost sure conclusion for the original sample.

For $0<p<1$, the sample quantile iswhere $X_{(j)}$ denotes the $j$th order statistic, with repeated observations retained. Suppose the population median $m=F^{-1}(1/2)$ is unique and $F$ is continuously differentiable in a neighbourhood of $m$, with probability density function $f(m)>0$. ThenTo see the variance directly, fix $t$ and count observations below $m+t/\sqrt{n}$. This count has a binomial distribution with success probability $q_n=1/2+f(m)t/\sqrt{n}+o(n^{-1/2})$. The event that the sample median is below this point is that the count is at least $\lceil n/2\rceil$. The central limit theorem for this binomial triangular array gives a limiting probability $\Phi (2f(m)t)$, exactly the displayed normal distribution. Centering is necessary for stating the limiting distribution; subtracting the deterministic population median has no effect on the asymptotic variance requested in the comparisons.

For the standard normal distribution, $m=0$ and $f(0)=1/\sqrt{2\pi }$, so the sample median has asymptotic variance $\pi /2$ under scaling by $\sqrt{n}$. The central limit theorem gives $\sqrt{n}{\bar{X}}_n\stackrel{d}{\to }N(0,1)$ for the sample mean. Thus the sample median has $\pi /2$ times the asymptotic variance of the sample mean, and its asymptotic relative efficiency against the sample mean is

This is a symmetric beta distribution with parameters $(2,2)$, so its mean and median are both $1/2$. At the median its probability density function is $f(1/2)=3/2$, giving asymptotic variance $1/9$ for the sample median. Direct integration givesConsequently $\sqrt{n}({\bar{X}}_n-1/2)\stackrel{d}{\to }N(0,1/20)$. The sample median has $20/9$ times the asymptotic variance of the sample mean, with asymptotic relative efficiency

Write $u(Y_i;\theta )={\partial }_{\theta }\log f(Y_i;\theta )$ and $U_n(\theta )={\sum }_{i}u(Y_i;\theta )$ for the individual and total score functions. Under the usual regularity assumptions, ${\mathbb{E}}_{{\theta }_0}u=0$ and ${\mathrm{Cov}}_{{\theta }_0}(u)=I({\theta }_0)$, the nonsingular per-observation Fisher information matrix. The central limit theorem and weak law of large numbers giveRearranging the stipulated score expansion, including its remainder, givesSlutsky theorem therefore proves the asymptotic normality of a maximum likelihood estimator:A consistent plug-in Fisher information matrix yields the Wald statistic $W_n=n({\hat{\theta }}_n-{\theta }_0{)}^{T}I({\hat{\theta }}_n)({\hat{\theta }}_n-{\theta }_0)$. Under the null hypothesis, $W_n\stackrel{d}{\to }{\chi }_d^2$, so a test of asymptotic size $\alpha$ rejects when $W_n$ exceeds the $(1-\alpha )$ quantile of that chi-squared distribution. A consistent observed Fisher information matrix gives the same limiting test.

With an $r$-dimensional parameter of interest $\psi$ and nuisance parameter $\lambda$, partition $I$ accordingly. The marginal asymptotic covariance matrix of $\hat{\psi }$ is the $\psi \psi$ block of $I^{-1}$, whose inverse is the efficient informationThus the nuisance-adjusted Wald statistic is $W_{\psi ,n}=n(\hat{\psi }-{\psi }_0{)}^T{I}_{\mathrm{eff}}(\hat{\theta })(\hat{\psi }-{\psi }_0)$, with limiting null distribution ${\chi }_r^2$. Using $I_{\psi \psi }$ alone is generally incorrect when the nuisance parameter is estimated; it is valid when the cross-information vanishes.

For the Inverse Gaussian distribution in this problem, expand the exponent to obtain the log likelihood, up to terms independent of both parameters,Its score functions areFor each positive $\psi$, the first score changes sign from positive to negative at $\lambda =\bar{Y}$, so this is the global maximizing mean parameter of an exponential family. Substituting it in the second score givesThe reciprocal function is strictly convex, so the denominator is positive for a nonconstant sample. The profile log likelihood is strictly concave in $\psi$ and tends to minus infinity at both endpoints, proving this is a maximum. Under continuous model sampling, a sample of size at least two is nonconstant almost surely. For a constant sample, including a one-observation sample, no finite joint maximum-likelihood estimator exists: the likelihood increases without bound as $\psi \to \infty$ at $\lambda =\bar{Y}$.

For one observation, differentiating again and using $\mathbb{E}Y=\lambda$ givesThis parameter orthogonality proves that estimating $\lambda$ does not reduce the efficient information for $\psi$. The Wald statistic has the same algebraic form in the two cases:Using information evaluated at ${\psi }_0$ instead is another first-order equivalent convention.

The printed claim of coincidence needs a first-order interpretation. Known and unknown $\lambda$ need not give numerically identical Wald statistics, because their maximum-likelihood estimators differ. With $\lambda$ known, direct maximization givesFor example, observations $1,2$ and known $\lambda =1$ give ${\hat{\psi }}_K=4$ and ${\hat{\psi }}_U=12$. At ${\psi }_0=1$, the displayed Wald convention gives $9/16$ and $121/144$, respectively. Thus exact equality is false. Under the null hypothesis, however, $\bar{Y}-\lambda =O_p(n^{-1/2})$, hence ${\hat{\psi }}_K-{\hat{\psi }}_U=O_p(n^{-1})$. Their centered, standardized estimates differ by $o_p(1)$, so their Wald statistics differ by $o_p(1)$. This is the valid asymptotic coincidence supplied by inverse Gaussian shape estimation and nuisance orthogonality.

For a nondegenerate distribution function $G$, membership $F\in D(G)$ in its maximum domain of attraction means that there exist $a_n>0,b_n\in \mathbb{R}$ such thatat every continuity point of $G$. A max-stable distribution satisfies: for each positive integer $k$, there exist $c_k>0,d_k\in \mathbb{R}$ such that $G(c_{k}x+d_k{)}^k=G(x)$. Thus a maximum of $k$ independent copies has the same probability distribution after a change of location and scale.

Suppose first that $F\in D(G)$. For fixed $k$, the distribution functions $H_n=F^{kn}$ have the two nondegenerate limitsThe given convergence-of-types result supplies constants $a>0,b$ such that $G(x{)}^k=G(ax+b)$. Equality extends from continuity points to all points by right continuity. Substitute $x=(z-b)/a$ to obtain $G((z-b)/a{)}^k=G(z)$, which proves max-stability with $c_k=1/a,d_k=-b/a$. Conversely, if $G$ is max-stable, take $F=G$ and its max-stability constants for $k=n$. Then $F(c_{n}x+d_n{)}^n=G(x)$ exactly for every $n$, proving $G\in D(G)$. Therefore the maximum domain of attraction is nonempty exactly when $G$ is max-stable.

For the particular logarithmic tail, complete the distribution function by taking $F(x)=0$ for $x\le x_0$. Let $a_n>1$ be the unique solution of $a_n\log a_n=n$, and take $b_n=0$. Equivalently $F(a_n)=1-1/n$, including $n=1$ with $a_1=x_0$. For each fixed $x>0$, eventually $a_{n}x>x_0$, andHere $a_n\to \infty$ follows directly from its defining equation. Since $\log (1-z)=-z+O(z^2)$ as $z\to 0$, it follows that $n\log F(a_{n}x)\to -1/x$. For $x\le 0$, the probability is zero. ConsequentlyThis is the shape-one Fréchet distribution, and the calculation proves the limit at every real $x$ without needing a regular variation theorem.

The function $t\log t$ is strictly increasing on $t>1$. For every integer $n\ge 3$, $n\log n>n=a_n\log a_n$, so $a_n<n$. The defining equation now gives $a_n=n/\log a_n>n/\log n$. Taking logarithms yields $\log a_n>\log n-\log \log n$, whose right side is positive, soMultiplying by $\log n/n$ and squeezing gives $a_n\log n/n\to 1$. Slutsky theorem applied to the normalized sample maximum therefore gives the simpler logarithmic-tail Fréchet normalization

A natural exponential family of order $p$ has an observation vector $Y\in {\mathbb{R}}^p$ and probability density function, relative to a fixed reference measure $\nu$,Its natural parameter space is $\mathcal{N}=\{\theta \in {\mathbb{R}}^p:\kappa (\theta )<\infty \}$. A full natural exponential family allows every $\theta \in \mathcal{N}$. A regular natural exponential family has an open parameter space; fullness and regularity together mean that $\mathcal{N}$ itself is open. A minimal exponential family additionally requires that no nonzero linear combination of the components of $Y$ be constant almost surely. This is the condition that the genuinely needed number of canonical sufficient statistics is $p$.

Hölder's inequality shows that $\mathcal{N}$ is convex: for $0<t<1$, its normalizing integral at $t\theta +(1-t)\eta$ is at most the product of the respective integrals to powers $t$ and $1-t$. Taking logarithms also proves convexity of the cumulant function $\kappa$. In the minimal case its Hessian matrix is positive definite on the interior, as the covariance calculation below establishes. Keeping fullness, regularity and minimality separate matters when discussing existence or uniqueness of a maximum-likelihood estimator.

For $\theta +t\in \mathcal{N}$, integrating the exponential tilt gives the moment-generating functionAt an interior natural parameter this is finite in a neighbourhood of $t=0$, so differentiation under the integral sign is justified by the nearby exponential moments. Differentiating its logarithm once and twice at zero givesAlternatively the score function is $Y-\nabla \kappa (\theta )$; its mean zero and its covariance matrix give the same exponential-family derivative identities. For any vector $v$, $v^T{\nabla }^2\kappa (\theta )v={\mathrm{Var}}_{\theta }(v^{T}Y)\ge 0$. In a minimal exponential family this variance is strictly positive whenever $v\ne 0$, proving strict convexity of the cumulant function. For a general exponential family whose sufficient statistic is $T(X)$, these identities describe the mean and covariance matrix of $T(X)$, rather than necessarily those of $X$ itself.

A general exponential family of order $p$ has the representationThe canonical sufficient statistics $T_j$ are independent of the parameter, the support is fixed, and the cumulant function normalizes the probability density function. Minimality rules out a redundant affine relation among the sufficient statistics. If $\alpha$ ranges over an open subset of ${\mathbb{R}}^q$, $q<p$, and $\eta$ is a smooth embedding with derivative of rank $q$, its image is a $q$-dimensional parameter surface in the $p$-dimensional natural parameter space: this is a $(p,q)$ curved exponential family. In a genuinely curved example the surface is nonaffine.

Take the normal distribution $N(\mu ,{\mu }^2)$ with $\mu >0$. Its log density expands asIt is a $(2,1)$ curved exponential family with sufficient statistics $(y,y^2)$ and natural parameters ${\eta }_1=1/\mu$, ${\eta }_2=-1/(2{\mu }^2)$. The ambient full normal distribution family has ${\eta }_1\in \mathbb{R}$, ${\eta }_2<0$ andThe curve is ${\eta }_2=-{\eta }_1^2/2$ with ${\eta }_1>0$, and its derivative with respect to $\mu$ has rank one. Thus the mean and variance are constrained by $\mathrm{Var}(Y)=(\mathbb{E}Y{)}^2$; the two canonical parameters cannot vary independently. Concavity of an ambient canonical log likelihood need not persist along a curved parameter surface.

For independent observations in a natural exponential family, the log likelihood isIn a minimal exponential family the observed Fisher information matrix is positive definite, so the log likelihood is strictly concave. Any finite solution of the likelihood equation $\nabla \kappa (\hat{\theta })=\bar{Y}$ is therefore the unique global maximum likelihood estimator. Existence is a separate issue: the observed sufficient-statistic mean must belong to the image of the mean parameter map $\nabla \kappa$.

For a full regular minimal natural exponential family this image is the interior of the convex support of an exponential family, $C=\overline{\mathrm{conv}(\mathrm{supp}(h\nu ))}$. The necessity follows from a supporting hyperplane argument. If a finite parameter had its mean on a boundary face, some nonzero $v$ would satisfy $v^{T}Y\le b$ almost surely and $\mathbb{E}(v^{T}Y)=b$. Equality of the expectation forces $v^{T}Y=b$ almost surely. Exponential tilting preserves the base-measure null sets, so this contradicts minimality.

For sufficiency, let $\bar{Y}$ be an interior point of $C$. Choose finitely many points $y_1,\dots ,y_m$ of the support whose convex hull contains a ball about $\bar{Y}$, say of radius $2\epsilon$. Such points exist because $\bar{Y}$ is interior to the closed convex hull; one may approximate the vertices of a small simplex surrounding it by finite convex combinations of support points. Take support neighbourhoods of radius at most $\epsilon$ with positive finite base-measure masses $a_j$. These masses can be taken finite by restricting to bounded neighbourhoods and using finiteness of the normalizing integral at any fixed interior parameter. For every vector $\theta$, at least one $j$ has ${\theta }^T(y_j-\bar{Y})\ge 2\epsilon \Vert \theta \Vert$. Integrating over that neighbourhood yieldsThus the negative log likelihood grows to infinity as $\Vert \theta \Vert \to \infty$. At a finite boundary point of the full open natural parameter space, the normalizing integral is infinite, since otherwise that point would itself belong to the full domain. Fatou's lemma makes $\kappa (\theta )$ diverge on approach to such a boundary. A minimizing sequence for $\kappa (\theta )-{\theta }^T\bar{Y}$ therefore stays in a bounded subset away from the boundary; it has an interior limit where the minimum is attained. Its gradient vanishes, giving the required likelihood equation. Strict convexity gives uniqueness.

Hence, under these full, regular and minimal conventions,This is maximum likelihood existence in a full regular natural exponential family. Regularity alone does not make every sample admissible: in a Bernoulli distribution an all-zero or all-one sample has its sample mean on the boundary of $C=[0,1]$, and the canonical maximum is approached only as $\theta \to -\infty$ or $+\infty$. The usual probability parameter then has a boundary maximum in an enlarged model, but no finite canonical maximizer. A regular restriction of the natural parameter space requires membership in its actual mean-map image, rather than merely in $\mathrm{int}C$. Without minimality, affine redundancies may prevent uniqueness of the canonical parameter even when the fitted probability distribution is unique.

Write $R(v)={\int }_{\mathbb{R}}v(u{)}^{2}du$. The kernel density estimator with bandwidth $h>0$ isSufficient conditions are that $K$ be a nonnegative symmetric probability density function with $R(K)<\infty$ and finite nonzero second moment ${\mu }_2(K)=\int u^{2}K(u)du$, and that $h\to 0$, $nh\to \infty$. Symmetry and the second-moment assumption give $\int uK(u)du=0$. The expectation is a convolution:Taylor's theorem with integral remainder givesThe zeroth-order term integrates to $f(x)$, and the first-order term vanishes. The second derivative is bounded and continuous, so dominated convergence, with dominating function proportional to $u^{2}K(u)$, yieldsCombining this with the supplied leading variance gives the asymptotic mean squared errorWhen $f(x)>0$ and $f^{\prime \prime }(x)\ne 0$, its derivative is $-R(K)f(x)/(n{h}^2)+{\mu }_2(K{)}^2{f}^{\prime \prime }(x{)}^2{h}^3$. It changes sign exactly once, giving the pointwise optimal kernel bandwidthPositivity of $f(x)$ is needed for this positive interior optimum. The printed condition $f^{\prime \prime }(x)\ne 0$ alone does not ensure it. For instance, $f(x)=x^2\varphi (x)$ is a normalized bounded probability density function with bounded continuous square-integrable second derivative, but $f(0)=0$ and $f^{\prime \prime }(0)=2\varphi (0)>0$. At this point the displayed leading variance vanishes and the two-term AMSE has no positive interior minimizer; finer variance terms are needed to determine an appropriate bandwidth. The formal formula there gives zero, which does not satisfy the bandwidth assumptions.

For integrated error, the same expansion holds in $L^2$: translation continuity of $f^{\prime \prime }\in L^2$ applied to the integral remainder givesIndeed Minkowski's integral inequality bounds the norm of the difference, divided by $h^2$, by the integral of $u^{2}K(u){\int }_0^1(1-s)\Vert f^{\prime \prime }(\cdot -shu)-f^{\prime \prime }{\Vert }_{2}dsdu$; each translated difference tends to zero and is bounded by $2\Vert f^{\prime \prime }{\Vert }_2$. This proves the claimed integrated bias expansion without assuming a uniform pointwise remainder over the whole real line.

If $K_h(x)=h^{-1}K(x/h)$, direct integration of the variance givesThe last term is $O(n^{-1})$ because $f$ is bounded and integrable, hence square integrable, and convolution with a probability density function does not increase its $L^2$ norm. It is negligible compared with $1/(nh)$. Thus the asymptotic mean integrated squared error and its minimizer areHere $R(f^{\prime \prime })>0$: otherwise the continuous second derivative vanishes everywhere, making $f$ affine, impossible for a probability density function on the whole real line.

Now take the standard normal density $f=\varphi$. Differentiation gives ${\varphi }^{\prime \prime }(x)=(x^2-1)\varphi (x)$. Gaussian integration givesThe kernel constants cancel from the bandwidth ratio. At $x\ne \pm 1$,Put $t=x^2$. The logarithmic derivative of $e^{t/2}/(t-1{)}^2$ is $1/2-2/(t-1)$. On $0\le t<1$ the function increases, giving its minimum at $t=0$, with value one. On $t>1$ it decreases up to $t=5$ and then increases, giving its minimum $e^{5/2}/16<1$. Thus the global minimizers are $x=\pm \sqrt{5}$, and the normal-density local-to-global bandwidth ratio isThe ratio diverges at the two inflection points $\pm 1$, where the second-order pointwise bias approximation vanishes and needs a higher-order replacement.

Let $c=g^{\prime \prime }(\tilde{y})>0$. Near the interior minimum, Taylor expansion givesOn the scale $y=\tilde{y}+z/\sqrt{nc}$, the leading exponent is $ng(\tilde{y})+z^2/2$ and $dy=dz/\sqrt{nc}$. If the integral is localized at this minimum, the rescaled integration endpoints tend to infinity in both directions, so the Gaussian integral gives Laplace's method:Formally expanding the cubic term gives an order-$n^{-1/2}$ correction odd in $z$, whose Gaussian integral is zero. Quartic terms and the square of the cubic term produce the order-$n^{-1}$ correction. Sufficient smoothness near the minimum and control of the exterior integral justify this formal calculation.

The printed local assumptions alone do not guarantee localization. A useful localization condition for Laplace's method is that $g(y)\ge g(\tilde{y})+{\delta }_{\epsilon }$ outside each sufficiently small fixed neighbourhood of $\tilde{y}$, for some ${\delta }_{\epsilon }>0$, and that ${\int }_a^b{e}^{-n_{\ast }g(y)}dy<\infty$ for some $n_{\ast }>0$. For $n>n_{\ast }$, the exterior contribution is at mostwhich is exponentially smaller than the displayed local term. To see why something of this kind is needed, take $g(y)=y^2(1-y{)}^4$ on $(-1,1)$. Its unique minimum is at zero and $g^{\prime \prime }(0)=2$, yet on $1-n^{-1/4}<y<1$ we have $ng(y)\le 1$. This boundary interval alone contributes at least $e^{-1}{n}^{-1/4}$, dominating the proposed $n^{-1/2}$ local approximation. The Laplace formula therefore has its usual implicit tail assumption, rather than following from uniqueness of the interior minimum alone.

For the gamma function, set $y=nu$ to obtainHere $g(u)=u-\log u$ has its unique minimum at $u=1$, with $g(1)=1$ and $g^{\prime \prime }(1)=1$. It tends to infinity at both endpoints, and its integral is finite for positive exponents, so localization holds. Laplace's method yields Stirling's formula:

For a prior density $p$ and independent observations, Bayes' theorem writes the desired posterior expectation as a ratio of integrals:Suppose the log-posterior kernel has a unique concentrating interior mode $\tilde{\theta }$, with curvature $J_n=-H_n^{\prime \prime }(\tilde{\theta })>0$ of order $n$, and the smoothness and tail conditions hold for both integrals. Expanding $H_n$ quadratically gives a common Gaussian factor $e^{H_n(\tilde{\theta })}\sqrt{2\pi /J_n}$. Replacing the smooth amplitude $g(\theta )$ by $g(\tilde{\theta })$ givesThis approximation does not require $g$ to be positive: it treats $g$ as an amplitude, without taking its logarithm. For more accuracy, expand the amplitude and the cubic term in the log-posterior kernel. With $s=\theta -\tilde{\theta }$, the Gaussian moments are $\mathbb{E}{s}^2=J_n^{-1}$ and $\mathbb{E}{s}^4=3J_n^{-2}$. Terms multiplying $g(\tilde{\theta })$ cancel between numerator and denominator; the two remaining first corrections areunder the stronger smoothness, derivative-scaling and integrability assumptions needed for this expansion. The first term accounts for the curvature of the amplitude and the second for posterior skewness. This is posterior expectation by Laplace approximation. If there are several modes, boundary modes, or nonconcentrating tails, the appropriate separate contributions must be included; a single interior Gaussian approximation cannot then be assumed.