Past exam of the mathematics course of the University of Cambridge / 2023 / iii / Paper 225 (Functional Data Analysis)

The Karhunen–Loève expansion has the following Hilbert-space form. Let $X$ be a square-integrable random element of a Hilbert space $H$, with mean $\mu$ and covariance operator $C$. There are nonnegative eigenvalues ${\lambda }_1\ge {\lambda }_2\ge \cdots$ and orthonormal eigenvectors ${\varphi }_k$ spanning the closure of the range of $C$ such thatin $L^2(\Omega ;H)$. The functional principal component scoressatisfyIf $X$ is a Gaussian random element, the scores are independent normal random variables.

For the proof, $C$ is positive, self-adjoint, and trace-class, withIt is therefore compact, so the spectral theorem for compact Hermitian operators supplies the eigenpairs. Their score covariance isFor the residual $R_m=X-\mu -{\sum }_{k=1}^m{\xi }_k{\varphi }_k$, Parseval identity and the trace formula giveThe centered variable's projection onto $\ker C$ has zero second moment and is therefore zero almost surely, which completes the mean-square expansion. When $C$ has a continuous covariance kernel, Mercer's theorem additionally expands that kernel as $c(s,t)={\sum }_k{\lambda }_k{\varphi }_k(s){\varphi }_k(t)$.

The stated kernel is the Brownian bridge covariance kernel. Its eigenvalue equation isThe right-hand side vanishes at $s=0$ and $s=1$, and differentiating it twice givesThus the normalized eigenfunctions and eigenvalues areThe Karhunen–Loève expansion is consequentlywith convergence in $L^2(\Omega ;L^2[0,1])$, where $\mathbb{E}{Z}_k=0$ and $\mathbb{E}[Z_j{Z}_k]=1_{\{j=k\}}$. Covariance alone does not imply that the $Z_k$ are independent or normal; they are independent standard normal variables when $X$ is Gaussian.

Let $({\lambda }_j,{\varphi }_j)$ be the ordered eigenpairs of the common covariance operator $C_X$, and write $\delta =\mu -{\mu }^{\ast }$. Estimate $C_X$ by the pooled covariance operatorand denote its first $K$ eigenpairs by $({\hat{\lambda }}_j,{\hat{\varphi }}_j)$. The sign ambiguity of each eigenfunction disappears after squaring. Consider the Two-sample FPCA mean statistic

Under $H_0$, the Hilbert-space central limit theorem giveswhere $G$ is a centered Gaussian random element with covariance $C_X$. The assumed eigenvalue gaps give consistency of the estimated eigenvalues and eigenfunctions, so Slutsky's theorem yieldsAn asymptotic level-$\alpha$ test therefore rejects when $T_{n,K}$ exceeds the $(1-\alpha )$-quantile of the chi-squared distribution with $K$ degrees of freedom.

Under a fixed alternative,The test is consequently consistent whenever the mean difference has a nonzero projection onto one of the retained principal components. A difference orthogonal to their span is invisible to this fixed-$K$ test, so $\mu \ne {\mu }^{\ast }$ alone does not guarantee consistency. Increasing $K$ with $n$ can recover such alternatives, but requires additional eigenvalue and approximation control.

Because a strictly monotone map fixing both endpoints is increasing, the change of variables formula $t=h(s)$ givesAssuming $\mathbb{E}\Vert X{\Vert }^2>0$, defineThe Hilbert-space variance identity then gives

Under the regularity needed to interchange expectation and differentiation, $\mathbb{E}{h}^{\prime }(s)=1$ because $\mathbb{E}h(s)=s$. HenceThus $A=1$ exactly when the integrated covariance in the numerator vanishes. In particular, this holds when the amplitude $X$ and the time warp $h$ are independent random variables.

For these increasing warps, the square-root velocity function is $Q(h)(t)=\sqrt{h^{\prime }(t)}$. The chain rule givesThereforewhere the last equality again uses the change of variables formula. Taking square roots proves invariance under common right composition by $\gamma$.

For positive trace-class covariance operators $C_i=L_i{L}_i^{\ast }$, their Procrustes distance between covariance operators iswhere $\mathcal{O}(H)$ is the group of unitary operators on the real Hilbert space $H=L^2[0,1]$. Expanding the square gives the equivalent formula

Convergence in the Hilbert-Schmidt norm impliesProducts of two Hilbert-Schmidt operators are trace-class operators, and the Schatten norm Hölder inequality givesBy trace duality, every unitary $R$ satisfiesTaking the supremum over $R$ shows that the supremum terms in the two Procrustes formulas converge. Combining this with convergence of the squared norms proves

PutWith the $n$ by $K$ design matrix $\Xi =({\xi }_{ik})$, the scalar-on-function linear model becomesThe ordinary least squares estimator based on the truncated model isThus the omitted tail produces the conditional bias $({\Xi }^{\mathsf{T}}\Xi {)}^{-1}{\Xi }^{\mathsf{T}}r$.

LetThe supplied weak law of large numbers and noise limit giveFor a general fixed basis, $g$ need not vanish, so the retained coefficients are asymptotically biased and the estimator is inconsistent even for $c$. Moreover, with fixed $K$ it cannot recover the full slope $\beta$ when $\beta -{\beta }_K\ne 0$.

If the $B_k$ form the eigenbasis of $C_X$, thenThe retained functional principal component scores are therefore uncorrelated with every omitted score, so $g=0$. The least-squares estimates of $c_1,\dots ,c_K$ are consistent despite truncation. Fixed $K$ still estimates only the projection ${\beta }_K$; consistency for the complete function requires the truncation level to increase and the tail error to vanish.

Let ${\Xi }_{ik}=⟨X_i,B_k⟩$ and define the roughness penalty matrixFor $c=(c_1,\dots ,c_K{)}^{\mathsf{T}}$, the loss is the quadratic functionIts normal equations areWhenever ${\Xi }^{\mathsf{T}}\Xi +\rho \Omega$ is positive definite, the unique minimizer isFor the usual choice $\rho \ge 0$, the penalty matrix is positive semidefinite, so full column rank of $\Xi$ suffices. Since the question permits arbitrary real $\rho$, a sufficiently negative value can make the quadratic form indefinite; then the loss is unbounded below and no minimizer exists. In the singular positive-semidefinite case, the Moore-Penrose inverse describes the minimum-norm solution whenever the normal equations are consistent.