Past exam of the mathematics course of the University of Cambridge / 2017 / iii / Paper 205 (Modern Statistical Methods)

A real positive-definite kernel is a symmetric function $k:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ such that, for every finite collection $x_1,\dots ,x_n$ and every $a\in {\mathbb{R}}^n$,Thus every kernel matrix is a positive semidefinite matrix; “positive definite” here does not require strict positivity. A Reproducing kernel Hilbert space is a Hilbert space $\mathcal{H}$ of functions on $\mathcal{X}$ in which every point evaluation is a continuous linear functional. Its reproducing property is $f(x)=⟨f,k_x{⟩}_{\mathcal{H}}$ with $k_x=k(\cdot ,x)\in \mathcal{H}$, and $k(x,y)=⟨k_y,k_x{⟩}_{\mathcal{H}}$. For a sum of positive-definite kernels, the quadratic form of its kernel matrix is a sum of nonnegative quadratic forms. Symmetry also survives summation. Consequently $\boxed{k=\sum _j{k}_j\text{ is a positive-definite kernel}.}$

For the representer theorem, put $V=\mathrm{span}\{k_{x_1},\dots ,k_{x_n}\}$. This is a closed finite-dimensional vector space. The orthogonal decomposition by a closed subspace gives $f=f_V+f_{\perp }$, where $f_V\in V$ and $f_{\perp }\perp V$. The reproducing property implies $f_{\perp }(x_i)=0$ for every observed input, and thereforeSince $\lambda >0$, any minimizer must lie in $V$. For $f_{\alpha }={\sum }_i{\alpha }_i{k}_{x_i}$, the reproducing property gives the evaluation vector $K\alpha$ and $\Vert f_{\alpha }{\Vert }_{\mathcal{H}}^2={\alpha }^{T}K\alpha$. Hence $Q_1(f_{\alpha })=M(\alpha )$. If $\hat{f}$ minimizes $Q_1$, its representation yields a minimizing $\hat{\alpha }$. Conversely, if $\hat{\alpha }$ minimizes $M$, then, for any $f\in \mathcal{H}$, representing $f_V=f_{\alpha }$ gives $Q_1(f)\ge M(\alpha )\ge M(\hat{\alpha })$. ThusThis is an equivalence of minimizers, not an existence or uniqueness assertion: the loss function is arbitrary. A singular kernel matrix also allows nonunique coefficients, since differences in $\ker K$ represent the zero function. Strict convexity of the loss was not used.

Now use the stated minimum-norm description of a sum of reproducing-kernel Hilbert spaces. Let $\hat{f}={\sum }_j{\hat{f}}_j$. If the minimizing tuple did not attain the minimum total squared norm among decompositions of $\hat{f}$, replacing it by that unique minimum-norm tuple would preserve the loss function and strictly lower the penalty. This contradicts optimality. ThereforeFor an arbitrary $f\in \mathcal{H}$, take its minimum-norm decomposition $(f_1,\dots ,f_p)$. Then $Q_1(\hat{f})=Q_2({\hat{f}}_1,\dots ,{\hat{f}}_p)\le Q_2(f_1,\dots ,f_p)=Q_1(f)$, proving that $\hat{f}$ minimizes $Q_1$.

Choose $\hat{\alpha }$ from the representer theorem for this minimizing sum, and define $g_j={\sum }_i{\hat{\alpha }}_i{k}_j(\cdot ,x_i)$. Write $K_j$ for the component kernel matrices. ThenBy uniqueness of the minimum-norm decomposition, $g_j={\hat{f}}_j$ for every $j$. This proves the common representer coefficients for a sum of kernels conclusion:The same coefficient vector works simultaneously. Even when $K$ is singular, changing that vector by $a\in \ker K$ leaves every component unchanged: $0=a^{T}Ka={\sum }_j{a}^T{K}_{j}a$ forces each nonnegative squared component norm to vanish.

Use the normalizationfor the Lasso. A centered sub-Gaussian random variable with parameter $\sigma$ satisfies $\mathbb{E}{e}^{tW}\le e^{{\sigma }^2{t}^2/2}$ for all $t\in \mathbb{R}$; for a noncentered variable apply this definition to $W-\mathbb{E}W$. With fixed $X$, independence of the errors givesThe exponential Markov bound, optimized over $t$ separately for the two signs, gives $\mathrm{Pr}(|X_j^T\epsilon |/n>u)\le 2e^{-n{u}^2/(2{\sigma }^2)}$. Take $u=\lambda /2=\sigma A\sqrt{\log p/n}$ and apply the union bound over columns:The specified positive tuning parameter requires $p\ge 2$; at $p=1$ its formula is zero. For small $A$ the probability lower bound can be negative and hence uninformative. Although centering makes the centered errors dependent, $X^{T}1=0$ means $X^T(\epsilon -\bar{\epsilon }1)=X^T\epsilon$, so the preceding proof uses the original independent errors, not independence after centering.

The Karush-Kuhn-Tucker conditions, using the subdifferential of the L1 norm, areConsequently, for nonempty $B\subseteq \hat{S}$, on $\Omega$,This establishes the required active-set inequality without any rank condition on $X$.

For $s>0$, a sufficient Compatibility condition for the Lasso isIndeed, for $\delta =\hat{\beta }-{\beta }^0$, comparison of the two Lasso objective values and the score bound on $\Omega$ give the Basic inequality for the LassoIt follows that $\delta$ satisfies the Lasso cone condition and $\Vert {\delta }_S{\Vert }_1\le \sqrt{s{P}_E}/\varphi$. Thus $P_E\le 3\lambda \sqrt{s{P}_E}/\varphi$, including the trivial $P_E=0$ case. This proves the slightly stronger $P_E\le 9{\lambda }^{2}s/{\varphi }^2$, and in particular the requested $16{\lambda }^{2}s/{\varphi }^2$. If $s=0$, the same basic inequality forces $\hat{\beta }=0$ on $\Omega$; no nonempty-support compatibility constant is needed.

For the support-size argument, use the stated prediction bound with constant $16$. Set $b=|B|$. By the Cauchy-Schwarz inequality and the definition of the sparse maximum eigenvalue,Combining with the lower bound and squaring gives $\boxed{b\le 64{\kappa }_b^{2}s/{\varphi }^2}$ for every nonempty $B\subseteq \hat{S}$. If $m_{\ast }<\infty$ and $\hat{s}\ge m_{\ast }$, choose $B$ of size $m_{\ast }$; this contradicts the strict inequality defining $m_{\ast }$. If $m_{\ast }=\infty$, $\hat{s}\le p<\infty$ proves the assertion directly. Hence $\boxed{\hat{s}<m_{\ast }}$ in both cases.

When $1<m_{\ast }\le p$, minimality and monotonicity of the sparse maximum eigenvalues giveWhen $m_{\ast }=1$, the first result gives $\hat{s}=0$, so the same final bound holds. Thus the finite-index conclusion is $\boxed{\hat{s}\le 64{\kappa }_{m_{\ast }}^{2}s/{\varphi }^2(m_{\ast }<\infty )}$.

There is a genuine domain omission in the printed last assertion: the definition of ${\kappa }_m$ only makes sense for $1\le m\le p$, so ${\kappa }_{m_{\ast }}$ is undefined when $m_{\ast }=\infty$. This case can occur: take centered orthogonal columns with $X^{T}X/n=I_p$, $n\ge p+1$, $s=p$, and ${\varphi }^2=1$. Then ${\kappa }_m^2=1$ for every available $m$ and none exceeds $64p$, so $m_{\ast }=\infty$. A universally defined replacement, from $B=\hat{S}$ and monotonicity, isAlternatively, explicitly extend the definition by ${\kappa }_m={\kappa }_p$ for $m>p$, including infinity. Under that added convention the printed final expression is meaningful and follows from this replacement bound. Neither convention nor finiteness should be silently assumed.

Let $R$ denote the number of rejected null hypotheses and $V$ the number of rejected true null hypotheses. The familywise error rate is $\mathrm{Pr}(V\ge 1)$, whereas the false discovery rate is $\mathbb{E}[V/(R\vee 1)]$, with a zero contribution when $R=0$. A valid true-null p-value satisfies $\mathrm{Pr}(p_i\le t)\le t$ for $0\le t\le 1$.

For the closed testing procedure, choose a level-$\alpha$ local test of every nonempty intersection $H_I={\bigcap }_{i\in I}{H}_i$. Reject $H_i$ only if every intersection containing it is rejected by its local test. If any true null hypothesis is rejected, the local test of $H_{I_0}$ must reject. Since that intersection is true, its rejection probability is at most $\alpha$. If $I_0$ is empty there can be no false rejection. This proves strong closed-testing control of the familywise error rate, under arbitrary dependence, provided each local test is valid under its whole intersection null.

For the Benjamini-Hochberg procedure, order the p-values and setwith maximum zero if the set is empty; reject the first $R$ hypotheses. To prove its false discovery rate bound, fix a true-null index $i$. Replace $p_i$ by zero and let $R_i$ be the modified rejection count. This is a function of the other p-values and satisfies $R_i\ge 1$.

The Benjamini-Hochberg leave-one-out identity isHere is its deterministic justification. If $i$ was rejected, lowering its value cannot decrease $R$, and leaves all ordered values of ranks greater than $R$ unchanged. No such rank can newly satisfy its threshold, so $R_i=R$. Conversely, if $R_i=r$ and $p_i\le \alpha r/m$, at least $r-1$ other values and $p_i$ are at most this threshold. Therefore the original procedure has $R\ge r$. Monotonicity under decreasing $p_i$ also gives $R\le R_i=r$, so $R=r$ and $i$ is rejected. This establishes the identity; harmless deterministic tie-breaking also covers the artificial zero introduced in the modified data.

The assumed independence makes $p_i$ independent of $R_i$. Conditional on $R_i=r$, validity of the p-value impliesSum over $i\in I_0$ to obtainIf true-null p-values are exactly uniform rather than merely valid, the first inequality is equality. No restriction on dependence among the false-null p-values is needed beyond their joint independence from the true-null values.

For the requested Simes inequality, apply this result to the $m_0$ true-null p-values alone, for $m_0>0$. Every rejection is now false, so $V=R$ and the false discovery rate is the probability of any rejection. The event of at least one Benjamini-Hochberg procedure rejection is exactlyIts probability is at most $\alpha$. This holds for $\alpha \in [0,1]$; with no true nulls the written minimum and division by $m_0$ are undefined and the error-control statement is instead vacuous.

The final step-up rule is the Hochberg procedure. Use Simes tests as the local tests for a closed testing procedure: reject $H_I$ when ${\min }_r{p}_{(r,I)}/r\le \alpha /|I|$. Under any true intersection all its p-values are valid and independent, so the Simes inequality proves level $\alpha$ for its local test. The implication supplied in the question says that whenever the Hochberg procedure rejects $H_{(i)}$, every intersection containing $(i)$ passes its local Simes test. Thus every such rejection is also a closed-testing rejection. ConsequentlyThe true-null independence assumption is essential to this particular proof of local-test validity; the arbitrary-dependence guarantee of closed testing does not create valid local tests automatically.

The global Markov property for a directed acyclic graph requires, for all pairwise disjoint vertex sets $A,B,C$, that D-separation of $A$ and $B$ by $C$ in $G$ implies $Z_A\perp Z_B\mid Z_C$. Two Markov equivalent directed acyclic graphs encode exactly the same such D-separations. The structural characterization is that they have the same skeleton of a directed graph and the same unshielded colliders. A distribution has faithfulness of a directed acyclic graph when its conditional independences are exactly those encoded by D-separation; this includes the global Markov implication as well as its converse.

The population PC algorithm uses an exact conditional independence oracle. Start with the complete undirected graph, and conditioning-set size $\ell =0$. At each size, examine ordered adjacent pairs $(u,v)$. Test subsets $C\subseteq \mathrm{Adj}(u)\setminus \{v\}$ of size $\ell$ until independence $Z_u\perp Z_v\mid Z_C$ is found or all such subsets have been examined. On finding it, delete $u-v$ and record a symmetric separating set $\mathrm{Sep}(u,v)=\mathrm{Sep}(v,u)=C$. Ordered pairs ensure that subsets from either endpoint's neighbours are considered. Increase $\ell$ until no adjacent ordered pair has enough remaining neighbours to supply a set of that size. Rechecking eligibility after deletion, or freezing the neighbours at each level and then deleting, gives the same population skeleton under the hypothesis below.

Next, for each unshielded triple $u-v-w$ with $u,w$ nonadjacent, orient $u\to v\leftarrow w$ precisely when $v\notin \mathrm{Sep}(u,w)$. Propagate directions that are forced without introducing a directed cycle or an additional unshielded collider. In particular, $a\to b-c$ with $a,c$ nonadjacent forces $b\to c$; an undirected edge $a-b$ with a directed path from $a$ to $b$ forces $a\to b$; and $a-b,a-c,a-d,c\to b,d\to b$ with $c,d$ nonadjacent forces $a\to b$. The first avoids a new unshielded collider, the second a directed cycle. For the third, $b\to a$ would force $c\to a$ and $d\to a$ to avoid cycles, creating the prohibited unshielded collider $c\to a\leftarrow d$.

For a fully specified exact completion, list all acyclic orientations of the recovered skeleton which extend the collider arrows and have exactly the recorded unshielded colliders. Direct an edge if and only if all listed graphs orient it the same way; otherwise leave it undirected. This finite enumeration implements the definition of the completed partially directed acyclic graph; orientation propagation is the usual efficient way to obtain the same completion. The equivalence-class proof below uses the exact completion description and does not need an unproved completeness claim about a chosen subset of propagation rules.

To prove correctness, assume faithfulness of a directed acyclic graph to $G^0$. A true edge $u\to v$ cannot be D-separated by any set excluding its endpoints: its length-one path is always active. Thus a true edge is never deleted. Conversely, take nonadjacent $u,v$ and choose their order so that $u$ is earlier than $v$ in a topological ordering of $G^0$. Then $u$ is a nondescendant and nonparent of $v$. Its parent set $\mathrm{Pa}(v)$ gives D-separation of $u$ and $v$. To see this local graphical fact, a path beginning $v\leftarrow q$ is blocked by the conditioned parent $q$, a noncollider. A path beginning with an arrow out of $v$ must encounter a first collider before reaching the nondescendant $u$. That collider is a descendant of $v$, and neither it nor any of its descendants can be a parent of $v$ in an acyclic graph. Hence conditioning on $\mathrm{Pa}(v)$ does not open it.

All parent edges survive. If $u-v$ has not already been deleted, $\mathrm{Pa}(v)$ remains among $v$'s neighbours excluding $u$. It is therefore tested by size $|\mathrm{Pa}(v)|$, and faithfulness forces deletion. The size loop cannot stop prematurely while this set remains eligible. This proves exact recovery of the skeleton of a directed graph.

For an unshielded triple $u-v-w$, if it is a noncollider in $G^0$, its length-two path remains active unless $v$ is conditioned on. Thus every separator of $u,w$ contains $v$. If it is a collider, conditioning on $v$ opens the length-two path, so no separator can contain $v$. Faithfulness turns these graphical statements into the recorded separator test. Therefore the collider phase recovers precisely all unshielded colliders. By the skeleton and collider characterization of Markov equivalence, the acyclic extensions with exactly these colliders are exactly the Markov equivalence of directed acyclic graphs class of $G^0$. In particular that set is nonempty, and the completion returns its CPDAG. This proves population PC correctness under faithfulness.

For the four-variable example, the only removable pairs are $2,4$, with separator $\varnothing$, and $1,3$, with separator $\{2,4\}$. The remaining skeleton has edges $1-2,1-4,2-3,3-4$. The unshielded triples $2-1-4$ and $2-3-4$ both have center outside the separator of $2,4$, so both are unshielded colliders. Every remaining edge is thereby oriented:The other two unshielded triples are noncolliders, consistent with the separator $\{2,4\}$ of $1,3$. There is no undirected edge left, so the equivalence class consists of this unique Directed acyclic graph. This identification is about the given faithful observational distribution; general PC algorithm outputs need not identify a unique causal direction.

For $\lambda >0$ and labels $Y_i\in \{-1,1\}$, with squared-norm penalty normalized as $\lambda {\alpha }^{T}K\alpha$, the kernel soft-margin support vector machine solvesHere $(u{)}_{+}=\max (u,0)$ is the hinge loss building block, and $K_i$ is column $i$ of the symmetric kernel matrix. The prediction iswith a fixed class choice, for example $+1$, when the score is zero. Other positive rescalings of the penalty change numerical coefficients in the optimality equation but not the zero-coefficient conclusion.

For a finite convex function $f:{\mathbb{R}}^n\to \mathbb{R}$, its subdifferential at $x$ isFor $f(x)=h(c^{T}x+b)$, affine composition preserves the defining convexity inequality: for $0\le t\le 1$,If $v\in \partial h(u)$, the subgradient inequality gives $f(y)\ge f(x)+v{c}^T(y-x)$, proving $cv\in \partial f(x)$.

Conversely, let $g\in \partial f(x)$ and first suppose $c\ne 0$. For every $z\in \ker c^T$, the function is constant along $x+tz$. Applying the subgradient inequality for both signs of $t$ shows $g^{T}z=0$. Thus $g$ is in the span of $c$, or $g=cv$ with $v=g^{T}c/\Vert c{\Vert }_2^2$. Substituting $y=x+(a-u)c/\Vert c{\Vert }_2^2$ gives $h(a)\ge h(u)+v(a-u)$ for every real $a$. Hence $v\in \partial h(u)$. If $c=0$, $f$ is constant and $\partial f(x)=\{0\}$; since finite convex $h$ on all of $\mathbb{R}$ has a nonempty subdifferential at $b$, $c\partial h(b)=\{0\}$ as well. This handles the case in which the suggested orthogonal projection would be undefined. ThereforeThis proves the subdifferential under scalar affine composition identity with all its degenerate cases.

Let $m_i=Y_i(\hat{\mu }+K_i^T\hat{\alpha })$ be a fitted support-vector-machine margin. The subdifferential of the hinge loss yields choicesApplying the affine-composition identity and the subdifferential sum rule to the convex objective, its optimum satisfiesThe second equation is the intercept condition. Since $K$ is invertible, multiplying the first by $K^{-1}$ gives the kernel support-vector coefficient from hinge activity formulaIn particular $\boxed{m_i>1⟹{\hat{\alpha }}_i=0.}$ Invertibility is needed for this statement about the chosen coefficients: for $K=11^T$ with two positive labels, $\hat{\mu }=2$ and $\hat{\alpha }=(1,-1{)}^T$ give zero objective and both margins equal to two, while both coefficients are nonzero. A singular kernel matrix allows coefficient changes in the null space without changing the fitted function.

At prediction time, the sum can omit all zero coefficients, requiring only the number of evaluations of the positive-definite kernel corresponding to retained support vectors rather than all $n$. This need not remove the cost of forming or solving with a dense training kernel matrix. Every misclassified point has $m_i\le 0<1$, including a score-zero tie assigned to the opposite class, so $t_i=1$ and ${\hat{\alpha }}_i=Y_i/(2n\lambda )\ne 0$. Thus many classification errors imply many active support vectors and little prediction-time sparsity benefit. Correctly classified points inside the unit margin can also have nonzero coefficients; a point exactly on the margin need not have a nonzero coefficient.

Put $\hat{\Sigma }=X^{T}X/n$. The Debiased Lasso correction isFor the Nodewise Lasso, let $r_j=X_j-X_{-j}{\hat{\gamma }}^{(j)}$. Its Karush-Kuhn-Tucker conditions areso $z_{jk}=\mathrm{sgn}({\hat{\gamma }}_k^{(j)})$ at a nonzero coefficient and $|z_{jk}|\le 1$ at a zero coefficient. Multiplying by ${\hat{\gamma }}^{(j)}$ gives the nodewise-Lasso residual identityWe require nonzero design columns, a necessary qualification missing from the unrestricted printed formulation. Because ${\lambda }_j>0$, ${\hat{\tau }}_j^2=0$ would force both $r_j=0$ and ${\hat{\gamma }}^{(j)}=0$, and hence $X_j=0$. Thus nonzero columns suffice for ${\hat{\tau }}_j^2>0$ even when $p>n$ or the full Gram matrix is singular. A zero column instead gives ${\hat{\tau }}_j^2=0$ and its coefficient is unidentifiable from $Y$; the proposed normalization and confidence interval do not exist there.

For each $j$, define $c_j\in {\mathbb{R}}^p$ by $(c_j{)}_j=1$ and $(c_j{)}_{-j}=-{\hat{\gamma }}^{(j)}$. Set row $j$ of $\hat{\Theta }$ to $c_j^T/{\hat{\tau }}_j^2$. Since $X{c}_j=r_j$,This row construction is the nodewise normalization used by van de Geer and coauthors. The matrix $\hat{\Theta }$ need not be symmetric or be an actual inverse.

Substitute the linear model into the correction. The exact decomposition isIn the fixed-design normal linear model, or with $\epsilon \mid X\sim N_n(0,{\sigma }^{2}I)$ and nodewise estimates and tuning chosen from $X$ alone, a linear transformation of a multivariate normal distribution gives $W\mid X\sim N_p(0,{\sigma }^2\hat{\Omega })$. Marginal normality of $\epsilon$ without this conditional assumption would not suffice for a response-dependent random design. No independence between $W$ and $\Delta$ is asserted. The diagonal cancellation and the Holder inequality give the debiased-Lasso remainder bound

For a coordinate variance, ${\hat{\Theta }}_j{X}^T=r_j^T/{\hat{\tau }}_j^2$, whenceWriting $z_{1-\alpha /2}$ for the corresponding standard normal quantile, an approximate confidence interval isEquivalently its half-width is $z_{1-\alpha /2}\sigma \Vert X_j-X_{-j}{\hat{\gamma }}^{(j)}{\Vert }_2/(n{\hat{\tau }}_j^2)$. We assume $0<\alpha <1$ and $\sigma >0$. For asymptotic coverage, the standardized remainder ${\Delta }_j/(\sigma \sqrt{{\hat{\Omega }}_{jj}})$ must have convergence in probability to zero and the coordinate variance must be nondegenerate. Exact conditional normality of $W$ alone does not give finite-sample exact coverage for $\hat{b}$.

One sufficient random-design regime is the Gaussian identity design for debiased Lasso: take independent rows $X_i\sim N_p(0,I_p)$, independently of $\epsilon$, fixed $\sigma >0$, $p=p_n\to \infty$, $s=s_n\ge 1$, andChoose $\lambda =C\sigma \sqrt{\log p/n}$ and all ${\lambda }_j=D\sqrt{\log p/n}$ for sufficiently large fixed $C,D$. Here is why this regime has the requested remainder bound. Uniformly over columns, their squared norms divided by $n$ tend to one, while all off-diagonal sample cross-correlations have magnitude at most a constant times $\sqrt{\log p/n}$ with probability tending to one. Taking $D$ larger makes the zero vector satisfy all nodewise Karush-Kuhn-Tucker conditions strictly, so ${\hat{\gamma }}^{(j)}=0$ for every $j$ on that event, and ${\hat{\tau }}_j^2=\Vert X_j{\Vert }_2^2/n\ge 1/2$ eventually. This also gives ${\hat{\Omega }}_{jj}=1/{\hat{\tau }}_j^2\to 1$ uniformly.

In this regime, the entrywise entrywise concentration of a Gaussian Gram matrix and Lasso cone condition argument in Gaussian identity design for debiased Lasso gives a Compatibility condition for the Lasso bounded away from zero with probability tending to one. Together with the noise-score event and the Basic inequality for the Lasso, this yields, for a fixed $C_1$,For example, the cone inequality and compatibility bound give $\Vert \hat{\beta }-{\beta }^0{\Vert }_1\le 12\lambda s/{\varphi }^2$ on their joint event. Therefore, with a fixed $A\ge 2D{C}_1$,The stated sparsity condition makes this remainder vanish and supplies the preceding confidence-interval requirement. Both dimensions may grow: $p=n^2$ and $s=\lfloor n^{1/4}\rfloor$ satisfy these assumptions. The concentration and Gaussian-design compatibility results are sufficient standard conditions here; the question does not require their proofs. The displayed rate is a high-probability fixed-constant bound, rather than only a claim of boundedness in probability.