Past exam of the mathematics course of the University of Cambridge / 2022 / iii / Paper 205 (Modern Statistical Methods)

With the normalization used in the question, kernel ridge regression minimizesThe representer theorem and the normal equations give the fitted-value vector

Let ${\mathcal{H}}_X=\mathrm{span}\{k(x_i,\cdot ):1\le i\le n\}$ and let $\bar{f}$ be the orthogonal projection of $f^0$ onto this space. The reproducing property makes $f^0(x_i)=\bar{f}(x_i)$ for every $i$. Writing $\bar{f}={\sum }_j{\alpha }_{j}k(x_j,\cdot )$ gives

Conditionally on $X$, the covariance matrix of the fitted vector isSince the eigenvalues of $K$ are $n{\hat{\mu }}_i$, the average conditional variance isFor $t\ge 0$,Taking $t=n{\hat{\mu }}_i/\lambda$ and adding the supplied squared-bias bound gives

Because $\hat{\lambda }$ minimizes this conditional upper bound, its expected value is at most the expected bound at any deterministic $\lambda =n\gamma$. Using the assumed eigenvalue comparison,

The false discovery rate iswhere $R$ is the total number of rejections and $V$ is the number of rejected true nulls. The Benjamini-Hochberg procedure orders the p-values and takeswith $R=0$ when the set is empty, then rejects the $R$ smallest p-values.

For $i\in I_0$, apply the modified procedure to $p_{-i}$ with critical valuesand let $R_i$ be its number of rejections. If the full procedure rejects $i$ and makes $r$ rejections, then $R_i=r-1$, and converselyUnder Assumption A and the super-uniformity of a true-null p-value,Summing over $i\in I_0$ gives $\mathrm{FDR}\le \alpha m_0/m\le \alpha$.

Increasing any coordinates of $p_{-i}$ cannot increase the number of modified BH rejections. Henceis an increasing set.

Under Assumption B, put $t_r=\alpha r/m$ and $D_0=\varnothing$. The contribution of true null $i$ is bounded byNowPositive regression dependence and $t_{r-1}\le t_r$ implyThe resulting sum telescopes to at most one. Thus each true null again contributes at most $\alpha /m$, proving FDR control under Assumption B.

Under $H_0$, conditional independence givesMultiplication by the measurable sign $s(z_1)$ and the law of total expectation prove the first identity.

WriteThenConditional orthogonality under $H_0$ and the variance bounds in (ii) giveand similarly for the term containing ${\xi }_i{a}_i$. The Cauchy-Schwarz inequality bounds the last term bywhich is $o_p(n^{-1/2})$ by $n{\mathrm{MSPE}}_f{\mathrm{MSPE}}_g\to 0$.

The leading summands ${ϵ}_i{\xi }_{i}s(z_i)$ are independent, centered, and have variance $\mathrm{Var}({ϵ}_1{\xi }_1)$ because $s^2=1$. The central limit theorem and consistency of ${\tau }_D$ therefore give, by the Slutsky theorem,This is the generalized covariance measure statistic.

Without the null, condition first on $(x_1,z_1)$. Sincethe term involving $\mathbb{E}(x_1\mid z_1)$ vanishes, givingFor the specified alternative, independence and $\mathbb{E}{x}_1=0$ make the right sideThe constant choice $s=1$ gives zero because $\mathbb{E}h(z_1)=0$, so no first-order power is expected. Takinginstead gives $\mathbb{E}(x_1^2)\mathbb{E}|h(z_1)|>0$, producing asymptotic power.

The Group Lasso penalty isBy the Cauchy-Schwarz inequality within each group,

Put $\delta =\hat{\beta }-{\beta }^0$. Comparing the objective at $\hat{\beta }$ and ${\beta }^0$ givesOn $\Omega$, the preceding duality inequality givesThe triangle inequality $P(\delta )\le P({\beta }^0)+P(\hat{\beta })$ then yields

When $m_j=\sqrt{r}$ and $X_{G_j}^T{X}_{G_j}=n{I}_r$,Writing ${\delta }_0=n{\lambda }^2-1\in (0,1)$, the exponential Markov inequality and the supplied chi-square moment-generating-function bound giveThe prescribed equation makes this $q^{-(A+1)}$. A union bound over the $q$ groups therefore gives

For disjoint nonempty index sets $A$ and $B$, partition the mean and covariance conformably. The conditional multivariate normal distribution isIf the sets overlap, the shared coordinates are fixed and this formula applies to $A\setminus B$.

Partition ${\Omega }_0$ into the $A$ and $A^c$ blocks. The block-inverse formula identifies the displayed conditional covariance withFor $A=\{j,k\}$, inversion of the two-by-two precision block givesThe denominator is positive, so the conditional covariance vanishes exactly when ${\Omega }_{0,jk}=0$. A Gaussian pair is independent exactly when it is uncorrelated, proving

Profiling the Gaussian likelihood over $\mu$ gives $\hat{\mu }=\bar{X}$. Apart from constants and a positive factor, the negative log-likelihood for the precision matrix isIts derivative is $-{\Omega }^{-1}+S$, so the unpenalized minimizer is $S^{-1}$. The Graphical Lasso solveswith some conventions also penalizing the diagonal.

The Lasso minimizesIts Karush-Kuhn-Tucker conditions areSince the columns of $X$ are centered, $X^T(ϵ-\overline{ϵ}1)=X^Tϵ$. Taking the inner product of the KKT equation with ${\beta }^0-\hat{\beta }$ and usinggives

Put $\delta =\hat{\beta }-{\beta }^0$. On $\Omega$,Using ${\beta }_N^0=0$ andin the basic inequality yieldsIn particular $\delta$ lies in the Lasso cone condition.

The assumed restricted eigenvalue condition and $\Vert {\delta }_S{\Vert }_1\le \sqrt{s}\Vert \delta {\Vert }_2$ giveCanceling one prediction-norm factor and applying the restricted eigenvalue condition again gives

ChooseEvery null coordinate satisfies $|{\hat{\beta }}_j|<\tau$. For $j\in S$,Thus ${\hat{S}}^{\tau }=S$ on $\Omega$.