Past exam of the mathematics course of the University of Cambridge / 2019 / iii / Paper 218 (Statistical Learning in Practice)

Let $H_i$ be the number healthy among $m_i=25$ patients and $Y_i=H_i/m_i$. The fitted grouped-binomial logistic regression isFor $y=h/m$ its mass has exponential dispersion family formwhereThus $\mathbb{E}Y=p=b^{\prime }(\theta )$ and

The positive intercept means the fitted baseline probability for untreated women exceeds $1/2$. The positive coefficient ${\hat{\beta }}_S=0.1854$ makes each fitted male log odds exceed the corresponding female log odds, while ${\hat{\beta }}_T=-0.3698$ makes treatment lower the fitted log odds for either sex. Numerically the fitted probabilities are approximatelyTheir order is $F1<M1<F0<M0$, which differs from the observed order $F1<M0<M1<F0$. The additive model therefore cannot reproduce the observed ordering.

Add a sex-by-treatment interaction term:In R the formula becomes prop ~ sex * treatment. There are four parameters for four cell probabilities, so this model is saturated and, provided the cell proportions lie strictly between zero and one, fits each observed proportion exactly. It therefore recovers all the stated order relations.

For observed proportions $y_i$ and fitted proportions ${\hat{p}}_i$, the binomial deviance isA second-order Taylor expansion around $y_i={\hat{p}}_i$ givesthe generalized Pearson chi-squared statistic. Under the fitted binomial model this is approximately ${\chi }_{4-3}^2={\chi }_1^2$. The observed deviance is $1.6102$, whose upper-tail probability is about $0.20$. Since this exceeds $0.05$, there is no significant evidence of overdispersion.

For a model with maximized log likelihood $\ell (\hat{\theta })$, $k$ fitted parameters, and $n$ observations,Both combine goodness of fit with a complexity penalty; smaller values are preferred.

Both criteria are used for model selection. AIC estimates relative out-of-sample predictive or Kullback-Leibler risk and is attractive when prediction is the main aim. BIC approximates a log Bayes factor under regular fixed-dimensional models and is consistent for selecting a true finite-dimensional model when one is present. Their penalties differ by $2k$ versus $k\log n$. For $n>e^2$, BIC penalizes each additional parameter more strongly and therefore tends to select smaller models.

For model $k$, let $X_k$ contain its $p_k$ active columns. With known noise variance one,and full column rank gives

Let $P_k$ be orthogonal projection onto the column space of $X_k$, and let $\mu$ be the true mean. ThenUp to a common constant,The true model has $p_k=p^{\ast }$ and ${\lambda }_k=0$; every other candidate has $p_k>p^{\ast }$ and ${\lambda }_k\ge 0$. Its excess expected AIC is ${\lambda }_k+p_k-p^{\ast }>0$, while its excess expected BIC is ${\lambda }_k+(\log n-1)(p_k-p^{\ast })>0$ when $n>e$. Thus either minimum expected criterion selects the true model.

With one true regressor and one additional candidate regressor, the reduction in residual sum of squares from fitting the larger nested model is $Z^2\sim {\chi }_1^2$. AIC chooses the wrong larger model exactly whenwhose probability is fixed and positive, independent of $n$. BIC chooses it whenFor $n>e^2$ this probability is strictly smaller than the AIC error probability, and it tends to zero as $n\to \infty$.

A generalized linear model is overdispersed when the conditional variance exceeds the variance prescribed by its response family and mean, for example $\mathrm{Var}(Y_i\mid X_i)>V({\mu }_i)$ for a unit-dispersion Poisson or binomial model. Unobserved heterogeneity or dependence among repeated observations can cause it. A generalized linear mixed model adds random effects for the corresponding clusters or subjects, thereby representing this heterogeneity and induced dependence.

For player $i$, competition $c$, and rival $r$, the fitted Poisson generalized linear mixed model isA negative-binomial model is commonly obtained by gamma mixing of a multiplicative Poisson mean and has negative-binomial marginals. Here the Gaussian random intercept gives a Poisson-lognormal mixture and correlates observations from the same player.

The player random intercept models stable player-to-player scoring ability and accounts for the six repeated observations on each player. The manager included it because treating those observations as conditionally independent with one common baseline would ignore both heterogeneity and within-player dependence.

The coefficient $0.4385$ says that, holding competition and player effect fixed, the expected scoring rate against S is multiplied byrelative to R. More goals are scored against S, so R appears to be the stronger rival.

The usual residual-deviance chi-squared calibration is unreliable because random effects are estimated and integrated out, changing both the effective degrees of freedom and the null distribution. Use a parametric bootstrap: fit the reported GLMM; compute an observed dispersion statistic such as the Pearson statistic or its ratio to nominal residual degrees of freedom; for each bootstrap replicate draw ten player effects from $N(0,0.1168)$, simulate all 60 Poisson responses from the fitted conditional means, refit the same GLMM, and recompute the statistic. With $B$ replicates, estimate the upper-tail p-value byReject at $5\%$ when $\hat{p}<0.05$, equivalently when $T_{\mathrm{obs}}$ exceeds the empirical $95$th percentile of the bootstrap null distribution.

The fixed-effect correlation pattern changes only moderately after adding the player random intercept: the competition-B/competition-C correlation remains $0.488$, rival S remains essentially orthogonal to those competition contrasts, and the largest changes involve correlations with the intercept. Thus the principal coefficient dependence comes from the fixed-effect coding and design, while player heterogeneity mainly changes the intercept-related uncertainty. The random effect is scientifically useful without materially changing which fixed contrasts are confounded.

LDA assumes $Y\in \{1,\dots ,L\}$ with prior probabilities ${\pi }_l>0$ andwhere the class means may differ but the nonsingular covariance matrix is common. Bayes' rule chooses the class maximizing $\log {\pi }_l+\log f_l(x)$. Cancelling terms common to every class gives the discriminantThe boundary between classes $l$ and $m$ is ${\delta }_l(x)={\delta }_m(x)$, an affine hyperplane with normal ${\Sigma }^{-1}({\mu }_l-{\mu }_m)$. Hence the Bayes rule $\arg {\max }_l{\delta }_l(x)$ is a linear classifier.

The fitted LDA rule replaces ${\pi }_l,{\mu }_l,\Sigma$ by class proportions, class sample means, and the pooled within-class covariance, then maximizes the resulting ${\hat{\delta }}_l(x)$. Means and covariances have unbounded sensitivity, so a gross outlier can substantially move every LDA boundary. A soft-margin linear support vector machine uses hinge loss; observations beyond the correctly classified margin cease contributing, though mislabeled or extreme points can still matter according to the penalty $C$. Thus the SVM is generally more robust to well-classified extremes, while neither method is automatically robust to adversarial outliers.

When $p>n$, the pooled within-class covariance has rank at most $n-L<p$, so it is singular and ordinary LDA is not defined. For every $\lambda >0$,is positive definite because $v^T{\hat{\Sigma }}_{\lambda }v\ge \lambda \Vert v{\Vert }^2>0$. Replacing the covariance by this matrix gives well-defined regularized LDA. Choose $\lambda$ for predictive performance by cross-validation.

Let $k$ be a positive-definite kernel with feature map $\Phi$ into a reproducing-kernel Hilbert space. Perform regularized LDA on $\Phi (x)$, replacing the within-class covariance operator $C$ by $C+\lambda I$ with $\lambda >0$, which is invertible. The representer property expresses all required inner products and discriminants through the Gram matrix $K_{ij}=k(x_i,x_j)$ and vectors $k(x_i,x)$. This is Kernel LDA. For a nonlinear kernel such as the Gaussian radial-basis kernel, its affine boundaries in feature space pull back to nonlinear boundaries in the original input space.

With column vectors and learned biases,The dimensions are $16\to 100\to 75\to 50\to 26$. The number of parameters is

The loop divides each feature by its sample maximum, putting features with nonnegative values on comparable scales and improving numerical conditioning for gradient optimization. If exactly one hidden layer must use the sigmoid function, use layer 3. Sigmoid derivatives can become small in saturated regions; placing it latest minimizes the number of subsequent gradient multiplications affected by this saturation, while the earlier ReLU layers retain efficient gradient propagation.

For one-hot labels $y_{il}$ and predicted probabilities $p_{il}(\theta )$, the categorical cross-entropy loss isStochastic gradient descent initializes $\theta$, randomly orders the observations in each of five epochs, and for each single-observation batch computes a forward pass, the sample loss, and its gradient, then updatesBackpropagation is used after the forward loss evaluation to compute this gradient from the output layer back through the hidden layers.

With identity hidden activations and no biases, the pre-softmax map is a product of weight matrices. For any nonzero scalar $c$, multiplying the incoming weights of one hidden neuron by $c$ and dividing its outgoing weights by $c$ leaves that product, every output probability, and the cross-entropy unchanged. Every minimizer therefore belongs to an infinite continuum of equivalent parameterizations; more generally, invertible changes of hidden coordinates and their inverse in the adjacent layer give the same network function.

I would use batch_size=1. The resulting stochastic-gradient noise helps move along flat nonidentifiable directions and escape saddle regions, whereas full-batch gradient descent is deterministic and can stagnate in this highly nonconvex, singular parameterization.

An MA($q$) process iswhere $({\epsilon }_t)$ is white noise of variance ${\sigma }^2$. For MA(1), $X_t={\epsilon }_t+\theta {\epsilon }_{t-1}$ andThe transformation $(\theta ,{\sigma }^2)\mapsto (1/\theta ,{\theta }^2{\sigma }^2)$ leaves this autocovariance unchanged. A Gaussian process is determined by its mean and covariance, so the parameters are not identifiable unless one selects, for example, the invertible representative $|\theta |<1$.

An MA($q$) process is invertible when its innovations admit a causal absolutely summable linear representation in present and past observations. With the backshift operator $B$, MA(1) satisfiesIf $|\theta |<1$, the geometric series converges absolutely:Thus the process is invertible.

Since $X_1$ is the sum of two independent centered Gaussian innovations,For any nonzero $a\in {\mathbb{R}}^n$,Expanding each $X_t={\epsilon }_t+\theta {\epsilon }_{t-1}$ expresses this as ${\sigma }^2$ times a sum of squared innovation coefficients. If all coefficients vanished, the coefficient of the latest innovation gives $a_n=0$, and backward induction gives every $a_t=0$, a contradiction. Hence $a^T\Sigma a>0$ and the covariance matrix is positive definite.

A method-of-moments estimator usesUnder $|\theta |<1$, choose the invertible rootwith the continuous value zero when ${\hat{r}}_1=0$. Alternatively maximize the exact Gaussian likelihood using the positive-definite covariance matrix from part (i), producing ${\hat{\theta }}_{\mathrm{ML}}$. Given either estimate,is the moment estimate; likelihood estimation may instead profile ${\sigma }^2$. Under a fixed interior parameter $|\theta |<1$ and standard stationary ergodic finite-moment regularity, both estimators are consistent and asymptotically normal, with Gaussian maximum likelihood asymptotically efficient.

The MA(1) model has much smaller AIC, $2871.79$ versus $3217.1$, so select MA(1). A nominal Wald 95% interval for its non-intercept parameter isThe estimate is close to the noninvertible boundary $\theta =1$, where the regular asymptotic normal approximation becomes poor and likelihood curvature can understate the true one-sided uncertainty. Therefore statement (iii) is the most plausible: the nominal interval is too narrow to attain its stated coverage reliably.