Past exam of the mathematics course of the University of Cambridge / 2021 / iii / Paper 218 (Statistical Learning in Practice)

Put $s=x^{T}x>0$ and $z=x^{T}Y$. With objective $\Vert Y-x\beta {\Vert }_2^2+\lambda {\beta }^2$, ridge regression givesFor the duplicated design, the objective depends on ${\beta }_1+{\beta }_2$ through the loss and symmetry makes the minimum-penalty decomposition equal:Duplicating a predictor therefore halves its effective ridge penalty.

The one-variable constraint is ${\beta }^2\le t$, an interval whose endpoints expand as $\sqrt{t}$. The duplicated constraint is the disk ${\beta }_1^2+{\beta }_2^2\le t$; the loss contours are parallel strips perpendicular to $(1,1)$. Their first contact with the disk lies on ${\beta }_1={\beta }_2$. Equivalently, the fitted total coefficient is the least-squares coefficient clipped to $[-\sqrt{2t},\sqrt{2t}]$, compared with $[-\sqrt{t},\sqrt{t}]$ for one copy.

The two Lasso problems areandIf $\hat{\beta }$ solves the first problem, the complete solution set of the second isIndeed, the feasible totals are exactly $[-t,t]$, the same as in the first problem.

The duplicated feasible set is the diamond $|{\beta }_1|+|{\beta }_2|\le t$. A loss contour first touches an entire line segment of the diamond whenever the optimal total has the sign of a sloping face; all points on that segment give the same fit. As $t$ passes the unconstrained optimum, the solution set becomes the intersection of the interior diamond with the line ${\beta }_1+{\beta }_2=z/s$.

For prediction at covariate value $u$, the intercept contributes variance ${\sigma }^2/n$ in every case because $x$ is centered. The one-copy ridge shrinkage is $a_1=s/(s+\lambda )$, while the duplicated-design total has $a_2=2s/(2s+\lambda )$. ThusandSince $a_2>a_1$, duplication reduces ridge bias and increases variance.

For constrained Lasso, let $Z=z/s\sim N({\beta }_0,{\sigma }^2/s)$ and $C_t(Z)=\max (-t,\min (Z,t))$. Both designs have the identical fitted total $C_t(Z)$, so both haveDuplicating the predictor has no effect on Lasso predictions, despite making the coefficient vector nonunique.

For user $j$ and offer $r$, both fits useThe first model treats each click as Bernoulli$(p_j)$. The grouped model treats the click count $C_j=m_j{\mathrm{prop}}_j$ as $\mathrm{Bin}(m_j,p_j)$. Their likelihoods differ only by binomial coefficients independent of the parameters, so their fitted coefficients agree.

The grouped binomial likelihood assumes conditional independence of repeated offers to one user. This is doubtful: persistent unmeasured user preferences and temporal feedback make clicks from the same user positively correlated.

For $m_j$ exchangeable Bernoulli responses of mean $p_j$ and pairwise correlation $\rho$,A standard quasi-binomial model uses a common dispersion multiplier $\varphi$. It cannot represent this variance simultaneously when the offer counts $m_j$ vary, because the multiplier $1+(m_j-1)\rho$ then varies by user.

The beta-binomial regression iswhere the beta distribution is parameterized by mean ${\mu }_j$ and variance parameter $\theta$. Marginally,It matches part c when $\rho =\theta /(1+\theta )$, so it is appropriate at the mean-variance level for a common nonnegative intraclass correlation.

The beta-binomial model mixes binomials over a beta-distributed success probability. The random-intercept generalized linear mixed model instead takesBoth create within-user dependence and overdispersion, but use different mixing distributions. The mixed-model coefficients are conditional on the random effect, while beta-binomial regression is naturally phrased through a marginal mean.

Time is exposure: doubling the public duration should double the expected count without changing the viewing rate. Modeling the rate is therefore preferable to treating time as an additive linear predictor. In a Poisson regression the exact implementation should beusing $\log t_i$ as an offset; directly supplying noninteger $N_i/t_i$ without exposure weights is not generally likelihood-equivalent.

Overdispersion means that conditional variance exceeds the Poisson mean. Under a correctly specified Poisson model, the Pearson statistic is approximately chi-squared with $149$ residual degrees of freedom. Hereso a one-sided five-percent test rejects. The displayed model-based confidence interval is too narrow. A quasi-Poisson fit can multiply standard errors by $\sqrt{1.3119}$; negative binomial regression or a random-effects count model can model the extra variation directly.

There is one row per video. Giving every video its own categorical coefficient saturates the linear predictor. The investment column is a linear combination of the video indicator columns, so the design matrix loses full rank and the investment effect can be shifted into the video effects without changing any fitted value. The model is nonidentifiable.

Fit the exposure-offset Poisson modelFor a new video with investment $I$ and genre $g$, put $\mu =90\exp ({\beta }_0+{\beta }_{1}I+{\gamma }_g)$, computeand select the largest. For fixed genre these probabilities are nonlinear functions of investment, so this is a nonlinear classifier.

For hidden layers $h^{(r)}={\varphi }_r(W_r{h}^{(r-1)}+b_r)$ with $h^{(0)}=x$, the output logits are $z=W_{R+1}{h}^{(R)}+b_{R+1}$ and the softmax function gives

Multinomial logistic regression is the network with no hidden layer: $z=Wx+b$ followed by softmax. Binary logistic classification uses one logit $w^{T}x+b$ followed by the sigmoid function.

Deep sigmoid networks suffer from vanishing gradients because derivatives become small in saturated units and multiply across layers. The ReLU has derivative one on its positive half-line and therefore propagates gradients more effectively there.

For all weights and biases collected in $\vartheta$, fitUse backpropagation with stochastic gradient descent or a modern adaptive variant, treating the absolute-value derivative at zero as stated. Select $({\lambda }_1,{\lambda }_2)$ by validation or cross-validation and refit using the selected pair.

A dead ReLU unit has nonpositive preactivation for every training input, so its output and gradient are always zero. Replace $\max (0,z)$ by the leaky ReLUIts negative-side derivative $a$ permits recovery while it remains nonsaturating, preserving the gradient advantage over sigmoid activation.

For $\eta (x)=\mathbb{P}(Y=1\mid X=x)$, the risk isThe Bayes classifier is ${\psi }^{\mathrm{Bayes}}(x)=1_{\{\eta (x)\ge 1/2\}}$, with risk $\mathbb{E}\min \{\eta (X),1-\eta (X)\}$.

The K-nearest neighbors algorithm takes the majority label among the $k$ training features closest to the query, with a stated tie rule. Its data-dependent risk is the conditional test error given the training sample, and $R_n$ denotes its expectation over that sample.

For one nearest neighbour, condition on a feature value $X=x$ and couple the coincident nearest feature as $X_1=x$. The two labels are conditionally independent Bernoulli$(\eta (x))$, so their mismatch probability isIntegration over $X$ gives

For each $k=1,\dots ,100$, the call to knn.cv performs leave-one-out nearest-neighbour classification and the third line stores the fraction of omitted observations misclassified. Choose a minimizer of ls, then fit the nearest-neighbour classifier with that $k$ to the full data.

A weighted nearest-neighbours classifier predicts one whenwhere neighbours are distance ordered. Under the usual smooth-density and smooth-regression assumptions, asymptotically optimal weights downweight distant neighbours, for example normalized positive parts of $1-(i/k{)}^{2/p}$. The optimal weighted-nearest-neighbour theorem gives smaller leading asymptotic regret than equal weights.

A weakly stationary process has a constant finite mean and covariance $\mathrm{Cov}(X_t,X_{t+h})$ depending only on lag $h$. The plotted process is not stationary: it has a declining trend and a pronounced oscillation of period about $25$, so its mean depends on time.

Fit a trend ${\hat{T}}_t$, form $D_t=X_t-{\hat{T}}_t$, and estimate the period-25 seasonal effect byThe residual is $R_t=X_t-{\hat{T}}_t-{\hat{S}}_{t\mathrm{mod}25}$. This additive decomposition is sensible when seasonal amplitude does not systematically change with the level or trend; multiplicative seasonality would require a logarithmic transform or ratio decomposition.

A period-50 business cycle can be a stochastic cycle in the residual process and need not violate additive trend-plus-period-25 seasonality. Applying $1-B^{50}$ discards the first 50 of only 100 observations and introduces a noninvertible seasonal moving-average factor, creating strong artificial dependence and risking overdifferencing rather than modeling the cycle.

The sample autocorrelation has one substantial positive spike at lag one and then cuts off, while the partial autocorrelation tails off with alternating signs. This is the characteristic pattern of a moving-average process of order one.

The first line searches candidate ARIMA models and selects the one with smallest Akaike information criterion. The second constructs a one-step-ahead point forecast and nominal 95-percent prediction interval from the fitted values of the selected model.

The interval treats the selected model and estimated detrending and seasonal components as fixed, often assumes approximately Gaussian homoscedastic innovations, and ignores model-selection and parameter uncertainty. With only 100 observations these omissions can materially reduce coverage. A residual or parametric bootstrap that repeats decomposition, model selection, fitting, and forecasting can propagate those sources of uncertainty; time-series cross-validation can additionally assess empirical one-step coverage.