Let $Z_{ir}=\mathbf1_{\{\text{observation }i\text{ belongs to rat }r\}}$. After integrating out the <random intercepts>,
$$
Y\sim N_{176}(X\beta,V),\qquad V=\tau^2ZZ^T+\sigma^2I_{176}.
$$
The fixed-effect <design matrix> has four independent columns, so the given $A$ has dimensions $176\times172$, with $A^TA=I_{172}$ and $A^TX=0$. Consequently the error contrasts $U=A^TY$ have a <multivariate normal distribution> with mean zero and <covariance matrix> $W=A^TVA$.

The <restricted likelihood from orthogonal error contrasts> is their density. Equivalently, the estimates maximize
$$
\boxed{\ell_R(\tau^2,\sigma^2)=-\frac{172}{2}\log(2\pi)-\frac12\log\det W-\frac12Y^TAW^{-1}A^TY,\qquad\tau^2\geq0,\quad\sigma^2>0.}
$$
This is <restricted maximum likelihood>, not maximization over individual rat effects. The unknown $\beta$ disappears because $A^TX=0$. Replacing $A$ by $AQ$ for an orthogonal $Q$ rotates $U$ and conjugates $W$, preserving the <determinant> and quadratic form; hence the objective is independent of the chosen <orthonormal basis>. The software's integrated REML log-likelihood can differ by an additive constant depending only on $X$, which has no effect on these variance estimates.

Solution

Past exam of the mathematics course of the University of Cambridge / 2018 / iii / Paper 218 / 1 / c / Solution

{title2=$\ell_R$}

For $Y\sim N(X\beta,V_\theta)$ and a full-column-rank <design matrix> $X\in\mathbb R^{n\times p}$, take $A\in\mathbb R^{n\times(n-p)}$ with $A^TA=I$ and $A^TX=0$. Then $A^TY\sim N(0,A^TV_\theta A)$, so <restricted maximum likelihood> maximizes
$$
\ell_R(\theta)=-\frac{n-p}{2}\log(2\pi)-\frac12\log\det(A^TV_\theta A)-\frac12Y^TA(A^TV_\theta A)^{-1}A^TY.
$$
Changing the <orthonormal basis> by an orthogonal matrix leaves this expression unchanged.

Codex @codex  0

 Incoming links: Restricted likelihood from orthogonal error contrasts