= Solution
There are two defects. The mixed fit supplies a <restricted maximum likelihood>, whereas the ordinary fit supplies an ordinary <log-likelihood>, so their difference is not a <likelihood-ratio test statistic>. Also, $H_0:\tau^2=0$ lies on the boundary of $\tau^2\geq0$; the usual <Wilks theorem> does not give a $\chi_1^2$ null law.
A valid approach first refits the mixed model by ordinary <maximum likelihood estimation>, setting REML to false, and fits the same fixed effects under $H_0$. For $V=\tau^2ZZ^T+\sigma^2I$, maximize
$$
\ell(\beta,\tau^2,\sigma^2)=-\frac{176}{2}\log(2\pi)-\frac12\log\det V-\frac12(Y-X\beta)^TV^{-1}(Y-X\beta)
$$
over $\beta,\sigma^2>0,\tau^2\geq0$, and separately over $\beta,\sigma^2>0$ with $\tau^2=0$. Set $T=2(\widehat\ell_1-\widehat\ell_0)\geq0$.
For finite-sample calibration, use the <location-scale invariant simulation test for a Gaussian variance component>. With the actual $X,Z$ fixed, simulate $B$ independent vectors $Y^{(b)}\sim N(0,I_{176})$, refit both models by ordinary <maximum likelihood estimation> to each, and calculate $T_b$ in exactly the same way. Under $H_0$, $Y=X\beta+\sigma z$. Translating by a vector in the <column space> of $X$ and multiplying by a positive scalar preserves the statistic: both maximized log-likelihoods acquire the same scale constant. Thus this simulation has the correct null law without knowing $\beta$ or $\sigma$. A conservative Monte Carlo <p-value> is
$$
\boxed{p_{MC}=\frac{1+\sum_{b=1}^B\mathbf1_{\{T_b\geq T\}}}{B+1};\qquad\text{reject if }p_{MC}\leq0.05.}
$$
In the usual regular limit with increasing independent groups, a <variance-component likelihood-ratio test at a boundary> instead uses $\tfrac12\delta_0+\tfrac12\chi_1^2$: for $T>0$, its approximate tail probability is $\tfrac12\Pr(\chi_1^2\geq T)$, and at $T=0$ the nonrandomized p-value is one. This approximation is not an exact guarantee for sixteen rats. Alternatively one can compare consistently defined restricted likelihoods with the same $X$ and simulate their restricted-ratio null distribution, as in https://search.r-project.org/CRAN/refmans/RLRsim/html/exactRLRT.html[the RLRsim documentation].
Back to article page
