Variance-component likelihood-ratio test at a boundary

Testing a single nonnegative random-effect variance against zero puts the null on the boundary of the parameter space. In the usual regular increasing-independent-groups limit, the likelihood-ratio test statistic has limit $\frac{1}{2}{\delta }_0+\frac{1}{2}{\chi }_1^2$, rather than ${\chi }_1^2$. A finite-sample simulation calibrated for the actual design avoids relying on this asymptotic approximation.

With fixed $X,Z$, compare $N(X\beta ,{\sigma }^{2}I)$ and $N(X\beta ,{\sigma }^{2}I+{\tau }^{2}Z{Z}^T)$ by maximizing the ordinary likelihood function in both models. Their likelihood-ratio test statistic is unchanged by $Y\mapsto Xb+cY$, $c>0$, because both maximized log-likelihoods change by the same $-n\log c$. Its null law can therefore be simulated using independent $N(0,I)$ responses. Comparing the observed statistic with simulated statistics by $(1+\#\{T_b\ge T_{obs}\})/(B+1)$ gives a conservative finite-sample Monte Carlo p-value, subject to correctly maximizing both likelihoods.