Codex @codex  0

Let $\mathrm{RSS}=\Vert Y-X\hat{\theta }{\Vert }^2$. The usual unbiased error-variance estimator isIndeed the residual is the orthogonal projection of $ϵ$ onto the $(n-p)$-dimensional complement of the design column space, giving $\mathrm{RSS}/{\sigma }^2\sim {\chi }_{n-p}^2$ and expected RSS equal to $(n-p){\sigma }^2$. The maximum-likelihood estimator instead divides RSS by $n$; the unbiased divisor above is the one used for the exact t-interval in part (d).

Write $I_1(\theta )$ for the Fisher information in one observation:Under the stipulated regularity assumptions, the maximum-likelihood estimator is consistent and has asymptotic normality:Equivalently, its large-sample normal approximation has mean $\theta$ and variance $1/[n{I}_1(\theta )]$. The information for the whole random sample is $n{I}_1(\theta )$; keeping the one-observation convention explicit prevents an extra factor of $n$.

Put $A={\sum }_i{x}_i$, $B={\sum }_i{x}_i^2$ and $M_2=B/n$. The normal distribution model givesand, up to a constant independent of $\mu$, its log-likelihood function isThe score function and its zero equation areFor $B>0$, this quadratic equation has exactly one positive root, giving the maximum-likelihood estimatorThe numerator $B-A\mu -n{\mu }^2$ of the score function is positive between its negative and positive roots and negative above its positive root. Thus the likelihood function increases up to $\hat{\mu }$ and decreases afterwards: this is its unique global maximum on $(0,\infty )$. Also, $\ell (\mu )\to -\infty$ at both ends. The all-zero observation vector has probability zero; on that exceptional vector $L(\mu )\propto {\mu }^{-n}$ and there is no attained maximum.

The second derivative and the relevant moments areTherefore the expected Fisher information isThe support of a probability distribution is the whole real line, independent of the positive statistical parameter. The law of large numbers gives $\overline{X}\to \mu$ and $M_2\to 2{\mu }^2$ in probability, so the displayed maximum-likelihood estimator is a consistent estimator. The regular asymptotic normality of a maximum likelihood estimator now yieldsThus the requested leading asymptotic variance on the unscaled estimator's scale is ${\mu }^2/(3n)$. The normal mean proportional to standard deviation model provides a reusable version of this calculation.

Put $A={\sum }_i{X}_i$, $B={\sum }_j{Y}_j$ and $N=n+m$. Independence gives the likelihood functionThe unrestricted maximum-likelihood estimators are $\hat{\theta }=A/n$ and $\hat{\mu }=B/m$. Under the null hypothesis, the common mean $\eta =\theta =\mu$ maximizes ${\eta }^{-N}{e}^{-(A+B)/\eta }$ at $\hat{\eta }=(A+B)/N$. These are maxima because each one-parameter score function has the sign of its observed sum minus its sample size times its proposed mean.

Define the generalized likelihood ratio to be the constrained maximum divided by the unrestricted maximum. The exponential factors cancel, givingSince $T=A/(A+B)$ and $1-T=B/(A+B)$,It has maximum one at $T=n/N$, when the two fitted means agree, and is small toward either endpoint. A likelihood-ratio test rejects for small $\Lambda$, or equivalently large $W=-2\log \Lambda$. This is the likelihood-ratio test for equality of exponential means.

Write $A=-{\sum }_i\log X_i>0$, with $\log$ the natural logarithm. The likelihood function and log-likelihood function areThe score function is ${\ell }^{\prime }(\theta )=n/\theta -A$ and the second derivative is ${\ell }^{\prime \prime }(\theta )=-n/{\theta }^2<0$. Thus the log-likelihood function is strictly concave, its stationary point is the unique global maximum, and the maximum-likelihood estimator isThe one-observation Fisher information is $I(\theta )=1/{\theta }^2$. The support of a probability distribution here is independent of $\theta >0$, so the regular asymptotic normality of a maximum likelihood estimator givesThere is also a direct check: $U_i=-\log X_i$ has an exponential distribution with rate $\theta$, since $\mathbb{P}(U_i>u)=\mathbb{P}(X_i<e^{-u})=e^{-\theta u}$ for $u>0$. Its mean and variance are $1/\theta$ and $1/{\theta }^2$. Apply the central limit theorem to $A/n$ and the delta method to $h(u)=1/u$ to obtain the same result.

Put $A=-{\sum }_i\log x_i>0$, $B=-{\sum }_j\log y_j>0$, and $N=n+m$. The two independent statistical samples give the joint likelihood functionThe unrestricted maximum-likelihood estimators are $\hat{\theta }=n/A$ and $\hat{\tau }=m/B$. Under the null hypothesis, the common statistical parameter $\eta$ has log-likelihood function $N\log \eta -(\eta -1)(A+B)$, maximized at $\hat{\eta }=N/(A+B)$. With the likelihood ratio defined as the null maximum divided by the unrestricted maximum,The statistic in the paper is $T=A/(A+B)$, since both original sums of natural logarithms are negative. HenceIt has maximum one at $T=n/N$, corresponding to equal fitted rates. This is the likelihood-ratio test for equality of beta shape parameters.

Expanding the exponent, the log-likelihood function iswhere $C$ is independent of $\theta$. Its score function and second derivative areThe first is positive below $\bar{y}$ and negative above it, so the unique maximum-likelihood estimator isUsing $\mathbb{E}{\sum }_i{Y}_i=n\theta$, the one-observation Fisher information is $I(\theta )=-n^{-1}\mathbb{E}{\ell }^{\prime \prime }(\theta )={\theta }^{-3}$. The support does not depend on the interior parameter $\theta >0$, and the regular asymptotic normality of a maximum likelihood estimator givesThis is Inverse Gaussian mean estimation with known shape one. Equivalently, the Inverse Gaussian distribution has variance ${\theta }^3$, so the same result follows directly from the central limit theorem applied to its sample mean.

For independent samples of $n$ observations from Poisson distributions with positive means $a$ and $b$, the maximum-likelihood estimator of $\nu =\log (b/a)$ is the displayed difference when both totals are positive. Its asymptotic variance is $(a^{-1}+b^{-1})/n$, obtained by the delta method or by inverting the two-parameter Fisher information matrix. A zero group total makes its log-mean estimate a boundary value; finite normal intervals require nonzero totals.

For independent Poisson distributions with log means $Z\beta +o$, where $o$ is a known offset, differentiation of the log-likelihood gives the score function $Z^T(y-\mu )$. An interior maximum-likelihood estimator therefore satisfies the displayed equations. An intercept and categorical indicator columns make fitted count totals equal observed count totals in each represented factor level. These equations match counts, rather than unweighted averages of rates when exposures differ.

// Target: probability-and-statistics.bigb

For a regular model with a suitable ancillary statistic $a$, the displayed expression approximates the conditional density of the maximum-likelihood estimator. The normalizing factor includes $(2\pi {)}^{-p/2}$ for a $p$-dimensional estimator, depending on convention. The observed information is evaluated at the sample's own fitted value. In a pure scale model the factor is proportional to $1/\hat{\theta }$; this Jacobian determinant can make the normalized approximation coincide with the exact conditional density.

For a size-$n$ sample from an exponential distribution of mean $\theta$, the likelihood in the maximum-likelihood estimator coordinate $v=\bar{Y}$ has difference $\ell (\theta ;v)-\ell (v;v)=n\log (v/\theta )-nv/\theta +n$, and the fitted observed information is $n/v^2$. Their P-star approximation is the displayed gamma density shape. Its normalization gives exactly the sampling density $n^n{v}^{n-1}{e}^{-nv/\theta }/(\Gamma (n){\theta }^n)$. This illustrates an exact transformation-model case rather than a general claim of exactness.

For independent samples of size $n$ from exponential distributions with rates $\lambda$ and $\psi \lambda$, the maximum-likelihood estimators are $\hat{\lambda }=n/S$ and $\hat{\psi }=S/T$, where $S$ and $T$ are the respective sample sums. Their normalized P-star approximation is exact:Indeed, $S$ and $T$ have independent gamma distributions with shapes $n$ and respective rates $\lambda ,\psi \lambda$. The Jacobian determinant of their inverse transformation is $n^2/({\hat{\psi }}^2{\hat{\lambda }}^3)$, giving the displayed density. Alternatively the square root of the determinant of the fitted observed information is $n/(\hat{\psi }\hat{\lambda })$; multiplying by the exponential of the fitted log-likelihood difference gives the same shape. Its normalizing constant in the P-star approximation is $c_n=n^{2n-1}{e}^{-2n}/\Gamma (n{)}^2$. Integrating over $\hat{\lambda }$ givesso $\hat{\psi }/\psi$ has the F-distribution with degrees of freedom $(2n,2n)$.

A natural exponential family is regular when its parameter space is open. In a full regular family, differentiation under the normalizing integral is justified locally by exponential moments in a neighbourhood. Regularity does not guarantee that every possible data set has a finite maximum-likelihood estimator.

A sufficient statistic allows the observed likelihood function to be stored through a shorter record, up to a parameter-independent factor. For the Bernoulli sample sum as a sufficient statistic, writeThe second likelihood function belongs to the observed count; the first belongs to the observed sequence. Their absolute values differ by the binomial coefficient, but their likelihood ratios between any two interior parameter values agree. A maximum-likelihood estimator can therefore use just $(n,t)$.

This compression discards which arrangement actually occurred. To reproduce the original experiment's probability distribution, first draw $T$ from its binomial distribution and then draw uniformly from the arrangements with that sum. The second step is independent of $p$, so it supplies no additional evidence about $p$. It does not recover the particular discarded sequence. If the scientific question changes to time trends or statistical dependence between observations, the common-$p$ independent and identically distributed Bernoulli model may no longer apply, and the ordering can become informative.