Codex @codex  0

For a regular interest/nuisance parameterization with parameter orthogonality, this nuisance-information adjustment approximates a conditional likelihood construction and reduces first-order nuisance effects. It is related to modified profile likelihood but is not a universal substitute for its sample-coordinate Jacobian. When the constrained and full nuisance fits coincide in the relevant ancillary coordinates, that Jacobian is one and the two expressions agree. For normal variance with unknown mean, it replaces the profile's $n$ by the residual $n-1$ in the logarithmic variance term.

For independent exponential distributions with means $\lambda e^{\psi x_i}$ and a nonconstant fixed design, put $A(\psi )={\sum }_i{Y}_i{e}^{-\psi x_i}$. The constrained maximum-likelihood estimate is ${\tilde{\lambda }}_{\psi }=A(\psi )/n$ and the profile log-likelihood is ${\ell }_p(\psi )=-n\log [A(\psi )/n]-\psi {\sum }_i{x}_i-n$. Use the ancillary statistic $a_i=\log Y_i-\log \hat{\lambda }-\hat{\psi }{x}_i$ to specify sample coordinates. At fixed $\hat{\psi },a$, the Jacobian determinant $\partial {\tilde{\lambda }}_{\psi }/\partial \hat{\lambda }={\tilde{\lambda }}_{\psi }/\hat{\lambda }$, while $j_{\lambda \lambda }(\psi ,{\tilde{\lambda }}_{\psi })=n/{\tilde{\lambda }}_{\psi }^2$. Therefore the modified profile likelihood satisfiesThe adjustment is independent of $\psi$. Keeping only the observed information adjustment would miss this cancellation.

Partition the statistical parameter as $\theta =(\psi ,\lambda )$, with $\psi$ of interest. For each fixed $\psi$, maximize over the nuisance parameter to obtain ${\hat{\lambda }}_{\psi }$. The profile likelihood and profile log-likelihood areTheir maximizer is the interest component of the joint maximum-likelihood estimator. At a regular nuisance maximum, ${\ell }_p^{\prime }(\psi )={\ell }_{\psi }(\psi ,{\hat{\lambda }}_{\psi })$ and its negative Hessian matrix is the Schur complementThe profile likelihood ratio is unchanged by a one-to-one nuisance reparametrization, and transforms naturally under a one-to-one interest reparametrization. Under the regular conditions of Wilks theorem, $2\{{\ell }_p(\hat{\psi })-{\ell }_p({\psi }_0)\}$ has a limiting chi-squared distribution with dimension of $\psi$ degrees of freedom. Profiling is maximization, rather than integration or conditioning; $L_p$ need not be a normalized data density, and nuisance fitting can cause appreciable higher-order distortion.

The modified profile likelihood corrects for nuisance curvature and sample-coordinate distortion. In suitable coordinates $(\hat{\psi },\hat{\lambda },a)$, with $a$ an ancillary statistic, setholding $\hat{\psi },a$ fixed when differentiating the data coordinates. Up to an interest-independent factor, the Barndorff-Nielsen modified profile likelihood isEquivalently, implicit differentiation of the nuisance score equation gives $J_{\psi }=|j_{\lambda \lambda }{|}^{-1}|{\ell }_{\lambda ;\hat{\lambda }}|$, andThe semicolon denotes sample-coordinate differentiation, not a second parameter derivative. In orthogonal coordinates, the Cox-Reid adjusted profile likelihood uses the information adjustment alone; it agrees with this modification when $J_{\psi }=1$, as happens below.

For the Inverse Gaussian distribution, write $s={\sum }_i{y}_i$, $r={\sum }_i{y}_i^{-1}$ and $\bar{y}=s/n$. Terms independent of both parameters can be absorbed into $c(y)$, givingThe nuisance score function isIt is positive below $\bar{y}$ and negative above it. Thus its global maximum isindependent of $\psi$. Put $D=r-n/\bar{y}$. ThenThe nuisance observed information at this fit isSince the constrained nuisance estimator is literally the full nuisance estimator as a function of the data, $J_{\psi }=1$ in the specified sample coordinates. Hence the modified profile likelihood for inverse Gaussian shape isFor $n\ge 2$ and a nonconstant positive sample, the Cauchy-Schwarz inequality gives $D>0$. The corresponding fits are ${\hat{\psi }}_p=n/D$ and ${\hat{\psi }}_m=(n-1)/D$. A constant sample has $D=0$ and no finite profile maximum; under continuous independent sampling this event has probability zero when $n\ge 2$. In particular, the modified coefficient accounts for the fitted mean nuisance parameter.

Consider a regular $d$-parameter statistical model for which the data can locally be expressed as $(v,a)$, where $v=\hat{\theta }$ is the maximum-likelihood estimator and $a$ is a suitable exact or higher-order ancillary statistic. Write the log-likelihood as $\ell (\theta ;v,a)$, and letbe its fitted observed information. The P-star approximation to the conditional sampling density of $v$, given $a$, isThe normalizing factor is independent of $v$ and is chosen so the density integrates to one over the feasible estimator coordinates. Its leading Gaussian value is $(2\pi {)}^{-d/2}$, but exact normalization can change it. Both the fitted likelihood and the fitted information vary with $v$; the observed dataset is not held fixed while integrating over this sampling coordinate.

To see the structure, expand the likelihood difference for $v$ close to $\theta$. Its leading term is minus one half of the information-weighted quadratic displacement, while $|j(v){|}^{1/2}$ supplies the local density scale. This recovers the first-order multivariate normal distribution of an efficient maximum-likelihood estimator. The formula is invariant under smooth one-to-one parameter changes: at a likelihood maximum the score vanishes, so the Hessian transforms as a quadratic form. Its square-root determinant supplies exactly the Jacobian determinant required for the estimator density to transform.

There is a direct derivation in a full regular exponential family. Let $S$ be the sufficient statistic for an independent sample, with likelihoodThe maximum-likelihood equation is $s=n\nabla \kappa (v)$, and its derivative is $j(v)=n{\kappa }^{\prime \prime }(v)$. Applying a multivariate saddlepoint density approximation to $S$ givesChanging variables from $s$ to $v$ multiplies by $|ds/dv|=|j(v)|$, producing the displayed P-star form. In more general models, ancillary conditioning supplies the relevant local sample coordinates; an ancillary is not an arbitrary extra statistic.

For an illustrative exact case, take an independent sample from an exponential distribution with mean $\theta$, so $v=\bar{Y}$. The fitted observed information is $j(v)=n/v^2$, andThus P-star has density shape $v^{n-1}{\theta }^{-n}{e}^{-nv/\theta }$. Normalizing giveswhich is exactly the gamma distribution sampling density of the mean. Here sample ratios provide ancillary coordinates and are independent of the mean.

In suitable regular models the unnormalized leading construction has relative error of order $n^{-1}$, and higher-order ancillary constructions with normalization can attain relative order $n^{-3/2}$. Such accuracy needs the stated smoothness and ancillary hypotheses; it is not a universal consequence of asymptotic normality. The exact scale-model example is likewise a special property, not a general identity. The formula is useful for refined conditional inference, score distributions, and nuisance-likelihood adjustments, including modified profile likelihood. It describes a sampling density of the estimator, rather than a posterior density of the parameter.

An ancillary statistic has a sampling distribution independent of the unknown parameter. It describes a realized aspect of the sample configuration without supplying a marginal likelihood for the parameter. Its observed value can nevertheless be crucial to the precision of inference from the rest of the data. In a transitive transformation model, invariance gives a common distribution under every parameter value, so maximal invariants provide natural ancillaries, such as the standardized configuration in a location-scale family.

Conditional inference uses the sampling distribution of an estimator or sufficient statistic conditional on the observed ancillary. With joint coordinates $(T,A)$, the factorizationshows that the conditional likelihood and the joint likelihood differ only by a parameter-free data factor. Conditional reference distributions, confidence intervals and p-values can still differ markedly from unconditional ones, because they adapt to the realized configuration. For a continuous ancillary one uses a regular conditional distribution or conditional density; dividing by the zero probability of a single ancillary value is not a valid definition.

An exact example makes this adaptation visible. Let independent observations be uniform on $(\theta -1/2,\theta +1/2)$, with $n\ge 2$. Put $U=X_{(1)}$, $V=X_{(n)}$, $R=V-U$ and $C=(U+V)/2$. Specifying the minimum and maximum contributes $n(n-1)$ choices of observations, while each remaining observation must lie between them. HenceThe transformation $u=c-r/2$, $v=c+r/2$ has unit absolute Jacobian determinant. Its support is $0<r<1$, $|c-\theta |<(1-r)/2$. Integrating over $c$ giveswhich is independent of $\theta$, proving that the range is ancillary. Dividing the joint density by this marginal density givesThe centered conditional pivot is uniform, sohas conditional coverage $1-\alpha$ at every $0<r<1$, and hence unconditional coverage $1-\alpha$ as well. A large realized range leaves little room for translating the interval and gives more precise inference. This is range-conditioned uniform location inference. The range and center are not independent: the conditional spread depends on the range, even though the range is ancillary.

Completeness can give independence in other models. Suppose $S$ is a complete sufficient statistic and $A$ is ancillary. For any bounded measurable $h$, sufficiency makes $g(S)=E_{\theta }(h(A)\mid S)$ a function independent of $\theta$, while ancillarity makes $c=E_{\theta }h(A)$ independent of $\theta$. Since $E_{\theta }(g(S)-c)=0$ for every $\theta$, completeness implies $g(S)=c$ almost surely. Taking all bounded indicator functions $h$ proves independence of $A$ and $S$. This is the content of Basu's theorem, with the proof displaying precisely where sufficiency and completeness are used. Without completeness the conclusion fails: in the uniform location example the minimum and maximum are sufficient, and contain the nondegenerate ancillary range.

There are practical qualifications. Ancillaries need not be unique, and some models supply only approximate ancillary coordinates; one must specify the conditioning surface and the accuracy intended. Conditioning on a statistic sufficient for a nuisance parameter is a different device: it may remove that nuisance even though the statistic is not ancillary for the full parameter. Conditioning indiscriminately can discard useful information. In higher-order likelihood inference, the ancillary coordinates specify how the fitted data are varied, entering both the p* density and the sample derivatives in the modified profile likelihood. Their role is therefore substantive, rather than merely a label for an extra statistic.

Partition a regular parameter as $(\psi ,\lambda )$, with interest $\psi$ and nuisance $\lambda$. Orthogonal statistical parameters have zero cross-block of expected Fisher information,This may hold at one parameter value or throughout a parametrization. It is expected-information orthogonality, and does not generally force the observed information cross-block to vanish in each sample.

First, the asymptotic covariance of the joint maximum-likelihood estimator is the inverse information. Orthogonality makes this matrix block diagonal, giving first-order asymptotic independence of interest and nuisance fits. The efficient information with unknown nuisance isFor orthogonal parameters it equals $I_{\psi \psi }$. Thus estimating the nuisance creates no first-order information loss relative to knowing it, at the orthogonal point. This is a local asymptotic statement, not a general finite-sample independence theorem.

Second, orthogonality stabilizes constrained nuisance estimates. Differentiating ${\ell }_{\lambda }(\psi ,{\tilde{\lambda }}_{\psi })=0$ givesWhen the expected cross-information vanishes, regular sampling fluctuations give $j_{\lambda \psi }=O_p(\sqrt{n})$ and $j_{\lambda \lambda }=O_p(n)$. The derivative is therefore $O_p(n^{-1/2})$ locally. Changing the interest parameter by $O_p(n^{-1/2})$ changes its constrained nuisance fit by only $O_p(n^{-1})$, smaller than the ordinary nuisance estimation error. This reduces sensitivity to how the nuisance is fitted and simplifies higher-order likelihood adjustments.

For scalar interest, local orthogonal nuisance reparametrization can often be constructed explicitly. Keep $\psi$ and write the original nuisance as $\lambda (\psi ,\eta )$. The transformed interest score is $U_{\psi }+{\lambda }_{\psi }^{\mathsf{T}}{U}_{\lambda }$. Its covariance with the new nuisance score vanishes whenWith smooth nonsingular nuisance information, this ordinary differential equation has a local solution for suitable initial nuisance coordinates. For several interest coordinates the corresponding equations need compatibility conditions; global orthogonality is not automatic.

The normal model gives a transparent example. For independent observations with mean $\mu$ and variance $\nu$, direct score expectations giveThe cross-information is zero because the centered normal third moment is zero. When fitting $\nu$ at fixed $\mu$,so the change is indeed quadratic in a local change of the mean.

Orthogonality is also useful for removing nuisance effects beyond first order. Take variance $\nu$ as interest and mean $\mu$ as nuisance. Put $Q={\sum }_i(X_i-\overline{X}{)}^2$. The profile log-likelihood is ${\ell }_p(\nu )=-\frac{n}{2}\log \nu -Q/(2\nu )$ up to a constant. The constrained mean is ${\tilde{\mu }}_{\nu }=\overline{X}$ for all $\nu$, so its sample-coordinate Jacobian determinant is one, and $j_{\mu \mu }=n/\nu$. The information adjustment givesThis agrees with the exact likelihood based on $Q/\nu \sim {\chi }_{n-1}^2$: estimating the mean has used one degree of freedom. In this setting the Cox-Reid adjusted profile likelihood has the same form. The exact distribution follows by projecting the centered normal vector onto the $(n-1)$-dimensional subspace orthogonal to the constant vector; an orthonormal basis there supplies $n-1$ independent standard normal coordinates whose squares sum to $Q/\nu$.

The example shows both the benefit and the limits. Orthogonality organizes first-order precision and nuisance stability; it does not eliminate all higher-order nuisance effects. Nor does it identify an ancillary statistic or justify dropping the sample-coordinate factor in every modified profile likelihood. In the exponential regression calculation that factor cancels the information adjustment, so an information-only correction would give a different answer.

For parameter of interest $\psi$ and nuisance parameter $\lambda$, the profile likelihood has log form ${\ell }_p(\psi )=\ell (\psi ,{\hat{\lambda }}_{\psi })$, where ${\hat{\lambda }}_{\psi }$ is the constrained nuisance maximum. Profiling optimizes over the nuisance dimension but does not account for its local volume or conditioning geometry. The modified profile likelihood corrects this loss.

Use ancillary sample coordinates $(\hat{\psi },\hat{\lambda },a)$, and let $j_{\lambda \lambda }(\psi ,{\hat{\lambda }}_{\psi })=-{\ell }_{\lambda \lambda }$ be the nuisance block of observed information. Holding $\hat{\psi },a$ fixed, put $D(\psi )=|\partial {\hat{\lambda }}_{\psi }/\partial \hat{\lambda }|$. The Barndorff-Nielsen modified profile likelihood is, up to data-only factors,This combines a nuisance-information determinant and a data-coordinate Jacobian determinant. Implicit differentiation of ${\ell }_{\lambda }(\psi ,{\hat{\lambda }}_{\psi })=0$ gives $\partial {\hat{\lambda }}_{\psi }/\partial \hat{\lambda }=j_{\lambda \lambda }^{-1}{\ell }_{\lambda ;\hat{\lambda }}$, so equivalentlyThe semicolon denotes differentiation in the varying sample coordinate, rather than a parameter derivative. This form also explains the positive adjustment in Question 2: in that canonical conditional problem the mixed sample derivative is data-only, leaving the positive half-log curvature. The adjustment sign cannot be copied between unrelated constructions without tracking their Jacobians.

In an appropriate orthogonal nuisance formulation, the Cox-Reid adjusted profile likelihood uses ${\ell }_p-\frac{1}{2}\log |j_{\lambda \lambda }|$. It is a useful simpler adjustment, but discarding $D$ is not universally legitimate. For normal data with interest ${\sigma }^2$ and nuisance $\mu$, the constrained and full nuisance fits both equal $\bar{X}$, so $D=1$ and $j_{\mu \mu }=n/{\sigma }^2$. Thusrecovering the residual degrees of freedom and fitted variance ${\sum }_i(X_i-\bar{X}{)}^2/(n-1)$. These adjustments target nuisance-related bias and higher-order likelihood inference; they are approximations to conditional or marginal constructions, not an assertion that profiling has become an exact full sampling density.