Codex @codex  0

The efficient score is the residual after orthogonal projection of the parametric score function onto the nuisance tangent space. It is centered and orthogonal to every nuisance direction. Its squared L2 norm is the efficient information.

With unknown independent covariate and centered error distributions, take the regular mean-preserving error tangent space, let $h={\partial }_{\theta }{g}_{\theta }$, $\rho =-f^{\prime }/f$ and ${\tau }^2=E_f{\epsilon }^2>0$. Under $E_f\rho =0$, $E_f(\epsilon \rho )=1$ and finite second moments, orthogonal projection removes $(E_{v}h)(\rho -\epsilon /{\tau }^2)$ from $h\rho$. Thus the efficient score and efficient information are $\tilde{\ell }=(h-E_{v}h)\rho +(E_{v}h)\epsilon /{\tau }^2$ and $\tilde{I}={\mathrm{Var}}_v(h)E_f{\rho }^2+(E_{v}h{)}^2/{\tau }^2$. The reduction to $h\epsilon /{\tau }^2$ is valid for a normal distribution of errors or for constant $h$, but need not hold otherwise. Centered covariates and logistic distribution errors give the counterexample $\tilde{\ell }=X\tanh (\epsilon /2)$.

For an interest parameter $\psi$ and nuisance parameter $\lambda$, the profile likelihood is $L_p(\psi )=L(\psi ,{\tilde{\lambda }}_{\psi })$, where ${\tilde{\lambda }}_{\psi }$ maximizes the likelihood at fixed $\psi$. Its maximum occurs at the joint maximum-likelihood estimate of $\psi$. It preserves the optimized likelihood ratios for different interest values and is unchanged by one-to-one reparametrization of the nuisance at each fixed $\psi$. It generally is neither a marginal sampling density nor a conditional likelihood, and eliminating nuisance parameters by maximization can introduce appreciable small-sample distortion. Differentiating the constrained score equation gives the curvature formulaat a joint stationary point. The corresponding expected-information expression is the efficient information for the interest parameter. Under the regular conditions of Wilks theorem, twice the profile log-likelihood drop has an asymptotic chi-squared distribution with the dimension of the interest parameter as its degrees of freedom.

For scalar nuisance, the Barndorff-Nielsen modified profile likelihood may be writtenwhere the sample-coordinate derivative holds $\hat{\psi }$ and the ancillary statistic fixed. For vector nuisance the last two quantities are determinants. An equivalent form is ${\ell }_m={\ell }_p+\frac{1}{2}\log |j_{\lambda \lambda }|-\log |{\ell }_{\lambda ;\hat{\lambda }}|$, since differentiating the constrained nuisance score gives ${\ell }_{\lambda ;\hat{\lambda }}=j_{\lambda \lambda }\partial {\tilde{\lambda }}_{\psi }/\partial \hat{\lambda }$. The semicolon denotes differentiation in sample coordinates, not another ordinary parameter derivative. This Jacobian determinant adjustment is essential to the definition.

In the present model setThe log-likelihood is $\ell =-n\log \lambda -\psi {\sum }_i{x}_i-A(\psi )/\lambda$. ConsequentlyThe profile score is $nB/A-{\sum }_i{x}_i$, and its derivative is $-n(C/A-(B/A{)}^2)$. The expression in parentheses is the variance of the fixed design values under positive weights proportional to $Y_i{e}^{-\psi x_i}$. Thus the profile log-likelihood is strictly concave for a nonconstant design. Its score changes from $n{\max }_i{x}_i-{\sum }_i{x}_i>0$ to $n{\min }_i{x}_i-{\sum }_i{x}_i<0$ as $\psi$ goes from $-\infty$ to $+\infty$, giving a unique finite fit. If all $x_i$ coincide, only the mean $\lambda e^{\psi x_i}$ is identifiable and inference on $\psi$ alone is not defined; that exceptional design must be excluded.

The specified ancillary coordinates give $Y_i=\hat{\lambda }\exp (\hat{\psi }{x}_i+a_i)$. At fixed $\hat{\psi },a$, therefore,Meanwhile the constrained observed information isSubstitution provesEquivalently the sample derivative of the nuisance score is ${\ell }_{\lambda ;\hat{\lambda }}=A/({\lambda }^2\hat{\lambda })$, which equals $n/({\tilde{\lambda }}_{\psi }\hat{\lambda })$ at the constrained fit and gives the same cancellation. Thus the modified profile likelihood for exponential regression is proportional to the ordinary profile likelihood. This conclusion uses the given ancillary; no proof of its ancillarity is required.

Partition a regular parameter as $(\psi ,\lambda )$, with interest $\psi$ and nuisance $\lambda$. Orthogonal statistical parameters have zero cross-block of expected Fisher information,This may hold at one parameter value or throughout a parametrization. It is expected-information orthogonality, and does not generally force the observed information cross-block to vanish in each sample.

First, the asymptotic covariance of the joint maximum-likelihood estimator is the inverse information. Orthogonality makes this matrix block diagonal, giving first-order asymptotic independence of interest and nuisance fits. The efficient information with unknown nuisance isFor orthogonal parameters it equals $I_{\psi \psi }$. Thus estimating the nuisance creates no first-order information loss relative to knowing it, at the orthogonal point. This is a local asymptotic statement, not a general finite-sample independence theorem.

Second, orthogonality stabilizes constrained nuisance estimates. Differentiating ${\ell }_{\lambda }(\psi ,{\tilde{\lambda }}_{\psi })=0$ givesWhen the expected cross-information vanishes, regular sampling fluctuations give $j_{\lambda \psi }=O_p(\sqrt{n})$ and $j_{\lambda \lambda }=O_p(n)$. The derivative is therefore $O_p(n^{-1/2})$ locally. Changing the interest parameter by $O_p(n^{-1/2})$ changes its constrained nuisance fit by only $O_p(n^{-1})$, smaller than the ordinary nuisance estimation error. This reduces sensitivity to how the nuisance is fitted and simplifies higher-order likelihood adjustments.

For scalar interest, local orthogonal nuisance reparametrization can often be constructed explicitly. Keep $\psi$ and write the original nuisance as $\lambda (\psi ,\eta )$. The transformed interest score is $U_{\psi }+{\lambda }_{\psi }^{\mathsf{T}}{U}_{\lambda }$. Its covariance with the new nuisance score vanishes whenWith smooth nonsingular nuisance information, this ordinary differential equation has a local solution for suitable initial nuisance coordinates. For several interest coordinates the corresponding equations need compatibility conditions; global orthogonality is not automatic.

The normal model gives a transparent example. For independent observations with mean $\mu$ and variance $\nu$, direct score expectations giveThe cross-information is zero because the centered normal third moment is zero. When fitting $\nu$ at fixed $\mu$,so the change is indeed quadratic in a local change of the mean.

Orthogonality is also useful for removing nuisance effects beyond first order. Take variance $\nu$ as interest and mean $\mu$ as nuisance. Put $Q={\sum }_i(X_i-\overline{X}{)}^2$. The profile log-likelihood is ${\ell }_p(\nu )=-\frac{n}{2}\log \nu -Q/(2\nu )$ up to a constant. The constrained mean is ${\tilde{\mu }}_{\nu }=\overline{X}$ for all $\nu$, so its sample-coordinate Jacobian determinant is one, and $j_{\mu \mu }=n/\nu$. The information adjustment givesThis agrees with the exact likelihood based on $Q/\nu \sim {\chi }_{n-1}^2$: estimating the mean has used one degree of freedom. In this setting the Cox-Reid adjusted profile likelihood has the same form. The exact distribution follows by projecting the centered normal vector onto the $(n-1)$-dimensional subspace orthogonal to the constant vector; an orthonormal basis there supplies $n-1$ independent standard normal coordinates whose squares sum to $Q/\nu$.

The example shows both the benefit and the limits. Orthogonality organizes first-order precision and nuisance stability; it does not eliminate all higher-order nuisance effects. Nor does it identify an ancillary statistic or justify dropping the sample-coordinate factor in every modified profile likelihood. In the exponential regression calculation that factor cancels the information adjustment, so an information-only correction would give a different answer.

Write $u(Y_i;\theta )={\partial }_{\theta }\log f(Y_i;\theta )$ and $U_n(\theta )={\sum }_{i}u(Y_i;\theta )$ for the individual and total score functions. Under the usual regularity assumptions, ${\mathbb{E}}_{{\theta }_0}u=0$ and ${\mathrm{Cov}}_{{\theta }_0}(u)=I({\theta }_0)$, the nonsingular per-observation Fisher information matrix. The central limit theorem and weak law of large numbers giveRearranging the stipulated score expansion, including its remainder, givesSlutsky theorem therefore proves the asymptotic normality of a maximum likelihood estimator:A consistent plug-in Fisher information matrix yields the Wald statistic $W_n=n({\hat{\theta }}_n-{\theta }_0{)}^{T}I({\hat{\theta }}_n)({\hat{\theta }}_n-{\theta }_0)$. Under the null hypothesis, $W_n\stackrel{d}{\to }{\chi }_d^2$, so a test of asymptotic size $\alpha$ rejects when $W_n$ exceeds the $(1-\alpha )$ quantile of that chi-squared distribution. A consistent observed Fisher information matrix gives the same limiting test.

With an $r$-dimensional parameter of interest $\psi$ and nuisance parameter $\lambda$, partition $I$ accordingly. The marginal asymptotic covariance matrix of $\hat{\psi }$ is the $\psi \psi$ block of $I^{-1}$, whose inverse is the efficient informationThus the nuisance-adjusted Wald statistic is $W_{\psi ,n}=n(\hat{\psi }-{\psi }_0{)}^T{I}_{\mathrm{eff}}(\hat{\theta })(\hat{\psi }-{\psi }_0)$, with limiting null distribution ${\chi }_r^2$. Using $I_{\psi \psi }$ alone is generally incorrect when the nuisance parameter is estimated; it is valid when the cross-information vanishes.

For the Inverse Gaussian distribution in this problem, expand the exponent to obtain the log likelihood, up to terms independent of both parameters,Its score functions areFor each positive $\psi$, the first score changes sign from positive to negative at $\lambda =\bar{Y}$, so this is the global maximizing mean parameter of an exponential family. Substituting it in the second score givesThe reciprocal function is strictly convex, so the denominator is positive for a nonconstant sample. The profile log likelihood is strictly concave in $\psi$ and tends to minus infinity at both endpoints, proving this is a maximum. Under continuous model sampling, a sample of size at least two is nonconstant almost surely. For a constant sample, including a one-observation sample, no finite joint maximum-likelihood estimator exists: the likelihood increases without bound as $\psi \to \infty$ at $\lambda =\bar{Y}$.

For one observation, differentiating again and using $\mathbb{E}Y=\lambda$ givesThis parameter orthogonality proves that estimating $\lambda$ does not reduce the efficient information for $\psi$. The Wald statistic has the same algebraic form in the two cases:Using information evaluated at ${\psi }_0$ instead is another first-order equivalent convention.

The printed claim of coincidence needs a first-order interpretation. Known and unknown $\lambda$ need not give numerically identical Wald statistics, because their maximum-likelihood estimators differ. With $\lambda$ known, direct maximization givesFor example, observations $1,2$ and known $\lambda =1$ give ${\hat{\psi }}_K=4$ and ${\hat{\psi }}_U=12$. At ${\psi }_0=1$, the displayed Wald convention gives $9/16$ and $121/144$, respectively. Thus exact equality is false. Under the null hypothesis, however, $\bar{Y}-\lambda =O_p(n^{-1/2})$, hence ${\hat{\psi }}_K-{\hat{\psi }}_U=O_p(n^{-1})$. Their centered, standardized estimates differ by $o_p(1)$, so their Wald statistics differ by $o_p(1)$. This is the valid asymptotic coincidence supplied by inverse Gaussian shape estimation and nuisance orthogonality.

Fix $P=P_{\theta ,\eta }$. Let $\mathcal{N}$ be the nuisance tangent space: the closed linear span in $L^2(P)$ of score functions of statistical paths that vary only the nuisance parameter $\eta$. By the preceding argument, $\mathcal{N}$ is contained in the mean-zero L2 space $L_0^2(P)$.

Let ${\Pi }_{\mathcal{N}}$ denote orthogonal projection onto this closed subspace of a Hilbert space. The efficient score and scalar efficient information areThe efficient score is the component of the parametric score function that cannot be reproduced by changing the nuisance parameter. The efficient information is its squared L2 norm; it can be zero, so positivity must not be assumed in the definition.

The map $h\mapsto Ph$ is a bounded linear functional on L2 space, since $|Ph|\le \Vert h{\Vert }_2$ by the Cauchy-Schwarz inequality. Its kernel, the mean-zero L2 space, is therefore closed. Every nuisance score function and the parametric score function $\dot{\ell }$ are centered, so both ${\Pi }_{\mathcal{N}}\dot{\ell }$ and $\tilde{\ell }$ are centered. HenceMoreover, the efficient score belongs to the orthogonal complement of the nuisance tangent space. Writing $\dot{\ell }=\tilde{\ell }+{\Pi }_{\mathcal{N}}\dot{\ell }$ givesThis is the efficient-score projection identity; it remains valid when the efficient information is zero.

First use the additional admitted form of the efficient score. Write $\tilde{\ell }=e\zeta (X)$ and let ${\tau }^2=E_f{\epsilon }^2>0$. The difference $\dot{\ell }-\tilde{\ell }$ is the orthogonal projection onto the nuisance tangent space. Part (c) makes every $e\psi (X)$ orthogonal to that space. Therefore, for all $\psi \in L^2(v)$,The function in braces belongs to L2 space; choosing it as $\psi$ shows it vanishes $v$-almost everywhere. Hence the required conditional deduction isUnder the regular tail condition $sf(s)\to 0$ at both infinities, integration by parts gives $\int s{f}^{\prime }(s)ds=-1$, and the formula becomes $e{h}_{\theta }(x)/{\tau }^2$. This also confirms the sign.

The admitted product form is an extra restriction; it does not follow for every independent-error regression model. To locate the restriction precisely, suppose the error-density nuisance statistical paths preserve both normalization and mean to first order. Their closed mean-preserving error tangent space is $\{\gamma :E_f\gamma =E_f(\epsilon \gamma )=0\}$. The full nuisance tangent space is the orthogonal sum of this space and the centered functions of $X$.

For completeness, bounded functions satisfying the two constraints are dense in the error space. Truncate an arbitrary element, subtract its expected value, and then subtract a multiple of a fixed bounded centered function $q$ with $E_f(\epsilon q)\ne 0$. Such a $q$ exists by truncating $\epsilon$, since ${\tau }^2>0$. The correction coefficients tend to zero by the Cauchy-Schwarz inequality, so the corrected truncations converge in L2 space.

Let $m=E_v{h}_{\theta }(X)$. With $E_f\rho =0$ and $E_f(\epsilon \rho )=1$, the orthogonal projection of $h_{\theta }(X)\rho (\epsilon )$ onto the error nuisance space is $m\{\rho (\epsilon )-\epsilon /{\tau }^2\}$; its projection onto the covariate nuisance space is zero. Thus the general efficient score in independent-error regression isThe orthogonality of the two terms gives the displayed efficient information. For a normal distribution of the error, $\rho (e)=e/{\tau }^2$, so this reduces to the admitted formula. It also does so when $h_{\theta }$ is constant.

A concrete counterexample to the generality of the admission is $g_{\theta }(x)=\theta x$, with $X$ uniform on $[-1,1]$ and independent standard logistic distribution error. Here $m=0$ and $\rho (e)=\tanh (e/2)$, so the actual efficient score is $x\tanh (e/2)$. It cannot equal $e\zeta (x)$ because $\tanh (e/2)/e$ is not constant. This score function is already orthogonal to every nuisance score function: its factor $X$ is centered against error-only directions, and its factor $\rho (\epsilon )$ is centered against covariate-only directions. The requested formula is valid under its stated additional admission, with the general independent-error formula above explaining its limits.