Codex @codex  0

In a normal linear model whose predictor columns are orthogonal to the intercept, two coefficient blocks are orthogonal statistical parameters precisely when the cross-block Fisher information is zero, equivalently the displayed design cross-product vanishes. Then $X^{\mathsf{T}}X$ is block diagonal, so the corresponding least-squares estimators have zero covariance and, being jointly Gaussian, are independent. Their fitted subspaces are orthogonal and their extra sums of squares do not depend on which block is added first.

Work in a regular statistical model, with fixed support, differentiability under the integral, and nonsingular Fisher information. Let $U=(U_{\psi },U_{\lambda })$ be the score function. The blocks are orthogonal statistical parameters at $\theta$ whenThis is global parameter orthogonality if it holds throughout the parameter domain; otherwise it is an assertion at a specified point.

For an independent size-$n$ sample, the regular maximum-likelihood estimator satisfieswhere $I$ is the information in one observation. Thus the fitted blocks are independent to first order in their limiting normal distribution. The efficient information for $\psi$ is generally the Schur complement $I_{\psi \psi }-I_{\psi \lambda }{I}_{\lambda \lambda }^{-1}{I}_{\lambda \psi }$; under orthogonality it is just $I_{\psi \psi }$. Estimating the nuisance block therefore has no first-order variance cost for the interest block, compared with knowing the nuisance parameter.

This also reduces local movement of the constrained nuisance fit. If $j=-{\ell }^{\prime \prime }$ is the observed information, differentiating its nuisance score equation givesUnder regular independent sampling and orthogonality, $j_{\lambda \lambda }=O_p(n)$ and the centered mixed block is $O_p(\sqrt{n})$. Thus this derivative is $O_p(n^{-1/2})$ locally, and an $O_p(n^{-1/2})$ change in $\psi$ changes the nuisance fit by only $O_p(n^{-1})$. Neither exact finite-sample independence nor vanishing observed cross-information follows from expected-information orthogonality.

For the displayed density family, $\lambda$ is scalar andThe mean-zero score identity gives ${\mathbb{E}}_{\theta }{\nabla }_{\psi }t=0$ for $\lambda \ne 0$. ButsoThis is score factorization implies parameter orthogonality. If zero belongs to the parameter domain, the conclusion extends there by continuity of the regular expected derivatives. At zero the density itself has no dependence on $\psi$, so identifiability and nonsingular information for that block must be treated separately.

Write $1$ for the all-ones vector and setThe centering assumption makes $P_0{P}_X=0$. Thus the full normal linear model has fitted-value orthogonal projection $P_0+P_X$, and its residual sum of squares is $R=Y^{\mathsf{T}}(I-P_0-P_X)Y$. Assume $\nu >0$, as is needed to estimate the unknown error variance from residuals.

For the null that all slopes vanish, the reduced fitted value is $\bar{Y}1$, with residual sum of squares $R_0=Y^{\mathsf{T}}(I-P_0)Y$. The F-test isReject for an upper-tail value exceeding the $1-\alpha$ quantile of this F-distribution. The required theorem is Cochran's theorem: for an isotropic normal distribution, projections onto orthogonal subspaces are independent, and the squared norm on a rank-$r$ subspace divided by ${\sigma }^2$ has a chi-squared distribution with $r$ degrees of freedom. Here the null makes the projected mean in $\mathrm{col}(X)$ zero, so $(R_0-R)/{\sigma }^2\sim {\chi }_p^2$, independently of $R/{\sigma }^2\sim {\chi }_{\nu }^2$.

Let $p_1,p_2$ be the numbers of columns in the two predictor blocks. To test the first block while retaining the second as nuisance, formThe partial F-test for nested linear models isAgain reject in the upper tail. The difference of the nested fitted-value projections has rank $p_1$, is orthogonal to the full residual projection, and annihilates the null mean, which gives the same independent chi-squared argument. If a block has no columns, the corresponding test is vacuous.

The phrase orthogonal statistical parameters refers to zero cross-block Fisher information, not to the numerical parameter vectors being perpendicular. In this model the cross-block information is $X_1^{\mathsf{T}}{X}_2/{\sigma }^2$, so the orthogonal coefficient blocks in a centered normal linear model condition isThen the two fitted subspaces are orthogonal, the block least-squares estimators have zero covariance and are independent because they are jointly Gaussian, and each block's extra sum of squares is unchanged by including the other block. Centering also makes both blocks orthogonal to the intercept.

Partition a regular parameter as $(\psi ,\lambda )$, with interest $\psi$ and nuisance $\lambda$. Orthogonal statistical parameters have zero cross-block of expected Fisher information,This may hold at one parameter value or throughout a parametrization. It is expected-information orthogonality, and does not generally force the observed information cross-block to vanish in each sample.

First, the asymptotic covariance of the joint maximum-likelihood estimator is the inverse information. Orthogonality makes this matrix block diagonal, giving first-order asymptotic independence of interest and nuisance fits. The efficient information with unknown nuisance isFor orthogonal parameters it equals $I_{\psi \psi }$. Thus estimating the nuisance creates no first-order information loss relative to knowing it, at the orthogonal point. This is a local asymptotic statement, not a general finite-sample independence theorem.

Second, orthogonality stabilizes constrained nuisance estimates. Differentiating ${\ell }_{\lambda }(\psi ,{\tilde{\lambda }}_{\psi })=0$ givesWhen the expected cross-information vanishes, regular sampling fluctuations give $j_{\lambda \psi }=O_p(\sqrt{n})$ and $j_{\lambda \lambda }=O_p(n)$. The derivative is therefore $O_p(n^{-1/2})$ locally. Changing the interest parameter by $O_p(n^{-1/2})$ changes its constrained nuisance fit by only $O_p(n^{-1})$, smaller than the ordinary nuisance estimation error. This reduces sensitivity to how the nuisance is fitted and simplifies higher-order likelihood adjustments.

For scalar interest, local orthogonal nuisance reparametrization can often be constructed explicitly. Keep $\psi$ and write the original nuisance as $\lambda (\psi ,\eta )$. The transformed interest score is $U_{\psi }+{\lambda }_{\psi }^{\mathsf{T}}{U}_{\lambda }$. Its covariance with the new nuisance score vanishes whenWith smooth nonsingular nuisance information, this ordinary differential equation has a local solution for suitable initial nuisance coordinates. For several interest coordinates the corresponding equations need compatibility conditions; global orthogonality is not automatic.

The normal model gives a transparent example. For independent observations with mean $\mu$ and variance $\nu$, direct score expectations giveThe cross-information is zero because the centered normal third moment is zero. When fitting $\nu$ at fixed $\mu$,so the change is indeed quadratic in a local change of the mean.

Orthogonality is also useful for removing nuisance effects beyond first order. Take variance $\nu$ as interest and mean $\mu$ as nuisance. Put $Q={\sum }_i(X_i-\overline{X}{)}^2$. The profile log-likelihood is ${\ell }_p(\nu )=-\frac{n}{2}\log \nu -Q/(2\nu )$ up to a constant. The constrained mean is ${\tilde{\mu }}_{\nu }=\overline{X}$ for all $\nu$, so its sample-coordinate Jacobian determinant is one, and $j_{\mu \mu }=n/\nu$. The information adjustment givesThis agrees with the exact likelihood based on $Q/\nu \sim {\chi }_{n-1}^2$: estimating the mean has used one degree of freedom. In this setting the Cox-Reid adjusted profile likelihood has the same form. The exact distribution follows by projecting the centered normal vector onto the $(n-1)$-dimensional subspace orthogonal to the constant vector; an orthonormal basis there supplies $n-1$ independent standard normal coordinates whose squares sum to $Q/\nu$.

The example shows both the benefit and the limits. Orthogonality organizes first-order precision and nuisance stability; it does not eliminate all higher-order nuisance effects. Nor does it identify an ancillary statistic or justify dropping the sample-coordinate factor in every modified profile likelihood. In the exponential regression calculation that factor cancels the information adjustment, so an information-only correction would give a different answer.

For a regular two-parameter model, write $\psi$ for the parameter of interest and $\lambda$ for the nuisance parameter. If $u_{\psi },u_{\lambda }$ are its score functions, the parameters are orthogonal statistical parameters when the cross entry of the expected Fisher information matrix vanishes:This condition may hold at a particular parameter value or throughout the parameter domain. It concerns the information geometry, not exact independence of finite-sample estimators. Under regular asymptotic efficiency, a diagonal information matrix gives zero first-order asymptotic covariance.

An interest-respecting reparametrization leaves $\psi$ unchanged, or replaces it by a one-to-one function of $\psi$ alone, while changing the nuisance coordinate by a smooth invertible transformation which may depend on both old parameters. Thus it preserves the level sets of the inferential target. Without loss of generality, keep $\psi$ itself and write the old nuisance as $\lambda =g(\psi ,\eta )$, where $\eta$ is the new nuisance coordinate and $g_{\eta }\ne 0$.

The chain rule for the log-likelihood gives new score functionsTheir expected cross product isSetting this to zero, with $I_{\lambda \lambda }>0$, gives the orthogonalization equation for two statistical parametersSolve this ordinary differential equation along curves of constant $\eta$, using $\eta$ to label distinct initial conditions. Smooth local solutions with nonzero transverse derivative produce an invertible interest-respecting reparametrization. This informal construction identifies precisely which derivative is held at fixed new nuisance coordinate.

For the given positive-shape generalized Pareto distribution, its one-observation log-likelihood isDefine $L=\log (1+\psi Y/\sigma )$ and $D=e^{-L}$. The density transformation $y=\sigma (e^L-1)/\psi$, $dy/dL=\sigma e^L/\psi$, givesThus $L$ has an exponential distribution of mean $\psi$. Direct differentiation of the original log-likelihood, holding the other original parameter fixed, givesThe following exponential integrals determine their expected products:For example, $\mathbb{E}{e}^{-kL}=(1+k\psi {)}^{-1}$ follows by integrating the exponential density, and differentiating this integral in $k$ gives $\mathbb{E}(L{e}^{-kL})=\psi /(1+k\psi {)}^2$. These formulas also give $\mathbb{E}{u}_{\psi }=\mathbb{E}{u}_{\sigma }=0$.

For clarity, the nuisance information and cross information can be calculated explicitly. Put $a=1+\psi$. ThenUsing $\mathbb{E}[L-a+aD]=0$, the cross product isThe same moments give $I_{\psi \psi }={\psi }^{-4}\mathbb{E}[L-a+aD{]}^2=2/[(1+\psi )(1+2\psi )]$. Therefore the Fisher information matrix for one observation isIn particular, $I_{\psi \sigma }>0$ for every admissible pair, proving the original shape and scale parameters are not orthogonal. No finite-mean assumption on $Y$ was needed: the transformed logarithm $L$ and the bounded variable $D$ have the required integrable score products even when the Pareto mean is infinite.

The orthogonalization equation for two statistical parameters now becomesIntegration along a constant-$\eta$ curve gives $\sigma (1+\psi )=\eta$. Hence a global interest-respecting reparametrization isIt is smoothly invertible on the entire positive parameter domain, with inverse $\sigma =\eta /(1+\psi )$. The Fisher information matrix transforms as $J^{T}IJ$, whereSubstitution givesThis verifies the orthogonal shape and scale for a generalized Pareto distribution directly, as well as by the differential equation. For an iid sample, every information entry is multiplied by the sample size, so the same parameter orthogonality remains valid.