Informant function

The score is the gradient of the log-likelihood, $S_n(\theta )={\nabla }_{\theta }{\ell }_n(\theta )$.

For a regression function $g_{\theta }$, let $h_{\theta }={\partial }_{\theta }{g}_{\theta }$. With independent centered normal distribution error of known variance ${\sigma }^2$, the parametric score function is $h_{\theta }(X)\epsilon /{\sigma }^2$. It has zero conditional expectation given the covariate, so unknown covariate-density nuisance directions do not remove any of its Fisher information.

A score equation sets the score function equal to zero. Its solutions are stationary points of the log-likelihood, and a negative-definite likelihood Hessian identifies an interior local maximum.

Under regularity permitting differentiation under the integral, ${\mathbb{E}}_{\theta }{S}_1(\theta )={\nabla }_{\theta }\int f(x,\theta )dx=0$.

The Fisher information is $I_n(\theta )=\mathbb{E}[S_n{S}_n^T]$ and equals $n{I}_1(\theta )$ for an independent identically distributed sample.

Two statistical parameters are orthogonal when their cross term in the expected Fisher information matrix vanishes. This is pointwise orthogonality at a specified parameter value, or global orthogonality when it holds throughout the parameter domain. It gives a zero cross covariance in the regular first-order asymptotic distribution of efficient estimators, rather than asserting exact finite-sample independence.

In a regular common-support statistical model with probability density function $a(\lambda ,y)e^{\lambda t(y;\psi )}$, the score function in $\psi$ is $\lambda t_{\psi }$. Its zero expectation gives $\mathbb{E}{t}_{\psi }=0$ for $\lambda \ne 0$. The mixed log-likelihood derivative is $t_{\psi }$, so the cross Fisher information matrix block vanishes. At zero the same conclusion follows by continuity when the regular derivatives extend there. This proves parameter orthogonality; it does not establish exact independence of fitted parameters.

In a normal linear model whose predictor columns are orthogonal to the intercept, two coefficient blocks are orthogonal statistical parameters precisely when the cross-block Fisher information is zero, equivalently the displayed design cross-product vanishes. Then $X^{\mathsf{T}}X$ is block diagonal, so the corresponding least-squares estimators have zero covariance and, being jointly Gaussian, are independent. Their fitted subspaces are orthogonal and their extra sums of squares do not depend on which block is added first.

For a regular interest/nuisance parameterization with parameter orthogonality, this nuisance-information adjustment approximates a conditional likelihood construction and reduces first-order nuisance effects. It is related to modified profile likelihood but is not a universal substitute for its sample-coordinate Jacobian. When the constrained and full nuisance fits coincide in the relevant ancillary coordinates, that Jacobian is one and the two expressions agree. For normal variance with unknown mean, it replaces the profile's $n$ by the residual $n-1$ in the logarithmic variance term.

For scalar interest parameter $\psi$, choose new nuisance coordinates $\eta$ and keep $\psi$ fixed. The transformed interest score function is $U_{\psi }+U_{\lambda }^{\top }{\partial }_{\psi }\lambda$, so the displayed differential equation makes its Fisher information cross-block with the nuisance score zero. Under smoothness and nonsingularity it has a local solution from initial coordinates on a fixed interest slice. Orthogonality is local expected-information decoupling; it implies first-order asymptotic independence of fitted blocks, not exact independence or nuisance-free conditional inference.

For the negative binomial distribution with mean $\mu >0$ and size $\theta >0$, the one-observation score function in the mean is $u_{\mu }=\theta (y-\mu )/[\mu (\mu +\theta )]$. Its size derivative is $(y-\mu )/(\mu +\theta {)}^2$, whose expectation is zero. Thus the expected Fisher information matrix is diagonal in $(\mu ,\theta )$. Regular interior maximum-likelihood estimators have zero asymptotic cross covariance, without necessarily being independent in finite samples.

An interest-respecting reparametrization preserves the parameter of interest, or changes it only by a one-to-one function of itself. The nuisance parameter may depend on both old parameters, provided the complete transformation is smoothly invertible. Thus the level sets representing the inferential target are unchanged. Orthogonality can be sought by choosing the new nuisance coordinate along solutions of the orthogonalization equation for two statistical parameters.

Write the old nuisance parameter as $\lambda =g(\psi ,\eta )$ with $g_{\eta }\ne 0$. The transformed score functions are $u_{\psi }^{\mathrm{new}}=u_{\psi }+g_{\psi }{u}_{\lambda }$ and $u_{\eta }=g_{\eta }{u}_{\lambda }$. Their expected cross product is $g_{\eta }(I_{\psi \lambda }+g_{\psi }{I}_{\lambda \lambda })$. Setting it to zero gives the displayed ordinary differential equation along fixed-$\eta$ curves, assuming positive nuisance Fisher information. Smooth solutions with distinct initial conditions give a local interest-respecting orthogonal coordinate system.

The Jeffreys prior uses the square root of the determinant of the Fisher information matrix as a parameter-density kernel. It is invariant under smooth one-to-one reparameterization, since the information and density Jacobians transform compatibly. It may be an improper prior; invariance does not guarantee posterior propriety. With nuisance parameters, a scalar conditional Jeffreys prior and the joint Jeffreys prior need not coincide.

A Poisson distribution with mean $E\lambda$ and positive known exposure has Fisher information $E/\lambda$, so its Jeffreys prior kernel is ${\lambda }^{-1/2}$. The proper shape-rate gamma distribution $\mathrm{Gamma}(1/2,\epsilon )$ has kernel ${\lambda }^{-1/2}{e}^{-\epsilon \lambda }$, approximating the improper kernel where $\epsilon \lambda$ is small. Its posterior is $\mathrm{Gamma}(y+1/2,E+\epsilon )$, converging to $\mathrm{Gamma}(y+1/2,E)$. There is no normalized Jeffreys prior to which the proper priors converge weakly; this is kernel approximation and posterior convergence.

For a differentiable scale family with finite positive Fisher information, the score is $-[1+u(\log f{)}^{\prime }(u)]/\sigma$. Its squared expectation is a constant times ${\sigma }^{-2}$, yielding the displayed Jeffreys prior. This is an improper prior on the entire positive half-line.

For independent normal observations with fixed means and variances $v+r_s$, where $v\ge 0$ is unknown and all known $r_s>0$, the scalar Fisher information for $v$ is $I_{vv}=\frac{1}{2}{\sum }_s(v+r_s{)}^{-2}$. Thus its scalar Jeffreys prior is finite at zero and behaves like $1/v$ at infinity. Unlike a log-flat prior $1/v$, it avoids a divergent integral at the zero-variance boundary. After flat-prior elimination of a Gaussian common mean, the integrated likelihood is bounded by a constant times $v^{-(N-1)/2}$, so this prior yields a proper posterior for $N>1$ and a proper prior on any remaining mean-shape parameters. In the homoscedastic case it reduces to $1/(v+r)$, which is log-flat for the total variance rather than for the latent variance alone.

For $X\sim N_p(\theta ,\Sigma )$ with known nonsingular covariance matrix, the score function is ${\Sigma }^{-1}(X-\theta )$ and the per-observation Fisher information matrix is ${\Sigma }^{-1}$. In particular, covariance $I_p$ gives information $I_p$.

The empirical score outer-product matrix isUnder regularity and consistency of $\hat{\theta }$, $i_n(\hat{\theta })$ converges in probability to the Fisher information matrix. This convention differs from the negative-Hessian convention for Observed Fisher information.

The observed Fisher information is the negative Hessian of the realized log-likelihood. Its expectation, under standard regularity conditions, is the Fisher information matrix.

Fisher information tensorizes when the information in an $n$-observation model is the sum of the information contributions from its observations. For identically distributed observations this meansIndependence and the mean-zero score identity make the cross terms vanish.

Forwhere the ${\epsilon }_i$ are independent $N(0,1)$ and $0\le \gamma <1$,The information tensorizes exactly when $\gamma =0$.

Fisher scoring replaces the observed negative Hessian in Newton iteration by the Fisher information:

For $Y=n^{-1}\mathrm{Bin}(n,p)$, the score and information areConsequently one Fisher-scoring step sends every interior starting value directly to the MLE $\hat{p}=Y$.

Under standard regularity conditions, $I_n(\theta )=-{\mathbb{E}}_{\theta }[{\nabla }_{\theta }^2{\ell }_n(\theta )]$.

For $N(\mu ,v)$, the one-observation score is $((x-\mu )/v,-1/(2v)+(x-\mu {)}^2/(2v^2))$ and the information matrix is $\mathrm{diag}(v^{-1},(2v^2{)}^{-1})$.

A regular statistical model is locally asymptotically normal at ${\theta }_0$ when its log-likelihood ratio under local shifts ${\theta }_0+h/\sqrt{n}$ has the expansion

Under local asymptotic normality, a fixed $h/\sqrt{n}$ shift has a likelihood ratio limit $\exp (W-v/2)$ with $W\sim N(0,v)$. This is positive with mean one. Le Cam first lemma gives mutual contiguity of the product laws, so consistency at the central parameter transfers to these local alternatives.

A Gaussian shift model observes $X\sim N(h,\Sigma )$ with the shift $h$ as parameter. It is the limiting experiment of a locally asymptotically normal model when $\Sigma =I({\theta }_0{)}^{-1}$.