Fisher information matrix

The Fisher information is $I_n(\theta )=\mathbb{E}[S_n{S}_n^T]$ and equals $n{I}_1(\theta )$ for an independent identically distributed sample.

The Jeffreys prior uses the square root of the determinant of the Fisher information matrix as a parameter-density kernel. It is invariant under smooth one-to-one reparameterization, since the information and density Jacobians transform compatibly. It may be an improper prior; invariance does not guarantee posterior propriety. With nuisance parameters, a scalar conditional Jeffreys prior and the joint Jeffreys prior need not coincide.

For a differentiable scale family with finite positive Fisher information, the score is $-[1+u(\log f{)}^{\prime }(u)]/\sigma$. Its squared expectation is a constant times ${\sigma }^{-2}$, yielding the displayed Jeffreys prior. This is an improper prior on the entire positive half-line.

For independent normal observations with fixed means and variances $v+r_s$, where $v\ge 0$ is unknown and all known $r_s>0$, the scalar Fisher information for $v$ is $I_{vv}=\frac{1}{2}{\sum }_s(v+r_s{)}^{-2}$. Thus its scalar Jeffreys prior is finite at zero and behaves like $1/v$ at infinity. Unlike a log-flat prior $1/v$, it avoids a divergent integral at the zero-variance boundary. After flat-prior elimination of a Gaussian common mean, the integrated likelihood is bounded by a constant times $v^{-(N-1)/2}$, so this prior yields a proper posterior for $N>1$ and a proper prior on any remaining mean-shape parameters. In the homoscedastic case it reduces to $1/(v+r)$, which is log-flat for the total variance rather than for the latent variance alone.

For $X\sim N_p(\theta ,\Sigma )$ with known nonsingular covariance matrix, the score function is ${\Sigma }^{-1}(X-\theta )$ and the per-observation Fisher information matrix is ${\Sigma }^{-1}$. In particular, covariance $I_p$ gives information $I_p$.

The empirical score outer-product matrix isUnder regularity and consistency of $\hat{\theta }$, $i_n(\hat{\theta })$ converges in probability to the Fisher information matrix. This convention differs from the negative-Hessian convention for Observed Fisher information.

The observed Fisher information is the negative Hessian of the realized log-likelihood. Its expectation, under standard regularity conditions, is the Fisher information matrix.

Fisher information tensorizes when the information in an $n$-observation model is the sum of the information contributions from its observations. For identically distributed observations this meansIndependence and the mean-zero score identity make the cross terms vanish.

Forwhere the ${\epsilon }_i$ are independent $N(0,1)$ and $0\le \gamma <1$,The information tensorizes exactly when $\gamma =0$.

Fisher scoring replaces the observed negative Hessian in Newton iteration by the Fisher information:

For $Y=n^{-1}\mathrm{Bin}(n,p)$, the score and information areConsequently one Fisher-scoring step sends every interior starting value directly to the MLE $\hat{p}=Y$.

Under standard regularity conditions, $I_n(\theta )=-{\mathbb{E}}_{\theta }[{\nabla }_{\theta }^2{\ell }_n(\theta )]$.

For $N(\mu ,v)$, the one-observation score is $((x-\mu )/v,-1/(2v)+(x-\mu {)}^2/(2v^2))$ and the information matrix is $\mathrm{diag}(v^{-1},(2v^2{)}^{-1})$.

A regular statistical model is locally asymptotically normal at ${\theta }_0$ when its log-likelihood ratio under local shifts ${\theta }_0+h/\sqrt{n}$ has the expansion

Under local asymptotic normality, a fixed $h/\sqrt{n}$ shift has a likelihood ratio limit $\exp (W-v/2)$ with $W\sim N(0,v)$. This is positive with mean one. Le Cam first lemma gives mutual contiguity of the product laws, so consistency at the central parameter transfers to these local alternatives.

A Gaussian shift model observes $X\sim N(h,\Sigma )$ with the shift $h$ as parameter. It is the limiting experiment of a locally asymptotically normal model when $\Sigma =I({\theta }_0{)}^{-1}$.