Inverse-variance weighting

Inverse-variance weighting combines estimates of the same quantity by giving a more precise estimate a larger weight. Suppose $X_i$ are independent, unbiased estimators of $\theta$, with known variances $0<{\sigma }_i^2<\infty$. For normalized weights,The inverse is taken after squaring the standard uncertainty: twice the standard uncertainty gives one quarter as much weight. The corresponding weighted least squares criterion is ${\sum }_i(X_i-\theta {)}^2/{\sigma }_i^2$; differentiating it with respect to $\theta$ gives the same weighted arithmetic mean.^[ref]

The optimality statement is precise. Any linear unbiased estimator ${\sum }_i{a}_i{X}_i$ requires ${\sum }_i{a}_i=1$. By the Cauchy-Schwarz inequality,Equality holds for the displayed weights. Thus they minimize variance among linear unbiased combinations under these assumptions; a normal distribution is not needed for that proof.

For an original example, combine observed values 10 and 14 with standard uncertainties 1 and 2 in the same units. The normalized weights are $0.8$ and $0.2$, giving estimate $10.8$, variance $0.8$ and standard uncertainty approximately $0.894$. Equal weights give 12 and standard uncertainty $\sqrt{1.25}\approx 1.118$.

Correlated errors change the problem. A contribution common to every reading survives every weighting whose coefficients sum to one; precision limits of weighting correlated repeats derives the resulting floor. More generally, use the full covariance matrix and generalized least squares rather than treating correlated readings as independent.^[ref] Uncertainty in estimated inverse-variance weights is another limitation: variances estimated from the same data are not fixed known quantities, and an accidentally tiny estimated variance does not justify arbitrary confidence.

Uncertainty in estimated inverse-variance weights arises when the variances used in inverse-variance weighting are themselves estimated. A very small estimated variance can give one observation or group a large random weight. The familiar variance formula for fixed, known weights cannot simply be reused with estimated variances treated as exact. NIST specifically warns that weights estimated from few replicates can substantially and unpredictably affect a weighted least squares analysis.^[ref]

Suppose independent groups contain $n_i$ independent normal random variables with common mean $\theta$ and group variance ${\sigma }_i^2$. Let ${\overline{X}}_i$ and $S_i^2$ be the group sample mean and unbiased sample variance, and formFor normal samples, each sample mean is independent of its sample variance. Conditional on the estimated weights, the estimator remains unbiased and hasThus the law of total variance gives unconditional variance equal to the expectation of this expression. The true ${\sigma }_i^2$ still appear; replacing them by the same random $S_i^2$ does not establish the actual conditional variance. Outside the independence assumptions, even conditional unbiasedness can fail.

An original two-group example takes $n_1=n_2=5$ and true ${\sigma }_1^2={\sigma }_2^2=4$, so each group mean has variance $0.8$. If observed sample variances are $0.25$ and $4$, the estimated normalized weights are $16/17$ and $1/17$. For group means $10$ and $14$, the combined estimate is $174/17\approx 10.2353$. The plug-in variance isbut the true conditional variance under the stipulated model isThe corresponding standard deviations are about $0.2169$ and $0.8435$. Equal weights would have variance $0.4$. An accidentally small estimated variance has both overstated precision and moved the combination away from the optimal equal weights in this example.

Reciprocal moments of an estimated variance explain the unstable precision estimate. For a normal sample with $\nu =n-1$, $Z=\nu S^2/{\sigma }^2$ has a chi-squared distribution. Integrating its density givesConsequently $\mathbb{E}[1/S^2]=\nu /[(\nu -2){\sigma }^2]$ for $\nu >2$, while the second moment requires $\nu >4$. The density and its gamma-distribution parameterization are documented in the primary SciPy implementation notes.^[ref] With $n=5$, the inverse sample variance has a finite mean twice the true precision and an infinite variance; the normalized weights remain bounded, so this does not imply their variance is infinite.

More replication or a justified smooth variance model can stabilize weights. NIST describes direct replicate-based weight estimation and variance-function modeling as distinct approaches.^[ref] A proposed validation repeats the complete weight-estimation and combination procedure on simulated or replicated groups, checking actual interval coverage as well as point estimates. A bootstrap must reproduce the weight-estimation stage; holding the fitted weights fixed tests a different procedure.

Reciprocal moments of an estimated variance determine whether a precision estimate such as $1/S^2$ has a finite expected value or variance. Small positive values of a sample variance become large values after inversion. An unbiased estimator of a variance therefore need not give an unbiased or even integrable estimate of its reciprocal.

For $n\ge 2$, let $X_1,\dots ,X_n$ be independent normal random variables with common mean $\mu$ and variance ${\sigma }^2>0$. For the unbiased sample variance $S^2=(n-1{)}^{-1}{\sum }_i(X_i-\overline{X}{)}^2$, put $\nu =n-1$. Then $Z=\nu S^2/{\sigma }^2$ has a chi-squared distribution. Its probability density function is $z^{\nu /2-1}{e}^{-z/2}/[2^{\nu /2}\Gamma (\nu /2)]$ for $z>0$.^[ref]

For any real $p>0$, direct integration and the Gamma function giveThe convergence restriction is essential: near zero the integrand has exponent $\nu /2-p-1$, so the integral diverges when $\nu \le 2p$. Substitution of a negative Gamma function argument into the displayed quotient does not define a nonexistent moment.

Rescaling gives the useful general resultIn particular, the mean reciprocal exists for $n>3$, while its variance requires $n>5$:For $n=6$ the mean is $5/(3{\sigma }^2)$, the variance is $50/(9{\sigma }^4)$ and the coefficient of variation is $\sqrt{2}$. For $n=4$ the mean is finite but the variance is infinite. For $n=3$ even the mean diverges. Increasing the number of groups does not repair a nonexistent moment of each raw reciprocal estimate.

Inverse-variance weighting commonly normalizes these reciprocals. Those normalized weights lie between zero and one, so their moments can exist even when the raw reciprocal moments fail. This distinction matters in uncertainty in estimated inverse-variance weights: neither the moment calculation nor a plug-in precision formula alone establishes the accuracy of the final weighted estimator. The results here use the exact normal-sample law; a nonnormal sample requires a different analysis.