Horvitz-Thompson variance estimator (${\hat{V}}_{\mathrm{HT}}$)

The Horvitz-Thompson variance estimator estimates the sampling variance of a Horvitz-Thompson estimator using observed values and known individual and joint inclusion probabilities. It is a random estimate of a fixed design-dependent variance, rather than the variance itself.

Let $I_i$ be an inclusion indicator, ${\pi }_i={\mathbb{E}}_p{I}_i>0$, and ${\pi }_{ij}={\mathbb{E}}_p(I_i{I}_j)$, with ${\pi }_{ii}={\pi }_i$. For fixed population values $y_i$, write $a_i=y_i/{\pi }_i$. Bilinearity of covariance givesUnder pairwise inclusion positivity, a sample $S$ provides the estimatorThe diagonal terms are included. Taking expected values replaces $I_i{I}_j/{\pi }_{ij}$ by 1, proving ${\mathbb{E}}_p{\hat{V}}_{\mathrm{HT}}=V_p$. This is the pairwise counterpart of inverse-probability expansion for the finite population total.^[ref]

An original exact example uses fixed values $(3,0,1)$. Select $\{1,2\},\{1,3\},\{2,3\}$ with probabilities $1/2,1/4,1/4$, respectively. Then $({\pi }_1,{\pi }_2,{\pi }_3)=(3/4,3/4,1/2)$, and the possible total estimates are $4,6,2$. Their expected value is the true total 4, with varianceFor those same samples, the variance estimates are $4,-2,2$. Their expected value is also 2. In particular, for $S=\{1,3\}$, the two diagonal contributions are 4 and 2, while the combined off-diagonal contribution is $-8$, giving $-2$.

Thus this unbiased estimator of variance can be negative in a realized sample. Replacing a negative result by zero changes its expected value; in this example that modification raises the mean estimate to $5/2$. A positive square root is therefore not always available as a sample standard error from this particular formula. If some ${\pi }_{ij}=0$, dropping an unobservable cross-product does not preserve the proof; the missing term needs separate analysis.

The single-draw variance identifiability obstruction is the inability, for an unrestricted finite population, to construct a universally design-unbiased estimate of a total estimator's variance from just one observed unit. The estimator can estimate the total unbiasedly while the required cross-products for estimating its sampling variance are never jointly observed. This is an obstruction to the stated one-draw estimation problem, not a claim that a parameter can never be identified from repeated independent samples.

An original two-unit proof makes the distinction exact. Sample label $I=1$ or2 with equal probability and observe only $y_I$. The Horvitz-Thompson estimator of the finite population total is $\hat{T}=2y_I$. Its expectation is $y_1+y_2$, and its design variance isSuppose a proposed variance estimate returns $g_1(y_1)$ after observing unit1 and $g_2(y_2)$ after observing unit2. Unbiasedness for every pair would requireThe left side is additive in the two unknown values, while the right side contains an unobserved product.

To prove impossibility without smoothness assumptions, evaluate both sides at $(y_1,y_2)=(0,0),(1,0),(0,1),(1,1)$. Every additive function $H$ has rectangular contrast $H(1,1)-H(1,0)-H(0,1)+H(0,0)=0$. The required variance has contrast $0-1-1+0=-2$. The contradiction rules out such an unbiased estimator. Independent auxiliary randomization does not help: averaging it out merely replaces each $g_i$ by another function of the same single observed value.

Here the individual inclusion probabilities are positive, but the joint inclusion probability for distinct units is zero. The usual Horvitz-Thompson variance estimator needs pairwise inclusion positivity to turn sample cross-products into unbiased population cross-product estimates. The primary analysis of zero pairwise probabilities identifies precisely these missing terms.^[ref]

Extra information can change the problem: observing suitable pairs, imposing a justified population relation, or accepting a model-based or conservative estimate uses assumptions absent from the proof. Observing one value also does not prevent writing a biased variance estimate. What fails is a single-draw unbiased estimator valid for every unrestricted population, even though the total estimator itself remains unbiased.