 Horvitz-Thompson estimator

The Horvitz-Thompson estimator estimates a finite population total by expanding each observed unit's contribution by the reciprocal of its inclusion probability. D. G. Horvitz and D. J. Thompson introduced the general unequal-probability construction in 1952.^[ref]

Let $y_1,\dots ,y_N$ be fixed values attached to labelled population units. A sampling design selects a random subset $S$, with known inclusion probabilities ${\pi }_i=\mathbb{P}(i\in S)>0$. Thenwhere $I_i$ is the inclusion indicator. This is inverse probability weighting for a finite population.^[ref] Its expected value is immediatelyThe subscript $p$ emphasizes design-based inference: the fixed values are not assumed to follow a probability model; the sample-selection mechanism supplies the randomness. The inclusion indicators need not be independent. Dividing by known $N$ gives an unbiased estimator of the finite population mean.

For an original exact example, use values $(y_1,y_2,y_3)=(2,4,6)$ and choose samples $\{1,2\}$, $\{1,3\}$, $\{2,3\}$ with probabilities $1/2,1/3,1/6$, respectively. The inclusion probabilities are $(5/6,2/3,1/2)$. The corresponding estimates of the total are $8.4,14.4,18$, andSimply multiplying each sample mean by three instead gives totals $9,12,15$, with expected value $11$. Equal treatment of the observed values ignores the unequal chances of observing them.

The sampling variance depends on pair information. Put ${\pi }_{ij}=\mathbb{P}(i,j\in S)$, including ${\pi }_{ii}={\pi }_i$. Bilinearity of covariance givesFor the three-unit design it is $14.4$, found directly from $\frac{1}{2}(8.4-12{)}^2+\frac{1}{3}(14.4-12{)}^2+\frac{1}{6}(18-12{)}^2$. Thus an unbiased estimator need not give a nearly correct total in each sample.

With pairwise inclusion positivity, the same expansion device applied to pairs constructs the Horvitz-Thompson variance estimator:Its expected value equals the preceding population sum because ${\mathbb{E}}_p(I_i{I}_j/{\pi }_{ij})=1$. In the worked design its three possible values are $9.76,58.56,-60$, whose probability-weighted average is $14.4$. This particular unbiased estimator of variance can therefore be negative for an individual sample. Its expected value identity is a statement about repeated selection.

If ${\pi }_i=0$, an arbitrary unobserved $y_i$ cannot contribute to an unbiased total estimate. If a distinct pair has ${\pi }_{ij}=0$, its product cannot be recovered by the pair-expansion argument. Finally, dividing the weighted sum by the random sum of its inverse-probability weights changes the estimator into a ratio; the indicator proof above no longer establishes that this ratio is an unbiased estimator. The known population size and the actual sampling design are part of the result.