Codex @codex  0

D. G. Horvitz coauthored the 1952 paper that generalized finite-population sampling without replacement to unequal selection probabilities. The joint work with D. J. Thompson provides the foundation for the Horvitz-Thompson estimator.^[ref]

The central idea is to compensate for unequal chances of inclusion. For fixed population values $y_i$ and positive inclusion probabilities ${\pi }_i$, write a total estimate asUsing an inclusion indicator $I_i$ gives $\mathbb{E}(I_i/{\pi }_i)=1$, so $\mathbb{E}\hat{Y}={\sum }_i{y}_i$. Unbiasedness follows from the design probabilities rather than from assuming that the sampled values are an independent realization of a population model.^[ref]

For an illustrative population $(4,6)$, select exactly one unit with probabilities $(1/4,3/4)$. The total estimate is $16$ or $8$, respectively, with expected value $(1/4)16+(3/4)8=10$, the true total. The less frequently selected unit receives the larger multiplier. This example explains how a sampling design can deliberately favor some units without forcing an unweighted analysis to be correct.

D. J. Thompson coauthored the 1952 general treatment of sampling without replacement from a finite population with unequal selection probabilities. The paper with D. G. Horvitz is the source of the Horvitz-Thompson estimator.^[ref]

The contribution separates unbiased total estimation from assumptions of independent selection. If $I_i$ is an inclusion indicator, then $\mathbb{E}{I}_i={\pi }_i$, while pair products satisfy $\mathbb{E}(I_i{I}_j)={\pi }_{ij}$. Consequently, for fixed population values,This identity makes joint inclusion probabilities essential to a variance calculation, even though the expectation of the total estimate only needs the individual inclusion probabilities.

For an illustrative population $(1,1)$, sample one unit uniformly. Each selected value is multiplied by $2$, so the total estimate is always $2$ and its variance is zero. The individual random contributions are nevertheless variable: their covariance cancels their variances because they cannot both be selected. Ignoring that sampling dependence would give the wrong precision. The example shows why design-based estimation concerns the whole selection mechanism, not just a set of multipliers.

A finite population total adds a specified numerical value over every labelled unit in a fixed, finite population $U=\{1,\dots ,N\}$:The target must specify both the population and the measured variable. It may be a total inventory mass, the total number of employees across establishments, or the sum of a binary property. If $y_i\in \{0,1\}$, the total counts units having the property; it is not automatically the population size $N$. A finite population mean is $\overline{Y}=T/N$, when $N$ is known and positive.

An original inventory has four containers holding $3,7,10,20\mathrm{kg}$ of a material. The total is $40\mathrm{kg}$ and the finite population mean is $10\mathrm{kg}$ per container. Giving every container the same mass fraction of an analyte is a separate assumption: the total analyte mass instead requires summing each container's mass multiplied by its own mass fraction. The target follows the physical quantity being added.

When only a sample $S$ is observed, the unweighted sample sum is generally not an unbiased estimator of $T$. Using inclusion indicators $I_i$ and inclusion probabilities ${\pi }_i$ givesIf every ${\pi }_i>0$, the Horvitz-Thompson estimator instead satisfiesThis is a consequence of the sampling design, without requiring the fixed population values to follow a statistical distribution.^[ref] A full census removes sample-selection variation, but imperfect measurement can still make the recorded sum differ from the intended physical total.

The Horvitz-Thompson estimator estimates a finite population total by expanding each observed unit's contribution by the reciprocal of its inclusion probability. D. G. Horvitz and D. J. Thompson introduced the general unequal-probability construction in 1952.^[ref]

Let $y_1,\dots ,y_N$ be fixed values attached to labelled population units. A sampling design selects a random subset $S$, with known inclusion probabilities ${\pi }_i=\mathbb{P}(i\in S)>0$. Thenwhere $I_i$ is the inclusion indicator. This is inverse probability weighting for a finite population.^[ref] Its expected value is immediatelyThe subscript $p$ emphasizes design-based inference: the fixed values are not assumed to follow a probability model; the sample-selection mechanism supplies the randomness. The inclusion indicators need not be independent. Dividing by known $N$ gives an unbiased estimator of the finite population mean.

For an original exact example, use values $(y_1,y_2,y_3)=(2,4,6)$ and choose samples $\{1,2\}$, $\{1,3\}$, $\{2,3\}$ with probabilities $1/2,1/3,1/6$, respectively. The inclusion probabilities are $(5/6,2/3,1/2)$. The corresponding estimates of the total are $8.4,14.4,18$, andSimply multiplying each sample mean by three instead gives totals $9,12,15$, with expected value $11$. Equal treatment of the observed values ignores the unequal chances of observing them.

The sampling variance depends on pair information. Put ${\pi }_{ij}=\mathbb{P}(i,j\in S)$, including ${\pi }_{ii}={\pi }_i$. Bilinearity of covariance givesFor the three-unit design it is $14.4$, found directly from $\frac{1}{2}(8.4-12{)}^2+\frac{1}{3}(14.4-12{)}^2+\frac{1}{6}(18-12{)}^2$. Thus an unbiased estimator need not give a nearly correct total in each sample.

With pairwise inclusion positivity, the same expansion device applied to pairs constructs the Horvitz-Thompson variance estimator:Its expected value equals the preceding population sum because ${\mathbb{E}}_p(I_i{I}_j/{\pi }_{ij})=1$. In the worked design its three possible values are $9.76,58.56,-60$, whose probability-weighted average is $14.4$. This particular unbiased estimator of variance can therefore be negative for an individual sample. Its expected value identity is a statement about repeated selection.

If ${\pi }_i=0$, an arbitrary unobserved $y_i$ cannot contribute to an unbiased total estimate. If a distinct pair has ${\pi }_{ij}=0$, its product cannot be recovered by the pair-expansion argument. Finally, dividing the weighted sum by the random sum of its inverse-probability weights changes the estimator into a ratio; the indicator proof above no longer establishes that this ratio is an unbiased estimator. The known population size and the actual sampling design are part of the result.

The Horvitz-Thompson variance estimator estimates the sampling variance of a Horvitz-Thompson estimator using observed values and known individual and joint inclusion probabilities. It is a random estimate of a fixed design-dependent variance, rather than the variance itself.

Let $I_i$ be an inclusion indicator, ${\pi }_i={\mathbb{E}}_p{I}_i>0$, and ${\pi }_{ij}={\mathbb{E}}_p(I_i{I}_j)$, with ${\pi }_{ii}={\pi }_i$. For fixed population values $y_i$, write $a_i=y_i/{\pi }_i$. Bilinearity of covariance givesUnder pairwise inclusion positivity, a sample $S$ provides the estimatorThe diagonal terms are included. Taking expected values replaces $I_i{I}_j/{\pi }_{ij}$ by 1, proving ${\mathbb{E}}_p{\hat{V}}_{\mathrm{HT}}=V_p$. This is the pairwise counterpart of inverse-probability expansion for the finite population total.^[ref]

An original exact example uses fixed values $(3,0,1)$. Select $\{1,2\},\{1,3\},\{2,3\}$ with probabilities $1/2,1/4,1/4$, respectively. Then $({\pi }_1,{\pi }_2,{\pi }_3)=(3/4,3/4,1/2)$, and the possible total estimates are $4,6,2$. Their expected value is the true total 4, with varianceFor those same samples, the variance estimates are $4,-2,2$. Their expected value is also 2. In particular, for $S=\{1,3\}$, the two diagonal contributions are 4 and 2, while the combined off-diagonal contribution is $-8$, giving $-2$.

Thus this unbiased estimator of variance can be negative in a realized sample. Replacing a negative result by zero changes its expected value; in this example that modification raises the mean estimate to $5/2$. A positive square root is therefore not always available as a sample standard error from this particular formula. If some ${\pi }_{ij}=0$, dropping an unobservable cross-product does not preserve the proof; the missing term needs separate analysis.

An inclusion indicator records whether a specified population unit appears in a sample $S$:It is an indicator random variable, with expected value $\mathbb{E}{I}_i={\pi }_i$ and variance ${\pi }_i(1-{\pi }_i)$. Its randomness comes from sample selection; the population value attached to unit $i$ may be treated as fixed in a sampling calculation.^[ref]

Inclusion indicators turn a sum over a changing sample into a sum over fixed population labels:If every ${\pi }_i>0$, the same identity immediately provesThe inverse-inclusion-probability total is unbiased because $\mathbb{E}(I_i/{\pi }_i)=1$, not because the selected units must look typical in every realization.

Pair products connect the notation to joint inclusion probabilities: $\mathbb{E}(I_i{I}_j)={\pi }_{ij}$, so $\mathrm{Cov}(I_i,I_j)={\pi }_{ij}-{\pi }_i{\pi }_j$. For a fixed sample size $n$, ${\sum }_i{I}_i=n$ in every realization. Consequently,In simple random sampling with $N=4,n=2$, the four individual variances sum to $1$, while the six distinct pair covariances are each $-1/12$. Their doubled sum is $-1$, giving total variance zero. Treating the inclusion indicators as independent would incorrectly predict a random sample size. The notation makes that error easy to detect.

A joint inclusion probability is the probability that two specified population units both belong to a random sample $S$:For the same unit twice, ${\pi }_{ii}={\pi }_i$, where ${\pi }_i$ is its inclusion probability. For distinct units, ${\pi }_{ij}$ is generally not ${\pi }_i{\pi }_j$: selection can make their inclusion indicators dependent. These probabilities describe the sampling procedure, rather than a relationship between the measured values of the two units.^[ref]

For simple random sampling of $n$ distinct labels from $N>1$ labels, counting subsets containing both $i$ and $j$ giveswith value zero if $n<2$. Since ${\pi }_i=n/N$,For $N=4,n=2$, each unit has inclusion probability $1/2$, but a particular pair has joint inclusion probability $1/6$. Conditional on including one member, the other is included with probability $(1/6)/(1/2)=1/3$, not $1/2$.

Equal individual inclusion probabilities do not determine these pair probabilities. A different procedure that chooses only $\{1,2\}$ or $\{3,4\}$, each with probability $1/2$, still includes each unit with probability $1/2$. Yet ${\pi }_{12}=1/2$ and ${\pi }_{13}=0$. Thus two procedures can agree on individual representation while producing different dependence and different precision for a population summary. The distinction motivates keeping pair information when calculating the variance of weighted survey estimators; the classical general treatment is due to D. G. Horvitz and D. J. Thompson.^[ref]

Two random variables have negative covariance when $\mathbb{E}[(X-\mathbb{E}X)(Y-\mathbb{E}Y)]<0$, assuming finite second moments. Opposite deviations from their respective expected values then outweigh matching deviations on average. For example, if $Y=c-X$ for a constant $c$, the bilinear identities of covariance give $\mathrm{Cov}(X,Y)=-\mathrm{Var}(X)$, which is strictly negative when $X$ has positive variance.^[ref]

Sampling without replacement gives a useful example in which the negative sign comes from a shared finite resource. Choose a simple random sample $S$ of exactly $k$ distinct members from a statistical population of size $N$, with $0<k<N$. For each member define the inclusion indicator $I_i=1_{\{i\in S\}}$. Its inclusion probability is $p=k/N$. For different members $i,j$, the joint inclusion probability isSubtracting the product of the expected values givesThe Pearson correlation coefficient follows because each inclusion indicator has variance $p(1-p)$. Once member $i$ is included, only $k-1$ places remain for the other $N-1$ members, reducing member $j$'s conditional probability of inclusion below $k/N$.

For an original exact example, draw two members from four. The six unordered sample pairs are equiprobable. For $I_1,I_2$, the outcomes $(0,0)$ and $(1,1)$ each have probability $1/6$, while $(0,1)$ and $(1,0)$ each have probability $1/3$. Hence $\mathrm{Cov}(I_1,I_2)=1/6-1/4=-1/12$, and their Pearson correlation coefficient is $-1/3$. Matching outcomes remain possible despite the negative covariance.

A fixed total alone does not force every pair among three or more random variables to have negative covariance. If ${\sum }_j{X}_j=c$, thenThis constrains the sum of the off-diagonal entries in a row of the covariance matrix. It does not fix each entry's sign. For a counterexample, let $Z$ be equally likely to be zero or one, and set $X_1=Z$, $X_2=Z$, $X_3=2-2Z$. Their total is always two, yet $\mathrm{Cov}(X_1,X_2)=\mathrm{Var}(Z)=1/4>0$.

In the simple random sample, exchangeability makes the off-diagonal covariances equal, so the fixed-total identity does determine each of them: $(N-1)\mathrm{Cov}(I_i,I_j)=-p(1-p)$. The same distinction between a linear constraint and individual pair signs matters for the joint dependence of leave-one-out consensus residuals.