Probability sampling

Probability sampling chooses observational units by a specified random mechanism with known inclusion probabilities. This separates design-based representativeness from the convenience of obtaining particular observations. Simple random sampling, stratified sampling and cluster sampling are different designs with different precision and cost properties.

Sampling with replacement returns each selected label to the available population before the next draw. Independent uniform draws from a fixed population are independent and identically distributed; a label can recur. Replacement by itself does not imply uniformity or independence.

An inclusion probability is the probability that a particular unit belongs to the selected sample. For sample size $n$ in simple random sampling from $N$ labels it is $n/N$. It differs from the selection probability at a particular draw position.

Unequal probability sampling assigns different inclusion probabilities to different population units. Estimating an unweighted population summary generally requires accounting for these differences in the estimator.

The finite population mean for $N\ge 1$ is the arithmetic mean of all fixed values in the target population. Under simple random sampling, the sample mean is an unbiased estimator of it.

For $N>1$, the finite population variance measures spread about the finite population mean. Survey-sampling formulas commonly use denominator $N-1$; the denominator-$N$ version is ${\sigma }_N^2=(N-1)S^2/N$. The normalization must be specified when using a finite population correction.

Sampling without replacement selects distinct labelled units: after a unit is selected it is unavailable for the remaining draws. Distinct labels can carry equal numerical values, so repeated values do not necessarily imply a repeated unit. The rule alone does not specify selection probabilities; a procedure favoring particular units is still sampling without replacement but need not be simple random sampling.

In uniform sequential sampling without replacement, the next label is chosen uniformly among those remaining. For a population of $N$ labels, any ordered sequence of $n$ distinct labels has probability $1/[N(N-1)\cdots (N-n+1)]$. Each size-$n$ subset has $n!$ possible orders, so its probability is ${\left(\genfrac{}{}{0ex}{}{N}{n}\right)}^{-1}$. This gives the fixed-size simple random sampling design. Equivalently, take the first $n$ labels of a uniformly random permutation. Restricting a uniformly random permutation to a specified label set preserves uniformity; biased modulo mappings should not replace uniform selection.

The draw positions have identical marginal probability distributions, but exhibit statistical dependence. If the first label is $i$, the conditional probability that the second label is $i$ is zero, whereas its marginal probability is $1/N$. Thus these draws are not independent and identically distributed when $N>1$ and $n>1$. For a marked subset of labels, counting size-$n$ subsets with binomial coefficients gives the hypergeometric distribution. The resulting negative covariance explains the finite population correction for both counts and sample means.

Take a uniformly random permutation of $N>1$ distinguishable objects, of which $K$ are marked. Let $B_i$ mean that position $i$ is marked. Every object is equally likely at a specified position, so $\mathbb{P}(B_i)=K/N$. For distinct positions $i,j$, their ordered pair is uniform among $N(N-1)$ distinct-object pairs, of which $K(K-1)$ are marked pairs. This proves the second formula. For $K>0$, the conditional probability is $\mathbb{P}(B_i\mid B_j)=(K-1)/(N-1)$.

For the count $M_m$ of marked objects among the first $m$ positions, indicator random variables and linearity of expectation give $\mathbb{E}[M_m]=mK/N$. These draws are exchangeable but generally dependent; equal marginal probabilities do not imply independence.

Draw balls sequentially without replacement from $g$ green and $r$ red balls. Let $T$ be the first positive draw count at which the numbers of drawn green and red balls agree. For $1\le m\le \min (g,r)$,where $(q{)}_j=q(q-1)\cdots (q-j+1)$ denotes the falling factorial and $C_k$ is a Catalan number. Odd first return times are impossible because equality requires an even total number of draws.

To count the successful colour orders, encode green by a step $+1$ and red by a step $-1$. If the first step is positive, a first return time $2m$ means that the partial sums remain strictly positive until the last step. Delete the first positive step and the last negative step, and shift the remaining heights down by one. The result is a Dyck path of length $2m-2$, of which there are $C_{m-1}$. Interchanging the colours doubles this count. Every such colour order has the same probability under sampling without replacement, namely $(g{)}_m(r{)}_m/(g+r{)}_{2m}$, proving the formula. In particular, the numbers of successful colour orders at lengths two, four and six are two, two and four; balanced orders with an earlier return must not be counted.

When $m\ge 1$ items among $N$ are marked, inspect a uniformly random permutation until the first marked item. For its first position to be $k$, the other $m-1$ marked positions must lie among the last $N-k$ slots. This yields the displayed law with support $1\le k\le N-m+1$. It is the first-success form of sampling without replacement. For one marked item the position is uniform, while its conditional success chance after $r$ failures increases to $1/(N-r)$.

Cluster sampling chooses groups of observational units, such as production days, rather than dispersed individual units. Testing every unit in a selected group is a one-stage cluster sample. Positive within-cluster intraclass correlation reduces the information supplied by a fixed number of sampled units, although collecting clustered observations can be cheaper.

The design effect is a sampling variance under a complex design divided by the corresponding simple-random-sampling variance. For independent equal-size clusters of size $m$ with exchangeable intraclass correlation $\rho$, a common mean-estimation approximation is $1+(m-1)\rho$. It expresses the loss of effective sample size from positive correlation; finite-population and unequal-cluster-size designs require their own calculation.

Stratified sampling partitions a population into strata and samples separately within each. It guarantees specified representation and can improve precision when the strata describe population variation. Equal sampling fractions give equal unit inclusion probabilities; unequal fractions require appropriate weighting for population summaries.

Suppose the variance is ${\sum }_i{v}_i/x_i$ and the sampling cost is ${\sum }_i{a}_i{x}_i\le b$, with positive costs, variance coefficients and budget. The Cauchy-Schwarz inequality gives $A^2\le ({\sum }_i{v}_i/x_i)({\sum }_i{a}_i{x}_i)\le b{\sum }_i{v}_i/x_i$. Equality forces $x_i$ proportional to $\sqrt{v_i/a_i}$ and a tight budget, proving the displayed unique optimum. The minimal variance is $A^2/b$. For the optimization Lagrangian $f-\lambda ({\sum }_i{a}_i{x}_i-b)$, the multiplier and budget derivative both equal $-A^2/b^2$. Therefore a small resource change $\delta b$ changes the optimum by $\lambda \delta b+O((\delta b{)}^2)$. This is a continuous allocation; integer sample-size restrictions define a separate optimization problem.

Here simple random sampling means a random unordered subset $A$ of exactly $n$ distinct labels from $\{1,\dots ,N\}$, with every size-$n$ subset equally likely. The sample size is fixed in advance, $1\le n\le N$, and sampling without replacement prevents duplicate units. This is SRSWOR; some texts also use "simple random sampling" for independent uniform draws with replacement, which is a different design.^[ref]

Every label has inclusion probability $n/N$, but equal inclusion probabilities alone are insufficient. For example, from six labels choose either $\{1,2,3\}$ or $\{4,5,6\}$ with probability $1/2$. Each label is included with probability $1/2$, yet eighteen of the twenty size-three subsets are impossible: this is a two-group cluster sampling design. Unequal probability sampling deliberately changes inclusion probabilities, often to improve precision for a particular estimand, and generally requires an appropriately weighted estimator rather than the plain sample mean.

Consider the original mathematical population with labelled values $(2,4,6,8,10,12)$, and let $n=2$. The fifteen equiprobable subsets, followed by their sample means, areTheir sample means sum to $105$, giving expected value $105/15=7$. Their squared deviations from $7$ sum to $70$, giving variance $70/15=14/3$. This is an enumeration over hypothetical selections from six stipulated numbers, not measured survey data.

With independent uniform sampling with replacement, the $6^2=36$ ordered outcomes each have probability $1/36$. The two orders $(2,4)$ and $(4,2)$ are distinct outcomes, and $(2,2)$ is now possible. Forgetting order does not make the resulting unordered multisets equiprobable: $\{2,4\}$ has probability $2/36$ but $\{2,2\}$ has probability $1/36$. The thirty-six sample means still have expected value $7$, but their squared deviations sum to $210$, giving variance $210/36=35/6$. Avoiding duplicate labels improves precision for this fixed population.

Fix $N>1$ real population values $y_1,\dots ,y_N$. Define the finite population mean and two normalizations of the finite population variance byThe values are fixed; only the selected subset $A$ is random. For its sample mean ${\bar{y}}_A=n^{-1}{\sum }_{i\in A}{y}_i$, the exact design expectation and variance areThe sample mean is an unbiased estimator of the finite population mean. At $n=N$, every unit is observed and ${\bar{y}}_A=\bar{Y}$ exactly, so the variance is zero. The $N=1$ case is deterministic and is handled directly, since $S^2$ is undefined then.^[ref]

The finite population correction depends on the baseline. With the $N-1$ normalization it is $1-n/N$ multiplying $S^2/n$. With the $N$ normalization it is $(N-n)/(N-1)$ multiplying ${\sigma }_N^2/n$, the exact variance for independent uniform sampling with replacement. The standard error uses the square root of the relevant variance factor. Mixing the two normalizations gives a wrong answer even at $n=1$, when replacement cannot matter. For the six-value example, $\bar{Y}=7$, ${\sum }_i(y_i-7{)}^2=70$, $S^2=14$ and ${\sigma }_N^2=35/3$, reproducing $14/3$ without replacement and $35/6$ with replacement.

A short proof exposes the statistical dependence. Let $X_r$ be the value at position $r$ in uniform sequential sampling without replacement, so each $X_r$ has expected value $\bar{Y}$ and variance ${\sigma }_N^2$. Write $d_i=y_i-\bar{Y}$. Every ordered pair of distinct labels is equally likely, and ${\sum }_i{d}_i=0$, hence for $r\ne s$,Linearity of expectation proves unbiasedness without requiring independence. The variance of a sum includes all $n(n-1)$ ordered covariance terms, givingFor the original population, each pair of distinct draw positions has covariance $-7/3$. Drawing a high value leaves a lower mean among the remaining values, balancing subsequent draws. Equal marginal probability distributions therefore coexist with negative covariance.

When $2\le n\le N$, the sample variance $s_A^2=(n-1{)}^{-1}{\sum }_{i\in A}(y_i-{\bar{y}}_A{)}^2$ is an unbiased estimator of $S^2$. To check this independently, use ${\sum }_{i\in A}(y_i-{\bar{y}}_A{)}^2={\sum }_{i\in A}(y_i-\bar{Y}{)}^2-n({\bar{y}}_A-\bar{Y}{)}^2$: its expected value is $n{\sigma }_N^2-n\mathrm{Var}({\bar{y}}_A)=(n-1)S^2$. Thus $(1-n/N)s_A^2/n$ unbiasedly estimates the variance of the sample mean; its square root estimates the standard error, but need not be unbiased.^[ref]

The diagram compares variance with the expected fraction of distinct population labels observed. Without replacement this fraction is exactly $n/N$; with independent uniform sampling with replacement, a label is missed with probability $(1-1/N{)}^n$, so linearity of expectation gives expected fraction $1-(1-1/N{)}^n$. At six draws, replacement covers only about $66.5\%$ of labels on average and retains sample mean variance $35/18$; sampling all six distinct labels covers the population completely and removes selection uncertainty. This population coverage is distinct from confidence interval coverage probability.

The correction matters when the sampling fraction is appreciable. It calibrates uncertainty about this fixed population under the specified selection mechanism. It does not correct unequal selection weights, missing observations or measurement errors; observing every label also does not eliminate uncertainty about a larger unobserved population. Moreover, an exact variance alone does not guarantee the coverage probability of a proposed confidence interval: a small discrete sampling distribution can make a normal approximation inaccurate.

The sampling fraction is the sample size divided by the finite population size. In simple random sampling, it determines how much the finite population correction reduces the variance relative to $S^2/n$.