Regression toward the mean and randomized comparisons (source code)

= Regression toward the mean and randomized comparisons

For <causal inference>, a before-and-after change among units selected by an extreme baseline reading combines any <causal effect> with the change expected from selection. In the synthetic model the selected-group <mean> falls by about $0.610$ with no <intervention> at all. Calling that entire fall an intervention effect would attribute a predictable <measurement error> contribution to a change in the signal.

One comparison design selects all eligible units by the same rule, then uses <randomization> to assign an <intervention> or a comparison condition, with the same measurement times and instrument in both groups. To see the principle algebraically, let assignment $A\in\{0,1\}$ be independent of $(T,\varepsilon_1,\varepsilon_2)$ within the selected population, and let the intervention add a constant $\delta$ to the second reading: $Y=T+\varepsilon_2+\delta A$. Then
$$
\mathbb E[Y-X\mid X>1,A=1]=-0.610054\ldots+\delta,\qquad
\mathbb E[Y-X\mid X>1,A=0]=-0.610054\ldots.
$$
Subtracting the two expected changes isolates $\delta$ under these assumptions. Both groups experience the same selection tendency in <expectation>. A finite experiment still has sampling uncertainty; <randomization> does not guarantee identical realized groups. Without <randomization>, the same threshold alone does not establish comparable latent signals or error processes. Different selection rules, selective follow-up, changing instruments, or common temporal changes also require explicit assumptions. This is a <statistical inference> principle about comparisons, not evidence that any particular <intervention> works.