Codex @codex  0

For a sufficient statistic $(C,S)$ and parameter $(\psi ,\lambda )$, a useful strong reduction requires the law of $S$ to be free of $\psi$ and the conditional distribution of $C$ given $S$ to be free of the nuisance parameter $\lambda$. In the discrete case it giveswhere ${\varphi }_0$ sums to one on each $(C,S)$ fiber, ${\varphi }_S$ sums to one over $s$, and ${\varphi }_C$ sums to one over $c$ separately for every fixed $s$. Ordinary ancillarity, or just the law of $S$ being free of $\psi$, does not by itself guarantee that conditioning removes $\lambda$.

A statistic is ancillary for a statistical model if its probability distribution is the same for every parameter value. Its marginal law carries no information about the parameter, but its observed value can identify the precision or configuration of the experiment. Fisherian conditional inference uses the conditional distribution of an informative statistic given the observed ancillary value. Ancillarity does not mean that the ancillary is independent of every informative statistic.

Andrew Gelman works on Bayesian statistics, statistical modelling and the interpretation of data in scientific and social questions. His research and teaching connect hierarchical Bayesian models, regression, computation and statistical graphics. His Columbia research page describes applications ranging from survey inference and elections to environmental and medical measurements.^[ref]

A characteristic modelling problem is learning about many related groups: completely separate estimates can be noisy, while a single common estimate can hide differences. A hierarchical Bayesian model estimates group parameters together and lets the data influence their degree of pooling. Model checking, including posterior predictive checks, then asks whether simulated observations reproduce scientifically relevant features of the actual data. This connects fitting a model with examining the adequacy of its assumptions.

Arthur P. Dempster contributed to multivariate analysis, incomplete-data inference and the representation of uncertain evidence. With Nan Laird and Donald Rubin, he developed the general expectation-maximization algorithm formulation published in 1977. Its expectation step averages a complete-data log likelihood using the current Conditional distribution of latent or missing variables; its maximization step updates parameters using that average. This makes a difficult incomplete-data likelihood accessible through a sequence of simpler calculations.

Harvard's institutional tribute also traces his work on Dempster-Shafer theory, in which evidence can leave an explicit component of uncertainty about an assertion. His dissertation concerned multivariate settings with more variables than observations, anticipating a central difficulty of high-dimensional statistics.^[ref]

Identity and life dates: Harvard Department of Statistics obituary and tribute.

The Bayesian model evidence is the prior predictive distribution evaluated at the data $D$. For a statistical model $H_j$ with a normalized prior density ${\pi }_j$, its marginal likelihood isThis is an expected value of the likelihood function under the prior distribution; for discrete statistical parameters, replace the integral by a sum. For continuous observations, $m_j(D)$ is a probability density function value with respect to a specified data reference measure, rather than a probability of one exact observation. Comparisons must use the same data and reference measure. The same $m_j(D)$ normalizes the within-model Bayesian posterior and supplies the between-model comparison in a Bayes factor.^[ref]

A normalized prior distribution makes the Bayesian model evidence an average, so it cannot exceed the largest likelihood function value on its support. A broad prior distribution can assign little mass to values that predict the data well, even when one value gives an excellent fit. The worked uniform-alternative binomial Bayes factor below makes the difference between averaging and maximum likelihood estimation explicit.

For observations $\mathcal{D}$ and graph $G$, Bayes' theorem gives $p(G\mid \mathcal{D})\propto \pi (G)\int p(\mathcal{D}\mid {\theta }_G,G)\pi ({\theta }_G\mid G)d{\theta }_G$. Here $\pi (G)$ is the graph prior distribution and ${\theta }_G$ the local distribution statistical parameters. The integral is Bayesian model evidence; it averages over nuisance parameters with a proper statistical parameter prior distribution. For independent complete observations, the node-factorized likelihood function and independence of local statistical parameter prior distributions make the evidence factor over nodes; suitable conjugate priors can make the local integrals analytic. With missing node observations, integrating out unobserved values can couple the local parameters, so independence of local prior distributions alone does not guarantee this factorization. Proper priors and coherent hyperparameters matter for comparing different graphs.

The Bayesian posterior is the conditional probability distribution of an unknown statistical parameter after observing data. Suppose a continuous prior distribution has prior density $\pi (\theta )$ relative to Lebesgue measure $d\theta$, and $p(d\mid \theta )$ is the sampling probability mass function or probability density function of the observed data $d$ in the statistical model. Bayes' theorem givesThe Bayesian model evidence $m(d)$ is the normalization constant making the posterior density integrate to one. The strict positive, finite condition is essential: a zero or infinite normalizer does not define this posterior distribution, even if the unnormalized expression can be written down. Other reference measures give the corresponding density formula; discrete parameter values replace the integral by a sum. The posterior propriety article gives examples where normalization fails.

With $d$ fixed, $L(\theta ;d)=p(d\mid \theta )$ is a likelihood function of $\theta$. Its normalization as a sampling distribution is over possible data, not over parameter values. In particular, a likelihood function is not a probability density function for $\Theta$ and need not integrate to one over $\theta$. The prior density supplies the probability weighting, and the Bayesian model evidence normalizes their product. This distinction prevents a tempting mistake: reversing the conditioning merely by exchanging the words “data” and “parameter”.

For eight heads among ten conditionally independent Bernoulli trials, take the observed data to be the heads count $D=8$. Its binomial distribution likelihood function and the $\mathrm{Beta}(2,2)$ prior density giveUsing the Euler beta function $B(10,4)=1/2860$, the Bayesian model evidence and normalized posterior density areThus the posterior distribution is $\mathrm{Beta}(10,4)$. If the data are instead one particular ordered sequence with eight heads and two tails, its likelihood function omits the factor 45 and its Bayesian model evidence becomes $3/1430$. The factor cancels in Bayes' theorem, leaving the same posterior distribution; these two data descriptions have different sampling probabilities but identical information about $\theta$ here.

The prior density and posterior density use the left axis and each integrate to one. The dashed likelihood function uses the right axis and is divided by its maximum solely to compare its shape; it is not a parameter probability density function. The likelihood function peaks at $8/10$, while the posterior mode is $9/12=3/4$ and the posterior mean is $5/7$. Multiplying by the prior density shifts the preferred values toward one half while preserving a range of possibilities.

The resulting posterior variance is $10\cdot 4/(14^2\cdot 15)=2/147\approx 0.0136054$, compared with prior variance $1/20$. In this example the update reduces uncertainty. The posterior predictive probability of heads on the next conditionally independent toss is $\mathbb{E}[\Theta \mid D=8]=5/7$, and the credible interval above reports a 95% range. These are summaries of the same posterior distribution, rather than competing estimates of one density peak.

Bayesian statistics expresses uncertainty about an unknown statistical parameter through a probability distribution, then uses Bayes' theorem to update it after observing data. The prior distribution describes uncertainty before the current observations; the likelihood function describes how the statistical model generates those observations at each parameter value; the posterior distribution describes uncertainty conditional on the observations and that model. A physical parameter can have one fixed value while its value remains uncertain to us. Assigning it a probability distribution represents that uncertainty under the chosen model, rather than requiring the physical parameter to change randomly between measurements.

For example, suppose ten Bernoulli trials are independent conditional on an unknown heads probability $\theta$, with a beta distribution prior $\mathrm{Beta}(2,2)$. Eight heads and two tails give the posterior distribution $\mathrm{Beta}(10,4)$ by Beta-binomial conjugacy. Its posterior mean is $10/14=5/7\approx 0.714286$, between the prior mean $1/2$ and observed binomial proportion $8/10$. The Bayesian posterior below computes the whole update, and the credible interval below summarizes its remaining uncertainty. The conjugate update is explained in MIT 18.05, Reading 15: beta and normal conjugate priors.

The useful feature is that one posterior distribution supports several questions: a posterior mean estimates a value, a credible interval reports a range, and a posterior predictive distribution predicts a future observation while averaging over parameter uncertainty. With conditionally independent new data, today's posterior distribution can become tomorrow's prior distribution. These conclusions remain conditional on the statistical model and prior distribution; a precise update does not establish that either assumption fits the experiment.

A historical starting point is Thomas Bayes's essay in the 1763 volume of the Royal Society's Philosophical Transactions, communicated after his death by Richard Price. Its opening problem asks for uncertainty about an unknown event probability given observed successes and failures: the same direction of reasoning illustrated by the coin example. See Bayes and Price, An Essay towards solving a Problem in the Doctrine of Chances, opening letter and problem.

A Bivariate standard normal distribution is a bivariate normal distribution whose two marginal means are zero and marginal variances are one; its remaining parameter is their correlation.

A capture-recapture model is a statistical model for repeated observations of identifiable individuals, including those missed on some occasions. A closed-population capture-recapture model estimates a fixed population cardinality; models allowing individuals to enter or leave can instead describe survival and recruitment. The sampling assumptions determine which likelihood function is appropriate.

For the two-sample closed-population capture-recapture model, condition on the sample sizes $M$ and $C$. If the second sample selects $C$ distinct individuals from $N$ by simple random sampling, then the overlap $R$ has the hypergeometric distribution:Here $r$ is an integer and the probability mass function is zero outside these bounds. The binomial coefficients count choices of $r$ marked and $C-r$ unmarked individuals, divided by all equally likely size-$C$ subsets. The hypergeometric distribution has expected valueEquating the observed overlap with this expected value gives the Lincoln-Petersen estimator. The underlying sampling without replacement, probability mass function, and expected value are developed in Raymond P. Guiteras, MIT 14.30 hypergeometric distribution handout, §§1–4 and 8.

For the toy counts $M=20$, $C=25$, and $R=5$, hold the observations fixed and vary the unknown integer $N$. The likelihood function isValues below forty cannot contain the observed set union. Exact comparison of consecutive likelihoods gives two tied maximum-likelihood estimates, $N=99$ and $N=100$. The Lincoln-Petersen estimator returns $100$; the Chapman estimator returns $90$ and is a different rule for estimation.

The vertical scale is $L(N)/{\max }_{n\ge 40}L(n)$. This relative likelihood compares how well candidate statistical parameters explain the same observations; it is not a posterior probability. The displayed range $40\le N\le 300$ is only a plotting window. The likelihood function remains positive for every integer $N\ge 40$ and tends to zero as $N$ increases, so the curve has a continuing right tail. The data do not justify truncating statistical inference at three hundred. The conditional hypergeometric distribution formulation also appears in Louis-Paul Rivest and Mamadou Yauck, Small Sample Estimators for Two-way Capture Recapture Experiments, §3, equation (5).

Zero recaptures require care. With $R=0$ and positive $M,C$, the Lincoln-Petersen estimator has no finite ratio, but the capture-recapture model still gives a valid likelihood function:This likelihood function increases toward one as $N$ grows, so there is no finite maximum-likelihood estimate. For the same sample sizes twenty and twenty-five with zero overlap, forty-five distinct individuals would be observed. Lack of overlap supplies a lower bound but very weak information about large population cardinalities.

In more elaborate capture-recapture models, a first-recapture array $m_{ij}$ counts individuals released at occasion $i$ and next observed at $j$. A probability of survival ${\varphi }_j$ and a probability of detection $P_{j+1}$ contribute a no-observation continuation factor ${\varphi }_j(1-P_{j+1})$. The resulting likelihood function depends on both survival and detection; these probabilities need not be separately identifiable at the last observation interval. Missing-count EM for capture-recapture describes one way to fit such a model, and identifiability remains a separate issue from computing an estimator output.

A sufficient statistic is complete when every integrable function of it with expectation zero for every parameter is almost surely zero for every parameter. Full regular one-parameter exponential families with natural parameter ranging over an open interval provide a standard completeness result. Completeness makes an unbiased function of a sufficient statistic unique, giving the Lehmann–Scheffé theorem.

Consider independent stage means $X\sim N(\delta ,1/I_1)$ and $Y\sim N(\delta ,1/I_2)$, with positive known information $I_1,I_2$. Continue to stage two only if $X\sqrt{I_1}\ge f$. The pooled mean is $M=wX+(1-w)Y$, where $w=I_1/(I_1+I_2)$. Write $m$ for its observed value. Its conditional mean under a candidate parameter $d$ isHere $\varphi$ and $\Phi$ are the standard normal probability density and cumulative distribution functions. The Conditional bias-corrected normal mean estimate solves $g(d)=m$. This inverts the selection-shifted mean instead of ignoring the fact that the first-stage mean exceeded a threshold.

For the upper-tail inverse Mills ratio $\lambda (a)=\varphi (a)/(1-\Phi (a))$, $g^{\prime }(d)=1-w{\lambda }^{\prime }(f-d\sqrt{I_1})$. The variance of a truncated standard normal variable gives $0<{\lambda }^{\prime }(a)<1$, hence $1-w<g^{\prime }(d)<1$. Since $0<w<1$, $g$ is strictly increasing and ranges over the whole real line; the root exists uniquely. Differentiating the joint normal likelihood conditional on continuation shows that the same root maximizes that conditional likelihood. Inverting a conditional mean does not itself make an estimator conditionally unbiased. The separate Rao-Blackwell estimator after interim selection provides that property in the equal-information setting.

A confidence interval is a random interval constructed from a statistical sample to cover a fixed unknown statistical parameter with a prescribed coverage probability. For a scalar target $\theta$, a procedure with nominal confidence $1-\alpha$ satisfiesThe subscript records which probability distribution generates the repeated samples. The endpoints are statistics, so they vary from sample to sample; the target is fixed. Equality gives exact nominal coverage probability, while the inequality also permits conservative procedures. For a general estimand $\psi (\theta )$, replace the target inside the interval by $\psi (\theta )$.

A realized confidence interval communicates the precision of an estimate in the target's own units. Reporting $1.2$ alone conceals the difference between an estimate with standard error $0.5$ and one with standard error $0.05$. A narrow confidence interval indicates greater precision for the stated statistical model, rather than a guarantee that its assumptions describe the experiment correctly.

Once $x$ is observed, $C(x)$ either contains the fixed target or does not. The 95% statement concerns the random procedure $C(X)$, not a posterior probability of 0.95 that this particular interval contains $\theta$. A credible interval instead assigns specified posterior probability to a region under Bayesian inference. These interpretations can differ even when the numerical endpoints agree. The repeated-sampling interpretation is explained in NIST's confidence-limits teaching reference.

Jerzy Neyman's 1937 paper formulates the fixed-target, random-endpoint construction in Section II, especially equation (20) on page 348. Its confidence coefficient must hold across the allowed statistical parameters, rather than only at the value suggested by one sample. See Neyman, Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability and the original paper scan.

A conservative variance estimator, in the finite-sample expectation sense used here, is an estimator $\hat{V}$ of a sampling variance $V_{\theta }$ satisfyingfor every statistical parameter value $\theta$ in the stated statistical model. Assume $V_{\theta }$ and ${\mathbb{E}}_{\theta }|\hat{V}|$ are finite. In design-based inference, the same definition applies for every permitted fixed population, with the sampling design supplying the randomness. The bias of an estimator is then nonnegative for this target. Aronow and Samii use this expectation criterion for their correction to the Horvitz-Thompson variance estimator when some pairs cannot be sampled together.^[ref]

A samplewise variance upper bound would require the stronger inequality $\hat{V}\ge V_{\theta }$ almost surely. An expectation bound allows some estimates below $V_{\theta }$. It does not even require that every realized value be nonnegative; in practice a variance estimate is commonly chosen to be a nonnegative random variable.

One construction starts with an integrable random variable $W$ that is an unbiased estimator of $V_{\theta }$ and might take negative values. Its positive part $\hat{V}=W_{+}=\max (W,0)$ satisfies $W_{+}\ge W$, hence is a conservative variance estimator. The amount of upward bias of an estimator is exactlyThe identity follows from the positive part and negative part decomposition $W=W_{+}-(-W{)}_{+}$. The Horvitz-Thompson variance estimator supplies a concrete setting in which the untruncated unbiased estimator can be negative.

A deliberately constructed example proves that conservative variance estimation does not guarantee Wald interval coverage. Observe $Y$ from a normal distribution $N(\theta ,1)$ and an independent $B$ from a Bernoulli distribution with success probability $0<\epsilon <1$. Use $\hat{\theta }=Y$ and $\hat{V}=B/\epsilon$. This nonnegative random variable hasThus it is even an unbiased estimator of the sampling variance. Yet the usual Wald confidence intervalhas coverage probabilitywhere $\Phi$ is the standard normal distribution function. When $B=0$, the interval is the singleton $\{Y\}$, which contains $\theta$ with probability zero because $Y$ is a continuous random variable. When $B=1$, its conditional coverage probability is the bracketed expression. At $\epsilon =0.01$, overall coverage probability is at most 1 percent, despite the nominal 95 percent rule and the exactly normal centre.

This example concerns the logical strength of the expectation criterion. Controlling the distribution of an estimated standard error, for example through an appropriate consistent estimator in an asymptotic argument, is additional work needed to justify confidence interval coverage.

In a normal linear model with an intercept and factor effects, set the effect of one factor level to zero. The intercept then denotes that reference level's mean, and the remaining effects are contrasts to it. The constraint removes redundant statistical parameters without altering the fitted mean space.

The coverage probability is the repeated-sampling probability that a random confidence interval contains the fixed true statistical parameter. A finite-sample level-$1-\alpha$ procedure has coverage probability at least $1-\alpha$ at every admissible statistical parameter; exact nominal coverage means equality. An asymptotically calibrated procedure approaches the nominal value as sample size grows, under its stated assumptions. Higher-order Edgeworth expansions can quantify the discrepancy from nominal coverage probability when their regularity conditions hold.

Observed coverage in finitely many repetitions is itself random. For $m$ independent repetitions of the exact normal mean confidence interval with known variance, the count $K$ of covering confidence intervals has binomial distribution $\mathrm{Binomial}(m,0.95)$. At $m=40$, its expectation is $38$, butConsequently, the 39 successes in the figure are compatible with exact 95% coverage probability. The guarantee follows from the pivotal quantity's probability distribution; the finite observed fraction is an illustration. Also distinguish a confidence interval for a fixed mean from a prediction interval for a future observation: the latter must accommodate the new observation's variation as well as estimation uncertainty.

A credible interval is an interval $C(d)$ containing a prescribed amount $1-\alpha$ of posterior probability for an unknown scalar statistical parameter $\Theta$. For a continuous posterior distribution with posterior density $p(\theta \mid d)$, its endpoints satisfyThe conditioning includes the chosen statistical model and prior distribution. For a discrete posterior distribution, exact equality may be impossible because of its probability masses; an interval can instead be required to contain at least the requested posterior probability. An interval compresses the full posterior distribution into a range whose uncertainty can be communicated in the parameter's own units. See MIT 18.05, Class 16: probability intervals.

For the eight-head, two-tail example in Bayesian statistics, the posterior density is $2860{\theta }^9(1-\theta {)}^3$ on $[0,1]$. Its cumulative distribution function, obtained by integration of this polynomial, isSolving $F(a)=0.025$ and $F(b)=0.975$ by the bisection method givesThese are numerical approximations to the exact quantiles defined by those equations; the endpoints were computed from the cumulative distribution function, rather than read off a plot. The resulting equal-tailed credible interval leaves 2.5% posterior probability on each side. Its shaded area is 95% of the total area under the posterior density.

Credible intervals are not unique. The one-sided credible interval $[0,F^{-1}(0.95)]\approx [0,0.887334219]$ also has 95% posterior probability, but leaves all 5% outside it on the right. A highest density region instead selects points above a density threshold; it need not be an equal-tailed credible interval, and when the posterior density has separated peaks it need not be one interval at all. Thus a reported credible interval should specify its construction.

A 95% credible interval makes a statement about the statistical parameter conditional on these observed data and the Bayesian model. A 95% confidence interval instead refers to the coverage probability of a procedure under repeated sampling at a fixed parameter value. These are different conditioning statements: 95% posterior probability does not by itself imply 95% coverage probability for every fixed parameter. The confidence interval article develops that sampling interpretation.

A demographer studies human populations quantitatively: how many people are present, how their characteristics are distributed, and how births, deaths and human migration change those counts. The work connects population census definitions and statistical sampling to statistical estimation and mathematical models of population dynamics. A forecast must distinguish observed counts from assumptions about future rates.^[ref]

For example, in a closed demographic cohort with $n_a(t)$ people aged $a$ and common one-year survival probability $s_a(t)$ conditional on that count, a basic projection isA cohort of 1000 people with survival probability $0.998$ contributes an expected 998 survivors to the next age class. Births create a new youngest class, and human migration requires additional terms. This elementary calculation illustrates why a demographer works with age structure and explicit statistical parameters, rather than extrapolating a single total without assumptions.

Double machine learning combines Neyman orthogonality with cross-fitting to estimate a low-dimensional statistical parameter while using machine learning for nuisance functions. Orthogonality means that the expected estimating score has zero first derivative with respect to nuisance perturbations at the truth. Small nuisance errors therefore affect the target equation only at second order, under the required smoothness assumptions. cross-fitting separately limits overfitting effects by evaluating scores on observations excluded from their nuisance training.

For the augmented inverse-probability-weighted estimator, a typical sufficient condition istogether with nuisance consistency, suitable moments and treatment probabilities bounded away from zero and one. Each error being $o_p(n^{-1/4})$ is sufficient for this product condition, but one may converge more slowly if the other converges faster. Under the full regularity conditions the target estimate has asymptotic normality at scale $n^{-1/2}$, enabling confidence intervals. Machine learning prediction accuracy alone does not supply these assumptions or Causal identification.^[ref]

The efficient information for a scalar target statistical parameter is the squared L2 norm of its efficient score. It cannot exceed the parametric Fisher information, by the Pythagorean theorem in an inner-product space. It can vanish when first-order target variation can be reproduced by nuisance variation.