Statistical model

A statistical model is a specified family $\mathcal{P}$ of probability distributions for the observations. A statistical parameter often indexes it as $\mathcal{P}=\{P_{\theta }:\theta \in \Theta \}$, where $\Theta$ is the parameter space. The choice states which patterns of variation are considered possible and supplies a basis for statistical inference; it does not guarantee that any member exactly describes the data-generating mechanism.

For example, suppose $n$ binary observations are independent and identically distributed with a common unknown success probability $p$. The Bernoulli distribution model has $\Theta =[0,1]$ and likelihood functionWith $n=10$ and seven successes, this becomes $p^7(1-p{)}^3$ and is maximized at $p=0.7$. The binary observation type alone does not supply independence or a common $p$: those are additional model assumptions. A regression model can instead let the response law depend on observed covariates, while a mixed model represents a specified grouping structure through random effects.

A Bayesian hierarchical model is a statistical model used in Bayesian statistics, in which parameters at one level have a prior distribution controlled by parameters at another level. Those shared hyperparameters can themselves receive a hyperprior. For grouped observations $y_j$, group parameters ${\theta }_j$ and shared parameters $\varphi$, one common conditional-independence structure isThe hierarchy specifies how information can be shared; the data are not assumed to become independent merely because they are listed in different groups.^[ref]

A normal distribution example makes partial pooling explicit. Let ${\bar{y}}_j\mid {\theta }_j\sim \mathcal{N}({\theta }_j,{\sigma }^2/n_j)$ and ${\theta }_j\mid \mu ,\tau \sim \mathcal{N}(\mu ,{\tau }^2)$. Conditional on known $\mu ,\tau ,\sigma$, completing the square gives the posterior meanFor $n_j=4$, ${\sigma }^2=4$, ${\tau }^2=1$, $\mu =2$ and ${\bar{y}}_j=6$, this is4, halfway between the group mean and the shared mean. More precise group data increase $w_j$. In a full hierarchy with uncertain hyperparameters, their posterior distribution must also be integrated over; simply fixing fitted values omits that uncertainty.

A measurement application appears in Blaza Toman's2017 calibration study. Shared uncertain calibration inputs induce correlations among results; the Bayesian hierarchical model represents this structure within the model rather than treating all replicates as independent.^[ref]

Model recovery tests whether a model selection procedure can identify the known model that generated simulated data. Generate observations from each candidate statistical model, fit the candidates using the intended pipeline, and record which candidate is selected. It is a check on the comparison procedure under stated simulation conditions.^[ref]

This differs from parameter recovery, which compares estimated parameters with the settings used to generate data within a model. Correctly identifying a model does not automatically imply accurate parameter estimation, and accurate fitting within one model does not establish that a procedure can distinguish it from its competitors.

For an invented example, suppose one hundred simulated datasets come from each of models A and B. The resulting counts areThis confusion matrix estimates selection behavior conditional on the generating model. Its empirical rate of choosing B when A generated is $20/100=0.20$.

The reverse question has a different denominator. Among the $85$ datasets for which B was selected, $65$ were generated by B, giving $65/85\simeq 0.765$. That calculation uses the example's balanced simulation design. It is not interchangeable with the $0.65$ recovery rate for B-generated datasets. Different generating-model frequencies can change the reverse proportion even if the conditional selection behavior stays fixed.^[ref]

Recovery depends on parameter ranges, observation count, noise and the selection criterion. Models with nearly indistinguishable predictions can be hard to recover despite successful numerical optimization. Repeat simulations in the regimes relevant to the planned inference. Finally, recovery among a chosen candidate set does not prove that the real system belongs to that set; checking how well the selected model reproduces relevant observations remains a separate task.^[ref]

A nested Markov model is a statistical model associated with an acyclic directed mixed graph. Besides ordinary conditional independence, it imposes Markov restrictions on probability kernels obtained through valid sequences of a fixing operation. Fixing changes a random vertex into a fixed input and transforms the distribution by an appropriate conditional-density ratio. It is not simply conditioning. Formally, every kernel reached by a valid sequence must satisfy the global Markov property of its reached graph.^[ref]

A concrete equality can arise from the positive joint distributionwith $U$ unobserved. DefineSubstitution and Bayes' theorem reduce this to ${\sum }_{u}p(d\mid c,u)p(u)$, independent of $a$. This is a Verma constraint; the graph can impose it without implying any ordinary conditional independence among the observed variables.^[ref]

For a numerical instance, make $U,A$ independent fair binary variables, and choose success probabilities $p(B=1\mid a,u)=0.1+0.2a+0.4u$, $p(C=1\mid a,b)=0.15+0.2a+0.3b$, and $p(D=1\mid c,u)=0.1+0.3u+0.4c$. Then $q(D=1\mid a,c)=0.25+0.4c$ for either $a$. Yet $p(D=1\mid A=0,C=0)=181/760$ differs from $p(D=1\mid A=1,C=0)=29/125$.

A latent-variable model can impose additional inequalities, so its observed margin may be strictly smaller than the associated nested Markov model.^[ref]

The fixing operation in a nested Markov model turns a random vertex of a conditional acyclic directed mixed graph into a fixed input and transforms its probability kernel. A vertex $v$ is fixable if no other vertex in its bidirected-connected district of an acyclic directed mixed graph is a descendant of $v$. Graphically, fixing removes edges with an arrowhead at $v$ and moves $v$ from the random to the fixed vertex set.^[ref]

For a positive Markov kernel $q$, the transformation divides $q$ by the conditional factor for $v$ given its district's other vertices and the parents of that district. Fixed arguments are retained as kernel inputs. Fixability is what permits the resulting object to be a normalized kernel over the remaining random variables. Valid fixing sequences must check this condition again after each step.

For the directed acyclic graph $A\to B\to C$, write $p(a,b,c)=p(a)p(b\mid a)p(c\mid b)$. Fixing $B$ givesOrdinary conditioning instead gives $p(a\mid b)p(c\mid b)$. If $A$ is a fair binary variable and $p(B=1\mid A=1)=0.8$, $p(B=1\mid A=0)=0.2$, then conditioning on $B=1$ makes $p(A=1)=0.8$, whereas fixing $B=1$ leaves it $0.5$. The operation is algebraically defined even without interpreting a graph causally.

The parameter space of a statistical model is the set of allowed statistical parameter values indexing its probability distributions. For an exponential distribution parametrized by its mean, $\Theta =(0,\infty )$; for a Bernoulli distribution, $\Theta =[0,1]$. Intersecting a confidence interval with the known parameter space cannot remove the true statistical parameter and therefore preserves its coverage probability.

Simulation of observations under known parameter values followed by refitting the model to assess whether those values can be recovered under the intended design. Recovery depends on information, noise and identifiability; good recovery in an assumed model does not establish that the model describes the real mechanism.

A statistical model combining fixed effects with random effects. The grouping structure determines the correlations represented by the model.

Given the currently infected set $I_t$ and susceptible set $S_t$, define $A_{it}(\eta )={\sum }_{j\in I_t}{K}_{\eta }(d_{ij})$ and $P_{it}=1-e^{-\alpha A_{it}}$. Conditionally independent new-infection indicators $Y_{it}$ have likelihoodA nonnegative infection intensity requires $\alpha \ge 0$; an unrestricted normal distribution prior distribution for $\alpha$ must therefore be restricted or replaced. With $K_{\beta }(d)=d^{-\beta }$ or $K_{\gamma }(d)=e^{-\gamma d}$ and positive distances, the conditional posterior kernels are not generally conjugate normal probability densities. Metropolis–Hastings algorithm updates, optionally on $\log \alpha$ with its Jacobian determinant, provide a direct sampling method.

A group action on the sample space induces a compatible action on the parameter space, preserving the family of probability laws. In a transitive model every law arises by transforming a fixed reference law. A location-scale model is generated by $y\mapsto a+by$, $b>0$, with parameter action $(\mu ,\sigma )\mapsto (a+b\mu ,b\sigma )$.

An estimator respects the group action if the displayed identity holds. If it takes values in the transformation group itself, normalization $T(y{)}^{-1}y$ is a maximal invariant: equivariance proves invariance, and equality of normalized samples reconstructs the transformation between them. For estimates in a homogeneous parameter space with a nontrivial stabilizer, normalization is defined only modulo that stabilizer; quotienting the remaining action is essential.

A statistic is invariant if $A(gy)=A(y)$; it is maximal when it also separates different group orbits. Consequently every invariant procedure factors through the maximal invariant, subject to the usual measurability conditions on the orbit space. Standardizing a sample by an equivariant estimator of location and positive scale gives a maximal invariant for the positive affine group.

On samples of distinct real observations, the labeled rank vector is a maximal invariant for simultaneous strictly increasing bijections of the real line. Such transformations preserve every comparison. Conversely two samples with the same ranks can be matched by a piecewise linear increasing bijection through their corresponding ordered observations, with positive-slope tails. Under a common continuous probability distribution, every labeled ordering is equally likely by exchangeability, giving distribution-free rank tests. This argument does not assume that all continuous distributions lie in one orbit of real-line homeomorphisms.

If $T$ is a maximal invariant for a group action, an invariant test function is constant on each group orbit and therefore equals $b\circ T$ for a function $b$ on the orbit space. Conversely every such function is invariant. Appropriate quotient measurability is understood. If the action is transitive on a null family of probability distributions, invariance also gives the same null law of $T$ under every member. Thus tests based on $T$ eliminate that null nuisance variation.

A statistical model indexed by a parameter vector of fixed finite dimension is parametric. Examples include a normal distribution with unknown mean and variance and a Poisson distribution with unknown mean. Its likelihood function is the joint sample density or mass function, regarded as a function of the parameter at the observed sample.

In a location-scale family, $Y=\mu +\sigma Z$ for a fixed base probability distribution of $Z$, with $\mu \in \mathbb{R}$ and $\sigma >0$. Location translates observations, and scale multiplies their deviations. For an independent sample with a base density and at least two observations, the normalized residual vector $(Y_i-\bar{Y})/s$, where $s^2=n^{-1}{\sum }_i(Y_i-\bar{Y}{)}^2$, is an ancillary statistic. Indeed, $\bar{Y}=\mu +\sigma \bar{Z}$ and $s=\sigma s_Z$, making the residual vector a function of the base sample alone. This algebraic result needs no population moments.

If $W$ has probability density function $f(w;q)$ and cumulative distribution function $F_W(w;q)$, then $T=\exp (\alpha +\sigma W)$, $\sigma >0$, has density $f((\log t-\alpha )/\sigma ;q)/(\sigma t)$ and distribution function $F_W((\log t-\alpha )/\sigma ;q)$ for $t>0$. The factor $1/(\sigma t)$ follows from the change-of-variables formula for a probability density. Upper truncation at $M$ divides the density by $F_W((\log M-\alpha )/\sigma ;q)$.

For normalized residuals $A_i=(Y_i-\bar{Y})/s$, Fisherian conditional inference uses the conditional distribution of $(\bar{Y},s)$ given the observed $A=a$. For almost every attainable ancillary value, the joint conditional density of $\bar{Y}=m$ and $s=t>0$ is proportional toThe factor $t^{n-2}$ is the radial Jacobian on the $(n-1)$-dimensional centered residual space. After setting $u=(m-\mu )/\sigma$ and $v=t/\sigma$, the conditional density is proportional to $v^{n-2}{\prod }_i{f}_0(u+v{a}_i)$, which is parameter-free. Conditioning on a continuous ancillary uses a regular conditional distribution, rather than division by the probability of a singleton ancillary value.

A regression model specifies features of the conditional distribution of a response given predictor covariates, often a conditional mean or a full parametric density. Linear regression specifies a linear conditional mean; logistic regression specifies a logit-linear conditional probability for a binary response. The conditional comparison represented by a coefficient depends on which covariates are included.

A statistical interaction means that the effect of one predictor depends on another on the chosen response scale. In a linear regression, an interaction term ${\beta }_{3}XZ$ changes the slope in $X$ to ${\beta }_1+{\beta }_{3}Z$. Whether interaction is present depends on the comparison scale; additive effects and multiplicative effects use different null contrasts. Comparing subgroup effects requires an interaction contrast, rather than contrasting their separate significance labels.

Analysis of one factor's effect at a specified level of another factor, for example a cue-congruence difference separately at each delay. The interaction compares those effects directly. One significant simple-effect test and one nonsignificant simple-effect test do not themselves establish an interaction; compare the estimates with the appropriate uncertainty and multiple testing treatment.

An interaction plot displays response means against levels of one factor, with separate lines for levels of another and optional panels for further factors. In an additive linear regression, the difference between two lines is constant across the horizontal factor, so the population-mean lines are parallel. Changing line separations or crossings show a statistical interaction on that response scale. Sample means may be nonparallel through noise, so the plot describes the effect pattern rather than replacing a formal hypothesis test.

An interaction term lets the effect of one predictor depend on another. Inthe slope with respect to $X$ is ${\beta }_1+{\beta }_{3}Z$.

A two-factor interaction means that changing one factor has a different effect at different levels of another. For a two-level factorial design, its sign column is $x_i{x}_j$, and four times its coefficient is the corresponding difference of differences. It can be confounded with other factorial contrasts in a fractional factorial design.

A hierarchical statistical model that contains an interaction term retains its associated lower-order effects. For two categorical predictors this means keeping both main effects when keeping their interaction. A nonsignificant averaged main effect may coexist with large opposite effects in different groups. Dropping its lower-order term while retaining the interaction can make the model depend on arbitrary factor coding.

A statistical path is a one-dimensional family of probability measures inside a statistical model, passing through a specified baseline $P$. A differentiability in quadratic mean condition gives its score function, the first-order direction of the family in square-root-density coordinates. The path must respect all restrictions of the statistical model, including normalization and any fixed moments.

A statistical tangent set is the collection of score functions attained by a specified family of differentiable-in-quadratic-mean paths through $P$. The choice of paths is part of the definition. Its elements lie in the mean-zero L2 space, but the set need not already be a closed vector subspace.

For bounded measurable $g$ with $Pg=0$, the formula $p_t=p(1+tg)$ defines a normalized positive probability density function for $|t|\Vert g{\Vert }_{\infty }<1$. A uniform Taylor expansion of the square root proves differentiability in quadratic mean with score function $g$: the squared remainder is $O(t^{4}P{g}^4)$. Such paths realize all bounded centered directions of an unrestricted density model.

For $h\in L_0^2(P)$, truncate $h$ to $h_m=\max (-m,\min (h,m))$ and subtract $P{h}_m$. Dominated convergence gives $h_m\to h$ in L2 space, and the Cauchy-Schwarz inequality gives $P{h}_m\to 0$. Thus bounded centered directions are dense in the mean-zero L2 space. The measure defining the L2 norm is essential: density-weighted L2 space can contain functions outside the unweighted Lebesgue space.

The statistical tangent space is the closed linear span in L2 space of a statistical tangent set. Taking this closure makes orthogonal projection available and ensures that a continuous derivative specified on attainable score functions extends to the whole space.

A dominated statistical path is differentiable in quadratic mean at $P$ if its square-root probability density function has an L2 space derivative $(1/2)g\sqrt{p}$, with $g\in L^2(P)$. The score function is $g$. This formulation controls mass near zeros of $p$ as well as the derivative on the support of $P$; a pointwise log-density derivative is insufficient by itself.

Let ${\delta }_t=\sqrt{p_t}-\sqrt{p}=(t/2)g\sqrt{p}+r_t$, with $\Vert r_t{\Vert }_2=o(|t|)$. Since $p_t-p=2\sqrt{p}{\delta }_t+{\delta }_t^2$, the Cauchy-Schwarz inequality gives $\Vert p_t-p-tpg{\Vert }_1\le 2\Vert r_t{\Vert }_2+\Vert {\delta }_t{\Vert }_2^2=o(|t|)$. Thus differentiability in quadratic mean suffices to differentiate bounded density integrals, even when the density itself has no useful pointwise derivative.

For a differentiable-in-quadratic-mean path, normalization gives $⟨(\sqrt{p_t}-\sqrt{p})/t,\sqrt{p_t}+\sqrt{p}⟩=0$. The two factors converge in L2 space to $g\sqrt{p}/2$ and $2\sqrt{p}$. Continuity of the inner product therefore gives $Pg=0$. This proves score function centering without differentiating the density integral.

A scale family rescales a fixed distribution by a positive parameter. The standardized variable $Y/\sigma$ has density $f$, independent of $\sigma$. This simplifies moments, simulations and the Jeffreys prior for a scale parameter.

A statistical parameter indexes a family of probability distributions in a statistical model. It is fixed in a frequentist sampling model; a prior distribution makes it random for Bayesian statistics. Different statistical parameter values should produce different observable distributions when identifiability is claimed.

A nuisance parameter is a statistical parameter required to specify the distribution of observations but not itself the target of inference. A profile likelihood maximizes over it; Bayesian model evidence integrates over it using a proper prior distribution. These operations differ and need not give the same inference. A baseline hazard in a Cox proportional-hazards model is an infinite-dimensional example.

On a common positive support, a factorization $f={\varphi }_0(x){\varphi }_C(x;\psi ){\varphi }_S(x;\lambda )$ implies ${\partial }_{\psi }{\partial }_{\lambda }\log f=0$. Thus any nonzero mixed derivative prevents this exact separation. For $n$ independent gamma distributed observations with shape $a$ and rate $b$, this mixed derivative is $n/b$, so exact separation of shape and rate into such factors is impossible.

A covariate is an observed explanatory variable used to model the conditional distribution of a response. A vector of covariates can enter a linear predictor, a treatment-effect adjustment or a hazard multiplier. Conditioning on a covariate does not itself establish that its effect is causal.

A baseline covariate is measured before treatment assignment or the exposure being studied. A prognostic baseline covariate predicts the outcome. In a randomized controlled trial, adjusting for such a covariate can improve statistical efficiency without conditioning on a consequence of treatment. For instance, a measurement taken before treatment can explain persistent between-subject variation in a later measurement.^[ref]

Here is a linear regression example with explicit noise assumptions. Letwhere treatment $Z$ is independent of the baseline covariate $X$, all displayed variances are finite, $\mathbb{E}[\epsilon \mid X,Z]=0$ and $\mathrm{Var}(\epsilon \mid X,Z)={\sigma }^2$. ThenThe first identity uses the law of total variance. It isolates the unexplained population variation removed by adjustment for $X$. For $\gamma =2$, $\mathrm{Var}(X)=1$ and ${\sigma }^2=1$, this variation falls from $5$ to $1$. Estimating additional coefficients still uses data, so this population calculation is not a guarantee that every finite-sample reported standard error becomes smaller. Measurement before treatment also does not by itself make a covariate prognostic or establish a causal effect.

Constancy of the error variance across observations or conditional covariate values. In fixed-design nonparametric regression, $Y_i=m(x_i)+\sigma {ϵ}_i$ with $\mathrm{Var}({ϵ}_i)=1$ has $\mathrm{Var}(Y_i)={\sigma }^2$ for every design point.

A statistical model is identifiable when distinct parameter values determine distinct probability distributions. In a nonidentifiable parameterization, data can at most estimate combinations of parameters that remain unchanged along each equivalent parameter set.

Nonregular identification at a flat mean map occurs when different parameter values give different probability distributions, yet the first derivative of the observable mean vanishes at a parameter of interest. Identifiability is a property of the entire parameter-to-distribution map; a first-order sensitivity calculation can miss it.

Consider an original statistical model with independent observationsThe cubic mean map is a strictly increasing function, so equality of the normal distributions implies equality of $\theta$. The model is globally identifiable, including at $\theta =0$. Nevertheless, the single-observation score function and Fisher information areThus $I(0)=0$ does not imply nonidentifiability. The rank is zero at the origin and one at every nonzero nearby point. This is precisely why a regular-point assumption matters: Rothenberg's local identification criterion assumes constant information-matrix rank in a neighborhood.^[ref]

Maximum likelihood estimation gives $\hat{\theta }=\mathrm{cbrt}(\overline{Y})$, using the real cube root. At the true value zero, write $\overline{Y}=Z/\sqrt{n}$ with a standard normal random variable $Z$. ThenThis is an exact sampling distribution identity for every $n$, not merely a convergence in distribution statement. The estimation error has scale $n^{-1/6}$ and a nonnormal shape. At a fixed nonzero $\theta$, the usual delta method instead gives asymptotic variance $1/(9n{\theta }^4)$; that approximation cannot be extended uniformly through zero.

An exact confidence interval through a cubic mean map follows by applying the strictly increasing function $\mathrm{cbrt}$ to both endpoints of the ordinary known-variance confidence interval for the mean:Here $z_p$ is the quantile function of the standard normal distribution. The transformed interval has exact coverage $1-\alpha$ for every $\theta$. When $n=64$ and the observed sample mean is zero, the 95% interval is approximately $[-0.626,0.626]$. Multiplying $n$ by 64 halves its width at that observed mean. A first-order interval based on dividing by the zero mean-map derivative would fail instead.

Quantity-specific identifiability asks whether a chosen function of a model's parameters is determined by the observable probability distribution, even when the complete parameter vector is not. For model $\{P_{\theta }\}$ and target $g(\theta )$, the criterion isEquivalently, the target must be constant on each class of observationally equivalent parameters. This separates identification of a scientifically meaningful quantity from reconstruction of every internal parameter. Thomas J. Rothenberg's primary treatment explicitly distinguishes identification of individual parameter components from that of a full system.^[ref]

For an original example, let observations have a Poisson distribution with mean $a+b$, where $a,b>0$. Parameter pairs $(a,b)=(2,3)$ and $(1,4)$ give exactly the same observable law. The sum $g(a,b)=a+b$ is identifiable, but $a/(a+b)$ is not: its values are $0.4$ and $0.2$. With $n$ independent observations, the sample mean estimates the sum with variance $(a+b)/n$. More observations can improve that estimate without distinguishing the two allocations.

Identifiability from linear observations gives a useful exact criterion. Suppose the observation law identifies $A\theta$ and contains no additional information about $\theta \in {\mathbb{R}}^p$. A linear target $c^{\mathsf{T}}\theta$ is identifiable exactly when $c$ belongs to the row space of $A$. To prove this, observationally equivalent parameter differences lie in the kernel of a linear map $A$. The target is unchanged for every such difference precisely when $c$ is orthogonal to that kernel, which is equivalent to membership in the row space. If $c=A^{\mathsf{T}}d$, the target can be recovered as $d^{\mathsf{T}}(A\theta )$.

A deterministic geometric analogue appears in additional measurements and anatomical identifiability: measured perimeter can determine the surface area of the stipulated cuboid family even when exchanging its two horizontal side labels remains invisible. The target and the allowed model class must be stated.

Identification is not the same as good precision. A formally identifiable target can still be weakly constrained by finite noisy data. Conversely, a nonidentifiable target requires additional assumptions or observations, rather than merely more repeated data from the same indistinguishable law.

Identifiability from linear observations asks which features of an unknown vector $\theta \in {\mathbb{R}}^p$ are fixed by observing $y=A\theta$. Two parameter vectors are observationally equivalent exactly when their difference lies in the kernel of a linear map $A$:Thus the full vector is identifiable precisely when $\ker A=\{0\}$, equivalently when $A$ has full column matrix rank. Repeating the same rows can improve precision in a noisy measurement without increasing this rank.

A linear target $g(\theta )=c^{\mathsf{T}}\theta$ requires less information. It is identifiable if and only if $c$ is in the row space of $A$. If $c=A^{\mathsf{T}}d$, then $g(\theta )=d^{\mathsf{T}}y$, so the target is recovered directly. Conversely, an identifiable target must vanish on every observationally invisible difference: $c^{\mathsf{T}}h=0$ for every $h\in \ker A$. Orthogonality to the kernel is exactly row-space membership. This is a concrete instance of quantity-specific identifiability; the primary identification literature also treats identification of individual parameter components by rank conditions.^[ref]

For an original example, supposeBoth $(1,4,3)$ and $(2,3,4)$ fit perfectly. Their difference is a multiple of $(1,-1,1)$, which spans the kernel. The target ${\theta }_1-{\theta }_3=y_1-y_2=-2$ is identified, whereas ${\theta }_1+{\theta }_2+{\theta }_3$ takes values 8 and 9 and is not. A new independent measurement of ${\theta }_2$ restores full identification; another copy of $y_1$ does not.

The same distinction applies to a statistical model $Y\sim N(A\theta ,\Sigma )$ with known positive-definite covariance matrix $\Sigma$: two parameters give the same observable law precisely when their means agree. A Moore-Penrose pseudoinverse selects one solution when the system is underdetermined, but that numerical convention does not show that the selected vector is the unique physical parameter. Additional restrictions must be justified as assumptions, not silently supplied by a solver.

In a shared zero-inflated Gamma-Poisson count model, write $m_j=(1-\pi ){\mu }_j$ and $\kappa =(\tau +\pi )/(1-\pi )$. Its first two moments are $\mathrm{Var}(Y_j)=m_j+\kappa m_j^2$ and $\mathrm{Cov}(Y_j,Y_k)=\kappa m_j{m}_k$. These moments do not separately identify $\pi$, $\tau$, and the component-mean intercept: taking ${\pi }^{\prime }=0$, ${\tau }^{\prime }=\kappa$ and ${\mu }_j^{\prime }=m_j$ gives the same first two moments. The full count distribution can carry information absent from the moments. A consistent estimator of a marginal mean ratio therefore need not consistently estimate a structural-zero fraction from a misspecified moment parameterization.

In a normal linear model with an intercept and factor effects, set the effect of one factor level to zero. The intercept then denotes that reference level's mean, and the remaining effects are contrasts to it. The constraint removes redundant statistical parameters without altering the fitted mean space.

A location family has distribution functions $F_{\theta }(x)=F_0(x-\theta )$, so changing the parameter translates the distribution without changing its shape.

The location score is the score function for translating a probability density function $f$. The minus sign comes from differentiating $f(y-\theta )$ in $\theta$. Under regular tail conditions, integration by parts gives $E_f\rho =0$ and $E_f(\epsilon \rho )=1$. These identities determine projections of the score function onto the span of the centered error.

For a known mean-zero noise law with finite variance ${\sigma }^2$, observing only $Y=a+ϵ$ with unrestricted $a\in \mathbb{R}$ has minimax risk ${\sigma }^2$ under squared-error loss. The observation itself attains this risk. For the lower bound, use a flat prior distribution on $[-A,A]$ and condition an easier experiment on $|ϵ|\le M$. Away from the prior endpoints, the Bayesian posterior noise has its original truncated law. Let $A\to \infty$ and then $M\to \infty$ in the resulting Bayes risk lower bound. No normal distribution or density for the noise is required.

A probabilistic graphical model represents factorization and conditional-independence structure using a graph whose vertices are random variables.

A junction tree connects the maximal cliques of a chordal graph into a tree satisfying the running intersection property. Its edges carry separators equal to the intersections of the endpoint cliques. Assigning original factors to containing cliques permits exact sum-product belief propagation without summing independently over the entire collection of variables.

The junction tree algorithm performs exact inference in a finite probabilistic graphical model. First moralize a Bayesian network if necessary; then apply triangulation of an undirected graph, organize its maximal cliques into a junction tree, and assign each original factor to one containing clique. Inward and outward junction-tree sum-product messages compute the normalizing constant and marginal distributions. Its cost depends exponentially on the largest clique's number of variables, rather than necessarily on the total number of variables.

Assign each factor exactly once to a containing clique and multiply the factors there into ${\psi }_i$. The displayed sum-product belief propagation message sums out variables of $C_i$ outside the separator $S_{ij}=C_i\cap C_j$. By the running intersection property, eliminating all variables in the subtree on the $i$ side creates precisely this function of the separator. An inward pass computes the normalizing constant at the root; an outward pass produces marginal distributions. For pedigree likelihoods, observed genotypes and penetrance factors simply multiply the relevant potentials.

The running intersection property says that every variable belongs to a connected subtree of a junction tree. Equivalently, every clique along the path between two cliques contains their intersection. This ensures that eliminating variables on one side of a tree edge leaves a message depending only on its separator with the other side.

A Bayesian network consists of a directed acyclic graph $G$ whose nodes are random variables, together with conditional distributions satisfying $p(x)={\prod }_{j}p(x_j\mid x_{{\mathrm{pa}}_G(j)})$. Its directed acyclic graph encodes conditional independence constraints. For unrestricted binary variables it has ${\sum }_j{2}^{|{\mathrm{pa}}_G(j)|}$ free conditional probabilities. A directed acyclic graph used this way need not have a causal interpretation.

For observations $\mathcal{D}$ and graph $G$, Bayes' theorem gives $p(G\mid \mathcal{D})\propto \pi (G)\int p(\mathcal{D}\mid {\theta }_G,G)\pi ({\theta }_G\mid G)d{\theta }_G$. Here $\pi (G)$ is the graph prior distribution and ${\theta }_G$ the local distribution statistical parameters. The integral is Bayesian model evidence; it averages over nuisance parameters with a proper statistical parameter prior distribution. For independent complete observations, the node-factorized likelihood function and independence of local statistical parameter prior distributions make the evidence factor over nodes; suitable conjugate priors can make the local integrals analytic. With missing node observations, integrating out unobserved values can couple the local parameters, so independence of local prior distributions alone does not guarantee this factorization. Proper priors and coherent hyperparameters matter for comparing different graphs.

A Gaussian Bayesian network specifies a linear normal conditional model at each node of a Bayesian network: $X_j\mid X_{\mathrm{pa}(j)}\sim N({\alpha }_j+{\gamma }_j^T{X}_{\mathrm{pa}(j)},{\sigma }_j^2)$. Independent local errors and an acyclic ordering generate a joint multivariate normal distribution. Positive conditional variances give a nonsingular joint multivariate normal distribution. The graph constrains regression coefficients and hence conditional independence.

Belief propagation passes local messages between factors or neighboring variables to compute marginal distributions or maximizing assignments. On a tree, two directed messages per edge give exact results in time linear in the number of vertices when state spaces are fixed.

The sum-product form of belief propagation sums over eliminated states in each message. On a tree it computes exact normalization constants and marginal distributions.

The max-product form of belief propagation replaces summation by maximization. On a tree, storing maximizing states while passing messages and then backtracking gives an exact maximum a posteriori estimate.

A Markov blanket of a variable is a set that makes it conditionally independent of every remaining variable. In a directed acyclic graph, the parents, children, and other parents of the children form the standard Markov blanket.

Under faithfulness of a directed acyclic graph, every conditioning set screening a vertex from all remaining vertices contains its graphical Markov blanket. A missing adjacent vertex leaves a one-edge active path; a missing other parent of a child leaves the active path $k\to h\leftarrow u$, since the child $h$ is necessarily conditioned. Thus the graphical blanket is the unique smallest screening set.

In a directed acyclic graph, let $B$ contain the parents, children, and other parents of children of vertex $k$. Then $k$ is D-separated from all remaining vertices by $B$. Every path out of $k$ meets a conditioned noncollider either at a parent, at a child, or at another parent just after a conditioned child-collider. The global Markov property for a directed acyclic graph gives $Z_k\perp Z_{(B\cup \{k\}{)}^c}\mid Z_B$.

A Markov random field is an undirected graphical model whose graph encodes conditional independence by vertex separation.

On a graph, a binary configuration has probability proportional to $\exp (\alpha {\sum }_v{x}_v+\beta {\sum }_{\{v,w\}}{x}_v{x}_w)$, with each edge counted once. The full conditional probability of a one is logistic with linear predictor $\alpha +\beta {\sum }_{w\sim v}{x}_w$. Local neighbor sums eliminate the need to evaluate the normalizing constant.

For an autologistic binary-image model on a finite undirected graph, let $x_v\in \{0,1\}$ and give the configuration prior mass proportional toSuppose $Y_v\mid x$ are independent Gaussian distribution observations with expected value $x_v$ and unit variance. Their local log-likelihood ratio for $x_v=1$ versus $0$ isWriting $z_v={\sum }_{w\sim v}{x}_w$, the full posterior conditional probability is thereforeThus the local evidence changes the one-site field while retaining the same neighbor interaction. For a noise variance ${\sigma }^2>0$, replace $y_v-1/2$ by $(y_v-1/2)/{\sigma }^2$.

The graphical conditional independence survives this update: vertices in any independent color class are conditionally independent given the other vertices and the observations. On a nearest-neighbor square grid, the two checkerboard colors give such classes, so one color can be updated in parallel in a Gibbs sampler. An arbitrary graph need not be bipartite and may require more colors. Updating every vertex simultaneously from its old neighbors is not the same as this exact conditional block update.

A clique potential is a nonnegative function of the variables belonging to one clique. Products of clique potentials define the unnormalized density of a Markov random field.

The global Markov property says that graph separation of vertex sets $A$ and $B$ by $S$ implies conditional independence of the corresponding random vectors given the variables at $S$.

Statistical modelling constructs, fits, checks, and interprets a statistical model for observed data.

A model assumption specifying a normal distribution for the relevant random quantities, often errors conditional on predictors. In an analysis of variance, it concerns within-group errors; in a paired t-test, it concerns the differences. It does not generally require pooled outcomes from distinct groups to be normal.

Data centering subtracts the sample mean from every observation, separately for each coordinate of multivariate data. The resulting coordinate sample means are zero.

Regression analysis studies how a response variable depends on explanatory variables through a regression model. It estimates that dependence and assesses uncertainty in fitted regression coefficients, predictions and model comparisons. Linear regression is one important case.

A statistical description in which an association between an explanatory variable and an outcome is decomposed using an intermediate variable. Such a fitted relation is not automatically a causal pathway; causal mediation requires additional identification assumptions and an appropriate design.

Mark and recapture estimates the cardinality of a finite set of individuals when counting everybody is impractical. The useful information is the overlap between observations: with fixed sample sizes, finding many of the same individuals again suggests a smaller population, while finding few suggests a larger one. More elaborate capture-recapture models also estimate survival and changing probabilities of detection.

In a closed-population capture-recapture model, let $N$ be the unknown population cardinality. Capture $M$ distinct individuals, mark them, release them all, and allow them to mix with the rest. Next take a second sample of $C$ distinct individuals using sampling without replacement, and count the $R$ individuals carrying a mark. The observed marked fraction $R/C$ estimates the population's marked fraction $M/N$, giving the Lincoln-Petersen estimatorRelease happens between the two samples; sampling without replacement means that an individual is counted at most once within the second sample. See Richard J. Barker, Open Population Mark-Recapture Models, §1.1.1.

For a constructed teaching example, mark $M=20$ individuals and later capture $C=25$, including $R=5$ marked individuals. One fifth of the second sample is marked, suggesting that the twenty marked individuals are one fifth of the population:Let $A$ and $B$ be the sets of individuals in the first and second samples. Their intersection has cardinality five. By the inclusion-exclusion principle, the set union containsThus $N\ge 40$ is an observed lower bound, whereas $100$ is the output of an estimator based on the capture-recapture model. These are toy counts, not measurements from a medical or ecological study.

The diagram shows only observed individuals. It does not draw an assumed population of one hundred. The capture-recapture model uses the overlap to infer how many individuals may have escaped both samples; its likelihood function shows why one estimator output cannot describe all that uncertainty.

A capture-recapture model is a statistical model for repeated observations of identifiable individuals, including those missed on some occasions. A closed-population capture-recapture model estimates a fixed population cardinality; models allowing individuals to enter or leave can instead describe survival and recruitment. The sampling assumptions determine which likelihood function is appropriate.

For the two-sample closed-population capture-recapture model, condition on the sample sizes $M$ and $C$. If the second sample selects $C$ distinct individuals from $N$ by simple random sampling, then the overlap $R$ has the hypergeometric distribution:Here $r$ is an integer and the probability mass function is zero outside these bounds. The binomial coefficients count choices of $r$ marked and $C-r$ unmarked individuals, divided by all equally likely size-$C$ subsets. The hypergeometric distribution has expected valueEquating the observed overlap with this expected value gives the Lincoln-Petersen estimator. The underlying sampling without replacement, probability mass function, and expected value are developed in Raymond P. Guiteras, MIT 14.30 hypergeometric distribution handout, §§1–4 and 8.

For the toy counts $M=20$, $C=25$, and $R=5$, hold the observations fixed and vary the unknown integer $N$. The likelihood function isValues below forty cannot contain the observed set union. Exact comparison of consecutive likelihoods gives two tied maximum-likelihood estimates, $N=99$ and $N=100$. The Lincoln-Petersen estimator returns $100$; the Chapman estimator returns $90$ and is a different rule for estimation.

The vertical scale is $L(N)/{\max }_{n\ge 40}L(n)$. This relative likelihood compares how well candidate statistical parameters explain the same observations; it is not a posterior probability. The displayed range $40\le N\le 300$ is only a plotting window. The likelihood function remains positive for every integer $N\ge 40$ and tends to zero as $N$ increases, so the curve has a continuing right tail. The data do not justify truncating statistical inference at three hundred. The conditional hypergeometric distribution formulation also appears in Louis-Paul Rivest and Mamadou Yauck, Small Sample Estimators for Two-way Capture Recapture Experiments, §3, equation (5).

Zero recaptures require care. With $R=0$ and positive $M,C$, the Lincoln-Petersen estimator has no finite ratio, but the capture-recapture model still gives a valid likelihood function:This likelihood function increases toward one as $N$ grows, so there is no finite maximum-likelihood estimate. For the same sample sizes twenty and twenty-five with zero overlap, forty-five distinct individuals would be observed. Lack of overlap supplies a lower bound but very weak information about large population cardinalities.

In more elaborate capture-recapture models, a first-recapture array $m_{ij}$ counts individuals released at occasion $i$ and next observed at $j$. A probability of survival ${\varphi }_j$ and a probability of detection $P_{j+1}$ contribute a no-observation continuation factor ${\varphi }_j(1-P_{j+1})$. The resulting likelihood function depends on both survival and detection; these probabilities need not be separately identifiable at the last observation interval. Missing-count EM for capture-recapture describes one way to fit such a model, and identifiability remains a separate issue from computing an estimator output.