Statistical modelling

Statistical modelling constructs, fits, checks, and interprets a statistical model for observed data.

A response variable is the outcome whose variation is described by a statistical model or whose change is compared across treatments. Its unit, measurement procedure and observation time must be specified. An outcome consisting of a feature count is one response per sketch, not one independent experimental response per feature.

Design of experiments chooses experimental units, treatments, replication, blocks in experimental design and randomization so that scientifically useful treatment contrasts can be estimated with meaningful standard errors. Allocation determines which measurements supply independent treatment information.

A split-plot design randomizes one factor to whole plots and another to subplots within each whole plot. Whole-plot treatment effects and subplot treatment effects therefore use different error ANOVA strata. Whole plots provide replication for the first factor; their subplots do not supply extra independent whole-plot replication. Within-whole-plot comparisons can remove a shared additive whole-plot variance component.

A factorial design includes every combination of specified factor levels. Balanced replication permits separate estimation of marginal effects and interaction terms. The factor allocation, not the number of subsamples, determines the appropriate error term for each effect.

A crossover design assigns each subject a sequence of treatments in successive periods. Subject blocking can reduce between-subject variation; balanced sequences can separate treatment from period effects. A carryover effect or irreversible change can invalidate a simple within-subject comparison.

A carryover effect is an effect of a previous treatment on a later response. Balanced transitions help prevent a treatment from being preceded disproportionately by one particular alternative, but do not ensure that carryover effects vanish or are separately identifiable in every model.

A completely randomized design assigns treatments directly to available experimental units using complete randomization, without blocking. Fixed group sizes can be enforced. The residual then includes uncontrolled between-unit variation.

A block groups experimental units expected to have similar background responses. Comparing treatments within blocks can reduce the variance of an estimator. Block allocation must be specified before responses are used to evaluate treatment effects.

A block design assigns treatments to experimental units partitioned into blocks. Its incidence counts $n_{ij}$ record how often treatment $i$ occurs in block $j$. Balance and connectedness determine which treatment contrasts are estimable and how much adjustment for blocks costs.

A row-column design has two crossed partitions of experimental units into rows and columns. Equal treatment proportions in every row and every column make treatment contrasts orthogonal to both centered block spaces. A repeated three-treatment arrangement in a six-by-six array can give a balanced example.

A randomized complete block design places every treatment in every block, with a common replication pattern, and uses independent within-block randomization. The usual basic version has one occurrence of each treatment per block; equal repeated occurrences give a replicated complete-block version. Such equal-proportion designs are orthogonal block designs.

An orthogonal block design has centered treatment-indicator vectors orthogonal to centered block-indicator vectors on the experimental units. With treatment replication $r_i$, block size $k_j$ and total $n$, this is equivalent to $n_{ij}=r_i{k}_j/n$. It permits additive block adjustment without changing treatment estimates.

Replication in experimental design assigns a treatment to multiple independently allocated experimental units. Repeated measurements on one unit can improve measurement precision without providing independent treatment replication.

A replicate is an independently allocated instance of a treatment on an experimental unit. Independent replicates permit estimation of response variation; subsamples within one unit do not supply additional independent treatment replication.

An experimental unit is a unit to which a treatment is assigned under the experimental allocation. A whole orchard can be an experimental unit even when responses are recorded separately on its trees. Different factors in a split-plot design can have different experimental units.

An observational unit is the object or occasion supplying a recorded response. Several observational units can belong to one experimental unit; counting them as independent treatment replicates creates pseudoreplication.

A fractional polynomial uses selected real powers of a positive covariate to model a smooth regression effect. Power zero denotes a logarithm, and a repeated power can generate a term $x^p\log x$. Such terms permit a wider family than ordinary integer-power polynomials, but their selection adds modeling uncertainty and requires validation.

A restricted cubic spline represents a smooth covariate effect by joined cubic pieces, with linear tails beyond the boundary knots. It allows nonlinear regression effects while controlling extrapolation. In a Cox proportional-hazards model, test the nonlinear basis coefficients jointly and report the fitted log-hazard or hazard-ratio curve with uncertainty.

Quantile regression models a conditional quantile of a response, rather than its conditional mean. Fitting by the check loss uses an asymmetric absolute-error penalty, so different values of $\tau$ describe different parts of the response distribution. Nonlinear models may use any identifiable continuous regression function, not only a linear predictor.

If the conditional error distribution function is continuous and strictly increasing with $\tau$-quantile zero, the conditional expected check loss is uniquely minimized at zero fitted displacement. If every incorrect parameter differs from the true regression function on an event of positive probability, averaging these nonnegative conditional risk differences gives strict identifiability of the true parameter. Integrable errors and bounded regression functions supply finite risks.

The check loss weights positive errors by $\tau$ and negative errors by $1-\tau$. It is convex, continuous and Lipschitz with constant $\max (\tau ,1-\tau )$. At $\tau =1/2$ it is half the absolute error. Its expected value elicits a quantile: asymmetry determines which probability level the fitted location targets.

For an integrable response with continuous strictly increasing distribution function $F$, the population risk for the check loss has derivative $F(q)-\tau$. Bounded difference quotients justify differentiation by dominated convergence. Integrating the derivative between the unique quantile and another location proves strict optimality. This proves identifiability without requiring a positive density.

Outcome propensities can vary between individuals because of their measured covariates. A linear regression can describe this variation by allowing its conditional mean to depend on those predictors.

A pairwise comparison model specifies the probability that one alternative beats another. Outcomes can be used to infer latent abilities through maximum likelihood estimation or another statistical method.

The Bradley-Terry model assigns positive abilities ${\theta }_i$ and the displayed pairwise winning probabilities. Multiplying all abilities by the same positive scalar leaves the probabilities unchanged, so identifiability requires a normalization. In log abilities ${\beta }_i=\log {\theta }_i$, the model has comparison probabilities obtained from the logistic function. Either parametrization induces the same ordering of players.

Let $W_i$ count observed wins and $n_{ij}$ count comparisons of players $i,j$. Differentiating the log-likelihood with respect to $\log {\theta }_i$ gives observed wins minus model-expected wins. Vanishing derivatives therefore give the displayed equations. One equation is redundant because abilities are identifiable only up to common scaling.

For a connected tree of comparisons with every observed win fraction strictly between zero and one, the Bradley-Terry model can fit every empirical edge probability exactly. Fix one positive strength to remove scaling ambiguity, then propagate strength ratios $p/(1-p)$ along tree edges. Edge log-ratios are independent real coordinates and each binomial log-likelihood term is strictly concave, proving existence and uniqueness of the normalized estimate. Cycles would impose extra compatibility relations.

For observed neighbor win fractions $p_0,\dots ,p_{n-1}\in (0,1)$ on the path graph $0,1,\dots ,n$, the normalized Bradley-Terry model estimate with ${\hat{\theta }}_n=1$ is ${\hat{\theta }}_i={\prod }_{k=i}^{n-1}{p}_k/(1-p_k)$. This is backward propagation of empirical odds along the path. Every edge proportion is fitted exactly, and strict concavity in edge log-ratios proves that this is the unique global maximum.

Orient the edges of a comparison tree and set ${\eta }_{ij}=\log {\theta }_i-\log {\theta }_j$. After anchoring one vertex strength, these edge quantities are unconstrained coordinates. The Bradley-Terry model probability is the logistic function of ${\eta }_{ij}$, and its binomial log-likelihood contribution is $m[p{\eta }_{ij}-\log (1+e^{{\eta }_{ij}})]$. Differentiating gives the fitted odds $p/(1-p)$.

For one win of player 1 over 2, one win of 3 over 1, and $k\ge 1$ wins of 2 over 3, the Bradley-Terry score equations force abilities proportional to $(r,1,r^2)$, where $r>0$ satisfies the displayed equation. Its left side is strictly increasing on $r>0$. For $k=1$, $r=1$ and all three estimates tie. For $k>1$, $r<1$, giving the ranking $2>1>3$.

In log abilities, the negative Hessian matrix is a weighted Graph Laplacian of the comparison graph. If that comparison graph is a connected graph and the abilities are finite, the quadratic form vanishes only for constant vectors. Thus the log-likelihood is strictly concave after fixing the common additive constant. If the directed graph of observed wins is a strongly connected directed graph, letting contrasts diverge forces at least one observed-win probability to zero, so the log-likelihood tends to negative infinity. A finite maximizer exists and is unique up to common scaling of abilities.

A fixed effect represents an unknown nonrandom coefficient in the specified mean model. In a Gaussian linear mixed model, $X\beta$ is the population mean, while random effects are latent draws from a distribution whose parameters are estimated. Treating a group coefficient as fixed or random concerns the modelling and inferential target, not whether its estimate changes across samples.

Geostatistics models spatial dependence for estimation and prediction. A mean or drift describes systematic variation, while a covariogram or semivariogram describes residual dependence. Kriging combines the two for spatial prediction.

Kriging predicts a spatial random field by a linear combination of observations chosen to minimize prediction variance subject to the appropriate mean constraints. Its different forms depend on whether the mean is known, an unknown constant, or an unknown regression drift.

For unknown drift $X\beta$, a full-column-rank design matrix $X$, a known positive-definite matrix observation covariance matrix $\Sigma$, and target drift row $x_0^T$, universal kriging minimizes prediction variance subject to $X^{T}w=x_0$. Equivalently, with $\hat{\beta }=(X^T{\Sigma }^{-1}X{)}^{-1}{X}^T{\Sigma }^{-1}z$, predict $x_0^T\hat{\beta }+c^T{\Sigma }^{-1}(z-X\hat{\beta })$. The first term is the estimated trend alone; the second is a correlated residual prediction.

For an unknown constant mean, ordinary kriging minimizes error variance subject to weights summing to one. This unbiasedness constraint distinguishes it from simple kriging.

For known zero mean, invertible observation covariance $\Sigma$, target covariance vector $c$ and target variance $v_0$, simple kriging predicts $c^T{\Sigma }^{-1}z$ with error variance $v_0-c^T{\Sigma }^{-1}c$. Completing the covariance quadratic form proves optimality. A multivariate normal distribution makes this the exact conditional mean and variance.

The covariogram is the displacement-dependent covariance $C(h)=\mathrm{Cov}[Z(s+h),Z(s)]$ of a second-order stationary field. It is an even positive-definite kernel. A semivariogram determines differences $C(0)-C(h)$, but not an arbitrary constant covariance component.

An intrinsically stationary random field has constant mean and finite increment variances depending only on displacement. Its semivariogram is $\gamma (h)=\frac{1}{2}\mathrm{Var}[Z(s+h)-Z(s)]$. Second-order stationarity implies this property, but the converse can fail: subtracting the value at a fixed origin from a stationary field preserves increments and makes marginal variance vary with location.

For an intrinsically stationary random field, the semivariogram is $\gamma (h)=\frac{1}{2}\mathrm{Var}[Z(s+h)-Z(s)]$. It is even and vanishes at zero. With constant mean, a second-order stationary field has $\gamma (h)=C(0)-C(h)$. An isotropic model writes it as a function of nonnegative distance.

For a distance bin $B$, the empirical semivariogram is $\hat{\gamma }(B)=(2|B|{)}^{-1}{\sum }_{(i,j)\in B}(z_i-z_j{)}^2$. A nonconstant drift adds half its squared pairwise differences to the expected raw semivariogram. Residual semivariograms estimate spatial dependence after adjusting for the drift.

The exact range is a separation beyond which the semivariogram reaches its sill. A Gaussian semivariogram with nonzero structured variance approaches its sill asymptotically and therefore has infinite exact range. Software range parameters may instead specify a scale.

A practical range specifies a distance at which a chosen fraction of the structured sill is reached. For the Gaussian semivariogram ${\sigma }^2(1-e^{-(h/a{)}^2})$, the 95-percent practical range is $a\sqrt{\log 20}$, obtained by solving $e^{-(h/a{)}^2}=0.05$.

The sill is the large-distance level of a semivariogram when that limit exists. For covariance tending to zero, the sill is the total marginal variance; a nugget plus a structured component has total sill ${\tau }^2+{\sigma }^2$. A nonzero constant covariance component is invisible to the semivariogram.

The nugget is the discontinuity of a distance semivariogram at zero. Independent location-specific noise of variance ${\tau }^2$ contributes covariance ${\tau }^2$ at zero displacement and zero covariance at nonzero displacement, so its semivariogram is ${\tau }^2{1}_{\{h\ne 0\}}$.

If $W(s)$ is stationary, then $Z(s)=W(s)-W(0)$ has exactly the same increments and semivariogram, but $\mathrm{Var}Z(s)=2{\gamma }_W(s)$ is generally nonconstant. Thus existence of a stationary covariance model for a semivariogram does not imply stationarity of every process with that semivariogram. Adding an independent random constant also leaves the semivariogram unchanged while changing the covariogram.

For $\varphi >0$ and nonnegative variances, $\gamma (h)={\tau }^2{1}_{\{h\ne 0\}}+{\sigma }^2(1-e^{-{\varphi }^2\Vert h{\Vert }^2})$ admits covariance $C(h)={\sigma }^2{e}^{-{\varphi }^2\Vert h{\Vert }^2}+{\tau }^2{1}_{\{h=0\}}$. The Gaussian kernel is positive definite because it is the Fourier transform of the density of a Gaussian distribution. Adding independent white noise supplies the nugget effect. Its scale is $a=1/\varphi$. For ${\sigma }^2>0$, its exact range of a semivariogram is infinite. If ${\sigma }^2=0$, it is a pure nugget effect with no nontrivial structured range.

A saturated model has enough independently varying mean parameters to match every observed response or response cell. In individual Bernoulli distribution data its fitted probabilities are $p_i=y_i$, and its maximum log-likelihood is zero under $0\log 0=0$. It defines the reference in a binomial deviance.

A hurdle model separately models whether a count is zero and, conditional on being positive, uses a zero-truncated count distribution. Unlike zero inflation, the positive-count component cannot generate zeros.

Zero inflation augments a count distribution with an additional mass at zero. An observed zero can arise either from the structural-zero component or from the ordinary count component; the latent source is not observed.

A structural-zero component in a count mixture produces a count identically equal to zero. If its mixture weight is $\pi$ and the ordinary component also assigns mass $p_0$ to zero, the observed zero probability is $\pi +(1-\pi )p_0$. Thus an observed zero does not reveal the latent component membership. A shared component for a repeated count profile makes the whole profile zero and induces dependence between its components.

With structural-zero probability $\pi$, size $r>0$ and count mean $\lambda$, the model assigns $\pi +(1-\pi )(r/(r+\lambda ){)}^r$ to zero and $(1-\pi )f_{\mathrm{NB}}(y;r,\lambda )$ to positive counts. Its expectation is $(1-\pi )\lambda$ and its variance is $(1-\pi )(\lambda +{\lambda }^2/r)+\pi (1-\pi ){\lambda }^2$, by the law of total variance. A logarithmic link function can relate the count-component mean to predictors.

Let $B=0$ with probability $\pi$, and otherwise let $B$ have a Gamma distribution with mean one and variance $\tau$. Given $B$, counts $Y_j$ are conditionally independent Poisson random variables with means $B{\mu }_j$. For $q=1-\pi$, the law of total variance and law of total covariance giveEach component has a zero-inflated negative binomial distribution, but the components are not independent after marginalizing $B$. The zero component is shared by the whole profile. At $\tau =0$, interpret the positive Gamma component as a point mass at one.

If $\mathbb{E}(Y_j\mid B,x)=B\exp ({\beta }_0+{\beta }^{T}x+{\alpha }_j)$ and the distribution of $B$ is the same at each observation time with finite positive mean, integrating out $B$ adds $\log \mathbb{E}B$ to the marginal log-mean intercept. The conditional positive-$B$ mean ratio and the marginal mean ratio both equal $e^{{\alpha }_j-{\alpha }_k}$. This is a property of the multiplicative logarithmic link function; it need not hold for nonlinear links such as the logit link.

For known size $r$ and ${\lambda }_i=e^{x_i^T\beta }$, the expectation-maximization algorithm gives structural-zero responsibility $t_i=0$ for positive counts and $t_i=\pi /[\pi +(1-\pi )(r/(r+{\lambda }_i){)}^r]$ for zero counts. This is Bayes theorem. Maximizing the expected complete-data log-likelihood gives ${\pi }_{\mathrm{new}}=n^{-1}{\sum }_i{t}_i$ and a weighted negative binomial regression with weights $1-t_i$. The objective separates into a Bernoulli mixing term and a weighted count term, which proves the update.

An estimator is a function of observed data used to estimate a parameter or a specified function of parameters. Its definition cannot involve the unknown parameter. Its bias, variance, and expected loss describe its performance under repeated sampling.

A U-statistic averages a symmetric function of $m$ distinct sample observations over all unordered $m$-tuples. For independent and identically distributed observations it is an unbiased estimator of $\mathbb{E}H(X_1,\dots ,X_m)$. Omitting repeated indices can remove diagonal contributions in quadratic plug-in estimators.