Analysis of variance

Analysis of variance decomposes variation into contributions associated with model terms and residual error. For nested normal linear models, differences of residual sums of squares give sequential or partial F-tests.

Multivariate analysis of variance compares mean vectors of several groups while accounting for correlations between their response variables. In the one-way normal model, independent observations $Y_{ri}\sim N_p({\mu }_r,\Sigma )$ share a positive-definite covariance matrix. Put $E={\sum }_{r,i}(Y_{ri}-{\bar{Y}}_r)(Y_{ri}-{\bar{Y}}_r{)}^T$ and $H={\sum }_r{n}_r({\bar{Y}}_r-\bar{Y})({\bar{Y}}_r-\bar{Y}{)}^T$. Under equal means, these matrices have independent Wishart distributions with degrees of freedom $N-g$ and $g-1$, respectively. This follows by separating the Gaussian data into orthogonal within-group and between-group projections. The Wilks lambda statistic tests equality of the means using both matrices, rather than independently testing each coordinate.

For groups with sizes $n_r$, means ${\bar{x}}_r$, and overall mean $\bar{x}$, define $W={\sum }_{r,i}(x_{ri}-{\bar{x}}_r)(x_{ri}-{\bar{x}}_r{)}^T$ and $B={\sum }_r{n}_r({\bar{x}}_r-\bar{x})({\bar{x}}_r-\bar{x}{)}^T$. Expanding $x_{ri}-\bar{x}=(x_{ri}-{\bar{x}}_r)+({\bar{x}}_r-\bar{x})$ gives total scatter $T=W+B$ because each within-group residual sum vanishes. Both matrices are positive semidefinite, and $\mathrm{rank}B\le \min (p,g-1)$. They separate within-group noise from between-group mean separation.

For nonsingular within-group scatter $E$ and between-group scatter $H$, the statistic is ${\prod }_j(1+{\theta }_j{)}^{-1}$, where ${\theta }_j$ solve the generalized eigenvalue problem $Hv=\theta Ev$. It is small when the group means differ strongly relative to within-group variation. Maximizing a common-covariance normal likelihood under equal and unrestricted means gives the likelihood ratio ${\Lambda }^{N/2}$. Its determinant factors cancel under any nonsingular change of response coordinates. For one response it is equivalent to the analysis of variance F-test.

An additive two-factor Gaussian model separates row and column effects. With one observation in each of $rc$ cells, constraints fixing one row and column effect leave $r+c-1$ mean parameters and $(r-1)(c-1)$ residual degrees of freedom. Balanced least squares gives fitted values equal to row mean plus column mean minus grand mean. With no cell replication, unrestricted interactions are confounded with residual error; the additive interpretation requires an assumption that those interactions are absent or negligible.

In a fully balanced crossed design, centered indicator columns for the two factors are orthogonal because each pair of levels occurs equally often. Missing cells can change the level-pair counts so that the displayed cross-product becomes nonzero. Then sequential sums of squares depend on term order, although the final additive fit is unchanged. Testing either factor conditional on the other requires its extra sum of squares from the appropriate reduced model.

For observed cells with means $a_i+b_j$, form a bipartite graph joining row $i$ to column $j$ when that cell is observed. If all levels occur, the design rank is $r+c-k$, where $k$ is the number of connected components. Indeed a coefficient perturbation in the kernel satisfies $a_i=-b_j$ on every observed edge, and hence has exactly one freely chosen constant on each component. A connected design has rank $r+c-1$ and is identifiable after a single reference constraint. Missing cells destroy the balanced-design orthogonality of row and column effects, so sequential sums of squares can depend on order.

A two-way analysis of variance with the full interaction and one observation in each observed factor cell has one independently varying fitted cell mean per observation. A cell-indicator design matrix has full row matrix rank, even when some combinations are missing, so the model is a saturated statistical model. Its residual sum of squares is zero and its residual degrees of freedom are zero. Missing combinations make some coefficients nonidentifiable; they do not restore a pure-error estimate. Standard errors and an ordinary residual-based F-test for interactions therefore cannot be recovered merely by fitting this larger model.

For an ordered sequence of nested linear regression models $M_0\subset M_1\subset \cdots$, the sequential sum of squares for the $j$th added term is $\mathrm{RSS}(M_{j-1})-\mathrm{RSS}(M_j)$. These are also called type-I sums of squares. They depend on term order when the design is not orthogonal. Dividing by the number of newly added coefficients and the full model's residual mean square gives the sequential F-test. A covariate added first is tested before adjustment for later terms, whereas a treatment added after the covariate is tested with that covariate already included.

For an orthogonal projection matrix $P$, a sum of squares in ANOVA is $Y^{T}PY=\Vert PY{\Vert }^2$. Mutually orthogonal projections yield an additive decomposition of total squared response magnitude. Removing the grand mean gives the corrected total.

A mean square in ANOVA divides a sum of squares in ANOVA by its positive number $d$ of statistical degrees of freedom. If the corresponding residual subspace has scalar error covariance $\lambda I$, its expectation is $\lambda$. Its scale depends on whether raw observations or group means were projected.

An ANOVA stratum is an orthogonal response subspace representing one level of experimental variation, such as between blocks or within blocks. In a covariance model scalar on each such subspace, its eigenvalue supplies the common variance scale for its projected coordinates. Treatment and residual sums of squares in ANOVA must be compared within appropriate strata.

The within-block ANOVA stratum contains response vectors whose coordinates sum to zero separately in every block. For $b$ blocks of size $k$ its dimension is $b(k-1)$. A shared additive block effect cancels in every vector in this subspace.

In a complete two-factor design with equal cell replication, centred factor indicator columns for different factors have zero cross-products. Consequently the two main effects in an additive normal linear model are orthogonal, and their extra sums of squares do not depend on the order in which these main effects are entered. An omitted interaction term can still require a separate adequacy check.