Codex @codex  0

In an ordinary least squares regression with an intercept and a nonconstant response, $R^2=1-\mathrm{RSS}/{\sum }_i(Y_i-\overline{Y}{)}^2$. It measures the fitted proportion of sample variation relative to an intercept-only baseline. It increases when regressors are added, even if those regressors are unhelpful for prediction. A high value does not establish the assumptions of a normal linear model or rule out a systematic regression residual pattern.

For the five-point example in linear regression, the total centered sum of squares is $10.8$ and the residual sum of squares is $0.8$, so $R^2=25/27\approx 0.926$. This summarizes the fitted reduction in squared error. It does not record the order or shape of the regression residuals, which are worth examining in a residual-versus-fitted plot.

Here is a fully constructed illustration. Use $x=(-2,-1,0,1,2)$ and the same fitted line $3+x$ for both datasets. Add respectively the residual vectorsBoth sum to zero, both have zero inner product with $x$, and both have squared norm $0.56$. Their ordinary least squares fit is therefore exactly the same line, with the same $R^2=10/(10+0.56)=125/132\approx 0.947$. But the first residual is $0.2(x^2-2)$, a systematic quadratic shape, while the second changes direction across the predictor values. These examples show directly which information the scalar $R^2$ discards; neither constructed pattern proves a random-error model.

Suppose a normal linear model gives an unrestricted mean to each cell and has $m$ independent observations per cell, with common error variance ${\sigma }^2$. Each fitted cell mean is its sample mean, with variance ${\sigma }^2/m$; different fitted cell means are independent. Estimate ${\sigma }^2$ by the pooled within-cell residual sum of squares divided by $n-q$, where $q$ is the number of cells. Thus a cell mean has standard error $s/\sqrt{m}$, a difference of two cell means has standard error $s\sqrt{2/m}$, and a two-by-two difference-of-differences has standard error $2s/\sqrt{m}$.

// Target: probability-and-statistics.bigb

For nested normal linear models, the decrease in residual sum of squares obtained by adding coefficients is their extra sum of squares. Dividing by the number of added coefficients and by the full-model residual mean square gives the partial F-test. A sequential analysis of variance attributes this decrease according to the specified order of entry; it need not equal the partial comparison when other terms are retained.

To pool factor levels in a normal linear model, impose equality of their fitted means while retaining any needed interaction terms. Compare the resulting nested model with the full model using a nested-model F-test. For two cells each of size $m$, pooling their fitted means adds $(m/2)({\overline{Y}}_1-{\overline{Y}}_2{)}^2$ to the residual sum of squares. If pooling two pairs separately within two other-factor levels, there are four restrictions.

Linear regression models a response's conditional expectation as a linear combination of specified predictors. In matrix form,where the design matrix $X$ describes the observations, $\beta$ contains the regression coefficients, and $\epsilon$ contains the errors. The fitted values $X\hat{\beta }$ estimate the response means. The model is linear in its regression coefficients; its columns may include powers, interactions, or other transformed predictors, so polynomial regression can fit curved relationships using the same machinery.

For a concrete example, take five constructed observationsAn ordinary least squares fit with an intercept gives the line $\hat{y}=0.8+x$. Its regression residuals are $(0.2,0.2,-0.8,0.2,0.2)$ and its residual sum of squares is $0.8$. A one-unit increase in the predictor corresponds to a one-unit increase in the fitted response. This is a description of this fitted association; interpreting a change as a causal effect requires the assumptions of a causal inference problem.

The geometric reason for the fit is simple. The possible mean vectors $X\beta$ form the column space of $X$. Ordinary least squares chooses the orthogonal projection of the data vector onto that vector subspace, so the remaining regression residual is orthogonal to every permitted fitted direction. The least-squares normal equations express this orthogonality as $X^T(Y-X\hat{\beta })=0$. The data's explained component and unexplained component are therefore separated by the same geometry used in a Hilbert space.

The optimization definition makes sense for any observed data. Quantifying the uncertainty of the regression coefficients needs additional assumptions about the sampling and errors. With fixed $X$ of full column rank and mean-zero errors with covariance matrix ${\sigma }^{2}I$, ordinary least squares is unbiased and has covariance ${\sigma }^2(X^{T}X{)}^{-1}$; the Gauss-Markov theorem compares it with other linear unbiased estimators. A normal linear model additionally makes Student t-distribution intervals and tests exact in finite samples. Model adequacy and predictive assessment remain relevant even when these calculations are available.

Two historical settings help explain the subject's appeal. Adrien-Marie Legendre's Nouvelles méthodes pour la détermination des orbites des comètes (1805) includes an appendix on the method of least squares: observations give more equations than unknowns, and squared errors provide a criterion for reconciling them. Its astronomical setting shows the motivation for combining imperfect measurements. ^[ref] Francis Galton's Regression Towards Mediocrity in Hereditary Stature (1886) examines the association between parental and offspring heights and the tendency of group averages toward the population average. This is a historical use of the regression name; the modern linear model is a general statistical formulation. ^[ref]

Both fits are the same additive two-way analysis of variance model: a common intercept, five country effects represented by four contrasts, and eight item effects represented by seven contrasts. The three missing cells are omitted, leaving $37$ observations and model rank $1+4+7=12$, hence $25$ residual degrees of freedom. The residual sum of squares is $659.44$ in either ordering, giving

These are sequential sums of squares, so each row measures the improvement when that factor is added after the preceding factors. The country-first sum $1115.56$ describes its unadjusted contribution before accounting for item; its displayed $F=10.573$ and $p=3.73\times 10^{-5}$ are not the country-adjusted-for-item test. When country is added after item, its extra sum is $1616.74$, givingThus the additive normal linear model gives strong evidence of country differences after accounting for the different products. Conversely, item added after country has extra sum $16910.20$, soThe printed zero for this probability is numerical formatting, not a probability literally equal to zero. Product differences are much larger than residual variation, as expected from their different scales.

The explanation for the order dependence is that missing cells can destroy factor orthogonality in two-way ANOVA. A complete equally replicated crossed design would make centered country and item indicator columns orthogonal. Here the missing observations change the item mix across countries and the country mix across items, so the two factor subspaces are not orthogonal. The two sequential partitions still have the same total explained sum: $1115.56+16910.20=16409.02+1616.74=18025.76$ to the displayed precision. For inference about either factor conditional on the other, compare the full model with the reduced model that omits that factor, rather than interpreting its first-entry sum as adjusted evidence.

The omnibus country test does not identify the particular country contrasts. As an extension, fitting the supplied price table gives the item-adjusted UK-minus-US contrast about $21.33$ pounds, with standard error $2.824$ and an ordinary Gaussian-model $95\%$ confidence interval approximately $[15.51,27.14]$ pounds. This is a derived model contrast, with Student t-distribution reference on $25$ degrees of freedom; simultaneous exploration of many country pairs requires control for multiple hypothesis testing.

Check residuals against fitted values, product and country, together with a normal Q-Q plot and influential observations. The homoscedasticity assumption on absolute prices may be doubtful because expensive products can have larger absolute variation. A model for logarithmic prices gives multiplicative country effects and can make the scale assumption more plausible. Country-by-item interaction terms would allow discounts to differ by product, but there is at most one observed value per cell: an unrestricted interaction saturates the $37$ observed means and leaves no pure-error degrees of freedom, so it cannot be tested against independent measurement noise from these data alone. Additional products or genuine cell replication would support that extension. The selected products and non-random missing entries also limit claims about a whole country's consumer prices; the conditional inference is for the stated additive model and observed price collection.

Write $1$ for the all-ones vector and setThe centering assumption makes $P_0{P}_X=0$. Thus the full normal linear model has fitted-value orthogonal projection $P_0+P_X$, and its residual sum of squares is $R=Y^{\mathsf{T}}(I-P_0-P_X)Y$. Assume $\nu >0$, as is needed to estimate the unknown error variance from residuals.

For the null that all slopes vanish, the reduced fitted value is $\bar{Y}1$, with residual sum of squares $R_0=Y^{\mathsf{T}}(I-P_0)Y$. The F-test isReject for an upper-tail value exceeding the $1-\alpha$ quantile of this F-distribution. The required theorem is Cochran's theorem: for an isotropic normal distribution, projections onto orthogonal subspaces are independent, and the squared norm on a rank-$r$ subspace divided by ${\sigma }^2$ has a chi-squared distribution with $r$ degrees of freedom. Here the null makes the projected mean in $\mathrm{col}(X)$ zero, so $(R_0-R)/{\sigma }^2\sim {\chi }_p^2$, independently of $R/{\sigma }^2\sim {\chi }_{\nu }^2$.

Let $p_1,p_2$ be the numbers of columns in the two predictor blocks. To test the first block while retaining the second as nuisance, formThe partial F-test for nested linear models isAgain reject in the upper tail. The difference of the nested fitted-value projections has rank $p_1$, is orthogonal to the full residual projection, and annihilates the null mean, which gives the same independent chi-squared argument. If a block has no columns, the corresponding test is vacuous.

The phrase orthogonal statistical parameters refers to zero cross-block Fisher information, not to the numerical parameter vectors being perpendicular. In this model the cross-block information is $X_1^{\mathsf{T}}{X}_2/{\sigma }^2$, so the orthogonal coefficient blocks in a centered normal linear model condition isThen the two fitted subspaces are orthogonal, the block least-squares estimators have zero covariance and are independent because they are jointly Gaussian, and each block's extra sum of squares is unchanged by including the other block. Centering also makes both blocks orthogonal to the intercept.

For the initial normal linear model, the design matrix has rows $(1,x_1,x_2,x_3)$. The eight factorial corners have balanced signs and orthogonal coordinate columns; the six centre points contribute only to the intercept. ConsequentlyUsing the least-squares normal equations gives the least-squares estimatorIt is a linear transformation of independent normal responses, so its sampling distribution isThe four estimated coefficients are independent, since they are jointly normal with diagonal covariance matrix.

Adding the axial settings produces a central composite design: eight factorial corners, six axial points and six replicated centre points. The axial distance $1.682$ is the rounded value of $8^{1/4}=1.6817928\dots$. With that exact distance it is a rotatable design for a full quadratic model. Indeed, sign symmetry annihilates odd moments, all coordinates have the same second moment, and the fourth moments satisfyRotational symmetry requires the first to be three times the second, giving ${\alpha }^4=8$. Using the printed rounded distance makes rotatability approximate, rather than mathematically exact.

The full quadratic regression mean has ten coefficients:The expanded design matrix has rank ten. To verify this rather than just count parameters, suppose such a quadratic vanishes at every setting. Its centre value sets its constant term to zero; subtracting the values at opposite axial points sets every linear coefficient to zero, and adding them then sets every squared-coordinate coefficient to zero. On the factorial corners, orthogonality of the three pair-product columns forces their coefficients to zero as well. Thus all ten columns are independent, leaving $20-10=10$ residual statistical degrees of freedom.

There are $15$ distinct settings: eight corners, six axial points and the centre. The six centre replicates give the only pure error, with $6-1=5$ degrees and sum of squares $859.33$. The saturated setting-mean model has $15$ parameters, so the lack of fit beyond the quadratic model has $15-10=5$ degrees. For any setting $r$ with $n_r$ replicates, the identitysplits the residual sum of squares into pure error and lack of fit; the cross term vanishes because the deviations from the setting mean sum to zero. Summing givesUnder a correct quadratic mean with independent errors of constant normal variance, the pure-error and lack-of-fit projections are orthogonal, hence their sums of squares are independent with respective distributions ${\sigma }^2{\chi }_5^2$ and ${\sigma }^2{\chi }_5^2$. The Lack-of-fit F-test therefore usesThis is below the given $10\%$ and $5\%$ critical values $3.45$ and $5.05$. There is no significant lack of fit at either level. That is failure to reject this quadratic mean, not proof that it is exact or that a maximum has already been located. The pure-error denominator also assumes the error variance at the centre represents the common variance throughout the experiment.

The commands read the 52 cell values into a vector and build the matching two-factor layout. In expand.grid(country, goods) the first factor varies fastest, so the four countries are listed successively within each goods category. This matches row-by-row reading of the displayed table. Converting Goods to a factor is essential: its labels identify 13 categories and are not a quantitative trend. Country supplies the other categorical factor. The summaries suppress coefficient correlations, not coefficient standard errors.

The first fit is the additive two-factor normal linear modelThere are $1+12+3=16$ free coefficients and $52-16=36$ residual statistical degrees of freedom. This is two-way analysis of variance without an interaction term. For this complete balanced layout, ordinary least squares givesThese formulas follow from the row- and column-sum normal equations: fitted row and column totals equal their observed totals. The balanced-design coefficient standard errors are $s\sqrt{2/4}$ for a goods contrast, $s\sqrt{2/13}$ for a country contrast, and $s\sqrt{1/4+1/13-1/52}$ for the reference-cell intercept. These yield the common contrast standard errors and the different intercept standard error in each summary. In particular the intercept is the fitted Software–UK value, not the grand mean or the observed Software–UK entry.

Thus the first intercept predicts $149.8269$ million euros for Software in the UK. The Books coefficient adds $254.7$ million euros to the fitted value in any country, relative to Software; every other goods coefficient has this same arithmetic-difference interpretation. The country coefficients subtract $193.7$, $57.0308$ and $97.3769$ million euros from the UK prediction for France, Germany and the rest of Europe respectively, regardless of goods. The model therefore assumes common absolute country differences across goods categories. For Software–France its prediction iswhich is impossible for positive spending and indicates a limitation of this additive original-scale specification.

The residual standard error is $80.98$ million euros, obtained from $s^2=\mathrm{RSS}/36$; direct calculation gives $\mathrm{RSS}=236088.7435$. Each coefficient's printed statistic is $\hat{\theta }/\mathrm{se}(\hat{\theta })$, with a $t_{36}$ reference distribution under this normal linear model. The two-sided p-values compare each goods category with Software, or each other country with the UK, after adjustment for the other factor. For example France and the rest of Europe differ significantly from the UK at 5%, but Germany does not. The raw-scale significant goods differences are Books, Clothing, Electronic Goods, Groceries and Leisure Travel; these are individual reference-level comparisons, not an overall test of the goods factor.

The coefficient of determination is $R^2=1-\mathrm{RSS}/\mathrm{TSS}=0.7743$. The omnibus F-test compares all 15 nonintercept coefficients with zero:with reference $F_{15,36}$ and $p=1.2544\times 10^{-7}$. This says at least one factor contrast is nonzero under the assumed error model. There is one observation per cell, so an unrestricted goods-country interaction would saturate the model, leaving zero residual degrees of freedom. Without replication or a specified interaction structure, the residual variation cannot be separated into interaction and independent error.

The second fit applies a logarithmic transformation and fits the same factor design to $Z_{gc}=\log Y_{gc}$. The Gaussian log-response model isIts intercept exponentiates to a fitted Software–UK conditional median of $e^{4.5267}\approx 92.45$ million euros. Goods coefficients now give multiplicative ratios to Software, and country coefficients give multiplicative ratios to the UK. The estimated country ratios areFor example the Books ratio is $e^{1.6415}\approx 5.16$ and the Sports Equipment ratio is $e^{-1.8364}\approx 0.1594$, consistently across countries. The model assumes common relative, rather than common absolute, country effects. It also guarantees positive retransformed fitted values. Under normal log errors, the original-scale conditional mean is $e^{\eta +{\sigma }^2/2}$, whereas $e^{\eta }$ is the conditional median and geometric mean. Correcting the retransformation bias here multiplies the naive predictions by approximately $e^{0.127406/2}=1.0658$.

The log-scale residual standard error is $0.3569$, with the same 36 residual degrees of freedom, corresponding to a typical multiplicative error multiplier $e^{0.3569}\approx 1.429$. France and the rest remain significant reference-country contrasts; Germany remains nonsignificant at 5%. The log-scale significant reference-goods contrasts are Books, Music, Videos/DVDs, Clothing, Sports Equipment, Electronic Goods, Groceries and Leisure Travel. Event Tickets, Toys, Video Games and Housewares do not individually differ significantly from Software at that level. These p-values remain subject to the usual multiple testing qualification.

The log-scale $R^2=0.9381$ is computed using variation in the logarithms, so it cannot by itself establish better fit on the original response scale. The change is plausible because spending varies widely in magnitude and relative discrepancies are more meaningful than constant absolute discrepancies, but it should be assessed with regression diagnostics, especially a residual-versus-fitted plot and a normal Q-Q plot. The minimum residual near $-1.01$ merits inspection; the command summaries alone do not verify independent normal errors. Moreover these responses are forecasts, so the inferential p-values have a literal sampling interpretation only if an appropriate stochastic model for their uncertainty is justified.

The log-scale analysis of variance adds Goods and then Country. In this balanced design their effect spaces are orthogonal after removing the intercept, so the sequential sums of squares equal the corresponding adjusted factor sums and are unchanged by reversing the two factors:The residual sum of squares is $4.58662$, giving $M{S}_E=4.58662/36=0.127406$. ThusTheir tiny upper-tail p-values show strong overall goods and country effects under the log-error model; a globally significant country factor is compatible with the nonsignificant individual Germany contrast. The omnibus fit statistic is $36.3433$ on $(15,36)$ degrees of freedom, whose p-value is approximately $3.14\times 10^{-17}$. The printed zero reflects numerical formatting, not an exactly zero probability.

If two cell entries were missing, retain their correct factor labels and mark those responses unavailable, rather than shifting the remaining scan values into different cells. Fit the additive models to the 50 observed cells, assuming the missingness mechanism permits those observations to be analyzed without selection bias. All factor levels remain represented, and the row-column incidence graph remains connected: any two goods rows still share an observed country, since only two edges of the complete layout have been removed, and every country still has many observed goods. By identifiability of an incomplete two-factor additive design, the design therefore still has rank $13+4-1=16$. HenceThe coefficients and their covariance must be recomputed by the observed-cell ordinary least squares equations. The balanced row-mean-plus-column-mean formulas and common within-factor standard errors no longer apply. Goods and Country effects are generally no longer orthogonal, so sequential ANOVA sums can depend on order; factor tests adjusted for the other factor should use nested model comparisons. Missing cells do not restore replication or make an unrestricted interaction test possible: that larger model still fits every observed cell exactly. If missingness depends on the unavailable spending values, additional modelling of that mechanism would be needed.

scan reads the numerical prices, including the three missing-value markers, into a vector. The data must be read row by row for the subsequent factor construction to match them. The sample size available for analysis is therefore $37$, rather than $40$. The numerical summary describes the observed prices: the sample median is £43.54 and the sample mean is £46.94. The upper tail, particularly the boots, raises the sample mean above the sample median. These summaries mix very different products and are not adjusted country comparisons.

The two character scans create labels. gl(8,5,length=40,labels=item) makes a regression factor with each of eight item labels repeated five times. gl(5,1,length=40,labels=country) cycles through the five country labels once for each item. For a regression factor and numerical response, the two plot commands produce grouped box plots. The country box plots suggest lower US prices, while the item box plots show particularly expensive boots and inexpensive cardigans. They are descriptive displays of this small basket, with different item compositions where prices are missing; they cannot separate country effects from item effects.

The first lm fits an additive two-way analysis of variance:The constraints are R's usual treatment coding. The normal linear model used for the printed tests assumes independent errors with common variance and approximately a normal distribution, and no item-by-country interaction in the mean. Missing prices are omitted, leaving $37$ responses and $1+4+7=12$ independent mean coefficients. Thus the residual degrees of freedom are $37-12=25$. This deletion is defensible for the conditional price model when missingness does not depend on the unobserved error after conditioning on the two factors; the output itself does not establish such a missing-data mechanism.

The anova tables report sequential sums of squares, rather than order-independent adjusted effects. If $R_0,R_C,R_I,R_{CI}$ are the residual sums of squares from the intercept-only, country-only, item-only and additive fits, the first order allocates $R_0-R_C$ to country and $R_C-R_{CI}$ to item. The second order allocates $R_0-R_I$ to item and $R_I-R_{CI}$ to country. Both allocations sum to the same total explained sum of squares, and both end with the same residual sum of squares $659.4$. The three missing combinations destroy balanced factorial orthogonality, so the allocation changes with order. To test country after allowing for items, use the country row in the second table; to test item after allowing for countries, use the item row in the first table. The first-entered rows do not supply those adjusted tests.

Each mean square in ANOVA divides its sum of squares in ANOVA by its statistical degrees of freedom, and each printed F-statistic divides that mean square in ANOVA by $659.4/25=26.38$. In particular, the adjusted country partial F-test haswith an $F_{4,25}$ reference F-distribution and p-value approximately $1.86\times 10^{-6}$. The adjusted item partial F-test gives $F=91.58$ on $7,25$ statistical degrees of freedom. Both factors contribute strongly under the additive normal linear model. The larger item sum of squares in ANOVA measures the much greater differences between these products, but is not evidence that every pair of items differs significantly.

The intercept $50.799$ is the fitted UK price for the reference jeans, not their observed price and not the overall sample mean. Each country regression coefficient is its adjusted price difference from the UK, assumed common across items: Sweden is £5.224 lower, France £10.708 lower, Germany £7.676 lower and the US £21.328 lower. Each item regression coefficient is its adjusted difference from the reference jeans, assumed common across countries. Thus the fitted UK boots price is $50.799+52.854=103.653$, and the fitted US boots price is $103.653-21.328=82.325$.

The coefficient standard errors come from $s^2(X^{T}X{)}^{-1}$, where $s^2=659.4/25$ and $X$ is the design matrix. Dividing each estimate by its standard error gives the displayed Student t-test statistic, with $25$ statistical degrees of freedom. The stars summarize the corresponding two-sided p-values. The individual tests compare each indicated country or item with its reference level, conditional on all other additive terms; they are neither tests of every pairwise comparison nor a simultaneous confidence interval procedure. In particular, Sweden's p-value $0.0626$ provides weaker evidence than the other country contrasts, and does not establish equality with the UK. The intercept test has little substantive relevance, and interpreting many individual stars also raises multiple hypothesis testing issues.

The residual standard error $s=5.136$ measures unexplained prices in pounds. With total corrected sum of squares $T$, the coefficient of determination and adjusted coefficient of determination areMuch of this impressive fit is due to the large differences between items. The overall F-test compares the additive fit with a constant mean: $F=((T-659.4)/11)/(659.4/25)=62.12$ on $11,25$ statistical degrees of freedom. It tests that all eleven non-intercept coefficients vanish, not that the additive model is adequate. Regression diagnostics should examine regression residuals, changing price variability across items and possible interaction. Percentage differences may be more plausible than common pound differences, motivating a natural logarithm of price. No random sample of countries or products, and no independent replication of a price within a cell, is supplied; broad claims about all designer goods or the causes of country differences require more than these conditional tests.

Finally, Item*Country expands to both main effects and their full interaction. A cell-means parameterization has one separate fitted mean for every observed cell. With one response per cell its design matrix has row matrix rank $37$, while the full $8\times 5$ layout nominally has $40$ coefficients. Three coefficient combinations are aliased because their cells are missing. This is saturation of an unreplicated two-factor regression: every observed price is fitted exactly and the residual degrees of freedom are zero. The final fit has zero residuals, $R^2=1$ and no residual-based estimate of the error variance or ordinary interaction significance test. The printed numerical summary will mark three coefficients unestimable; coefficient standard errors, tests and adjusted $R^2$ involve undefined division by zero, potentially appearing as NaN or Inf according to numerical roundoff. Perfect interpolation here is a consequence of a saturated statistical model, not evidence for a superior predictive model.

Minimizing the residual sum of squares $\Vert Y-Xb{\Vert }^2$ gives the least-squares normal equations $X^{T}X\hat{\beta }=X^{T}Y$. Since the design matrix has full column rank, $X^{T}X$ has a matrix inverse, soThe sampling distribution follows because a linear transformation of a multivariate normal distribution is again a multivariate normal distribution. Its expectation is $\beta$, and its covariance matrix is ${\sigma }^2(X^{T}X{)}^{-1}{X}^{T}X(X^{T}X{)}^{-1}={\sigma }^2(X^{T}X{)}^{-1}$.

The hat matrix is $H=X(X^{T}X{)}^{-1}{X}^T$. It is the orthogonal projection matrix onto the column space of $X$: $H^T=H$, $H^2=H$ and $HX=X$. Thus the regression residuals are $\hat{\epsilon }=(I-H)Y$, withThis is fitted-residual orthogonality. The vectors are jointly normal, so independence of uncorrelated jointly normal variables strengthens both zero-covariance statements to independence.

For the quadratic linear regression, the columns of the design matrix are $1,x,x^2$. The replicated design givesFor $v(x)=(1,x,x^2{)}^T$, the variance of a fitted regression mean is ${\sigma }^{2}v(x{)}^T(X^{T}X{)}^{-1}v(x)$, henceWriting $u=x^2\in [0,1]$, this is the quadratic $({\sigma }^2/20)(2-3u+3u^2)$. Its derivative vanishes at $u=1/2$; it decreases before that point and increases afterwards. Therefore the maximum is ${\sigma }^2/10$ at $x=-1,0,1$, and the minimum is ${\sigma }^2/16$ at $x=\pm 1/\sqrt{2}$.

At $100^{\circ }\mathrm{C}$, $x=0$ and the expected yield is $\alpha$. The unbiased residual variance estimate is $s^2=2.43/(30-3)=0.09$. By Cochran's theorem, $27s^2/{\sigma }^2\sim {\chi }_{27}^2$ independently of $\hat{\alpha }$, giving the Student t confidence intervalThe interval estimates the expected yield, so it uses the variance of a fitted regression mean without an additional future-observation error term. The supplied summary determines the half-width but does not give a numerical value of $\hat{\alpha }$.

A factor is a categorical predictor whose levels distinguish experimental conditions. Two factors have a statistical interaction when the effect of changing one depends on the other's level. For cell means ${\mu }_{ij}$, absence of an interaction means ${\mu }_{ij}=\mu +{\alpha }_i+{\beta }_j$; equivalently, ${\mu }_{ij}-{\mu }_{i^{\prime }j}$ is independent of $j$.

Index thickness by $a\in \{0,1\}$, temperature by $b\in \{0,1\}$, wash by $w\in \{1,2,3,4\}$ and replication by $r\in \{1,2\}$. The full factorial design has the normal linear modelThe errors are independent with a common positive variance. Sum-to-zero constraints in each factor index, or the displayed treatment coding, identify the mean coefficients. This full model allows an arbitrary mean in each of the sixteen cells.

The missing statistical degrees of freedom, in the output order, are $1,1,3,1,3,3,3,16$. The first seven entries total fifteen; with the regression intercept there are sixteen mean coefficients among thirty-two observations, leaving sixteen residual statistical degrees of freedom. Balanced factorial orthogonality makes the term sums of squares in ANOVA independent of their ordering.

All terms involving wash have nonsignificant individual F-tests, and it is useful to test their removal jointly. The reduced model omits twelve mean coefficients. Its residual sum of squares is $57.09875$, compared with $32.765$ for the full model, so the nested-model F-test givesThere is no detected wash effect over the four tested durations. Removing its main effect and all its interaction terms respects the principle of marginality. Retaining the very large thickness-by-temperature interaction is essential: the thickness effect changes substantially with temperature.

With numeric indicators $a,b\in \{0,1\}$, treatment coding expresses the reduced fitted mean asThus the thin/low, thick/low, thin/high and thick/high fitted means are $4.225,5.6875,17.425,29.950$, respectively, at every wash level. Each is the sample mean of eight observations. The equal-cell replication variance formula therefore givesAlternatively, use the variance of a fitted regression mean with $v=(1,a,b,ab{)}^T$: $s^2{v}^T(X^{T}X{)}^{-1}v=s^2/8$. Individual coefficient standard errors cannot simply be added; their covariances enter the fitted-mean calculation.

Increasing temperature raises fitted lustre by $13.2$ for thin film and $24.2625$ for thick film. Increasing thickness raises it by $1.4625$ at low temperature and $12.525$ at high temperature. Their difference, $11.0625$, is the estimated interaction contrast; its standard error is $1.0098$, giving $t\simeq 10.956$ on twenty-eight statistical degrees of freedom. The interaction plot shows this strong departure from parallel mean lines, while the wash panel shows the much smaller variation across wash durations. Neither the nonsignificant wash test nor the plot proves that its true effect is exactly zero.

Let $\mathrm{RSS}={\sum }_i(Y_i-\hat{\alpha }-\hat{\beta }{x}_i{)}^2$. Differentiating the log-likelihood in $v={\sigma }^2$ givesFor $\mathrm{RSS}>0$ this changes from positive to negative at $v=\mathrm{RSS}/n$, soTo find its bias, substitute the model and use the orthogonal fit directions $1$ and $\mathbf{x}$, obtainingThe three expectations are $n{\sigma }^2$, ${\sigma }^2$, and ${\sigma }^2$, respectively: $\mathrm{Var}(\bar{\epsilon })={\sigma }^2/n$ and $\mathrm{Var}({\sum }_i{x}_i{\epsilon }_i)={\sigma }^2{S}_{xx}$. Hence $\mathbb{E}\mathrm{RSS}=(n-2){\sigma }^2$, giving the unbiased estimatorWith $n>2$, a zero residual sum of squares is a probability-zero event in the model. For that exceptional dataset the likelihood is unbounded as ${\sigma }^2\downarrow 0$, rather than having a maximum at a positive variance. In the intercept-only case $S_{xx}=0$, the corresponding unbiased multiplier is $n/(n-1)$ for $n>1$.

Write $S_{xx}={\sum }_i{x}_i^2$ and $\bar{Y}=n^{-1}{\sum }_i{Y}_i$. For an identifiable two-parameter linear regression assume $S_{xx}>0$; for the variance estimate and test below also assume $n>2$. The least-squares normal equations areUsing ${\sum }_i{x}_i=0$ givesThe residual sum of squares has positive-definite quadratic part $n{\alpha }^2+S_{xx}{\beta }^2$, so these equations give its unique minimum. The normal distribution of the independent errors gives the log-likelihoodFor every fixed positive ${\sigma }^2$, maximizing this over $\alpha ,\beta$ is exactly minimizing the same residual sum of squares. Thus the ordinary least squares estimators are also the maximum-likelihood estimators.

The printed hypotheses do not explicitly require $S_{xx}>0$ or $n>2$. If $S_{xx}=0$, all $x_i$ vanish and $\beta$ is unidentifiable; this is an intercept-only model with $n-1$ residual degrees of freedom. For $n=1$ its residual is identically zero, so the likelihood has no finite maximizer with positive variance. If $S_{xx}>0$ but $n=2$, the fit is saturated, leaving no residual variance estimate or Student test with positive degrees of freedom. The following nondegenerate formulas use $S_{xx}>0$ and $n>2$.

This is an analysis of covariance: achievement is the response, aptitude is a quantitative covariate, and teaching method is a categorical explanatory variable. The usual normal linear model assumes independent errors with common variance ${\sigma }^2$.

read.table reads the data with a header row; attach makes its columns directly accessible, and the two printed vectors confirm their ordering. rep(1:3, each=7) supplies the three group labels, and factor makes them categorical. The contrast option selects treatment coding for unordered factors, with method A as the first-level reference. The polynomial option would apply to ordered factors and has no role here.

The formula using aptitude * Group includes both main effects and their interaction. Writing $I_B,I_C$ for the method indicators, its fitted mean has the formThus A, B and C have separate intercepts and slopes. There are six coefficients and $21-6=15$ residual statistical degrees of freedom. The analysis of variance table uses sequential sums of squares: adding aptitude to an intercept-only model removes $36.57548$ from the residual sum of squares; subsequently adding the two group indicators removes $16.93200$; adding the two slope interactions removes only $0.66714$. The remaining sum of squares is $9.63490$, giving mean square $9.63490/15=0.64233$.

Each displayed F-test divides the added term's sum of squares by its statistical degrees of freedom and then by this full-model residual mean square. In particular the interaction test isThere is no evidence that the slopes differ, so the common-slope model is preferable among the fitted models. The small group $p$-value concerns differences after including aptitude, whereas the first aptitude row is sequential and does not adjust for Group. It should not be interpreted as the covariate test after treatment adjustment.

The additive formula gives the parallel-lines modelIt has four coefficients and $17$ residual statistical degrees of freedom. Removing the interaction raises the residual sum of squares to $9.63490+0.66714=10.30204$, but lowers the residual mean square to $10.30204/17=0.60600$. The sequential aptitude and group sums of squares are unchanged. The adjusted group test is now $F=(16.93200/2)/0.60600=13.97024$ on $(2,17)$ statistical degrees of freedom, with $p=0.0002579$: after allowing for initial aptitude, the methods are not all equally effective.

summary(..., cor=F) displays the fitted coefficients, their standard errors, the individual $t$ statistics, and two-sided $p$-values; it omits the coefficient-correlation matrix. The intercept $2.8367$ is the fitted A mean at aptitude zero, which is outside the observed range and is mainly a parametrization device. The common slope $0.7429$ predicts about $0.74$ extra achievement units per aptitude unit, holding method fixed. Its adjusted $t$ test is $5.2267$ on $17$ statistical degrees of freedom, with $p$ about $0.0001$. The method coefficients are adjusted differences from A, not the B and C intercepts themselves. The three fitted lines areThe B-minus-A estimate is $2.1878$ with standard error $0.4545$ and $t=4.8139$; the C-minus-A estimate is $1.8612$ with standard error $0.4240$ and $t=4.3901$. Both comparisons are strongly significant.

The B-minus-C comparison needs its own treatment contrast; its significance cannot be read off either comparison with A. Reconstructing the coefficient covariance from the design givesThus there is no detectable difference between B and C. This is consistent with pooling their effects in a further reduced model, but does not establish exact equality.

The residual standard error is $\sqrt{0.60600}=0.7785$. The five-number residual summary shows deviations on both sides of zero, with the largest magnitude about $1.74$; it does not by itself verify constant variance or normality. The coefficient of determination $R^2=0.8386$ means about $84\%$ of the achievement variation about its overall mean is explained. The overall $F=29.43$ compares all three nonintercept coefficients with zero, on $(3,17)$ statistical degrees of freedom; its $p$-value is $5.889\times 10^{-7}$. Printed coefficient $p$-values rounded to 0.0000 are small, not literally zero.

The following original sketch shows the separate-slope model and the preferred parallel-line fit.

At the same initial aptitude, B and C give higher predicted achievement than A, with no convincing difference between B and C. Interpreting this adjusted association as a causal teaching effect additionally relies on the experimental allocation and error assumptions.

A useful full statement is the rank-sum form of Cochran's theorem. Let $Z\sim N_n(0,I_n)$ and let $A_1,\dots ,A_k$ be real symmetric matrices satisfying ${\sum }_i{A}_i=I_n$. Put $r_i=\mathrm{rank}(A_i)$ and $Q_i=Z^T{A}_{i}Z$. Then ${\sum }_i{r}_i\ge n$, and the following conditions are equivalent: the ranks sum to $n$; the $A_i$ are orthogonal projection matrices with mutually orthogonal ranges; and the $Q_i$ are independent, with $Q_i\sim {\chi }_{r_i}^2$. A rank-zero form is identically zero. In particular, the usual projection version of Cochran's theorem partitions the squared length of a standard normal vector into independent chi-squared distributions.

Here is a proof which also explains the rank condition. Let $L_i=\mathrm{im}{A}_i$. Since every vector satisfies $x={\sum }_i{A}_{i}x$, the spaces $L_i$ span ${\mathbb{R}}^n$. Therefore their dimensions sum to at least $n$. If they sum to $n$, their sum is a direct sum, so every vector has a unique decomposition into vectors belonging to the $L_i$. For $y\in L_i$, the identity $y={\sum }_j{A}_{j}y$ is one such decomposition, while the decomposition with just $y$ in position $i$ and zero elsewhere is another. Uniqueness gives $A_{j}y={\delta }_{ij}y$. ThusA symmetric matrix with this idempotence is an orthogonal projection matrix. Its different ranges are orthogonal: for $x=A_{i}u$, $y=A_{j}v$, their inner product is $u^T{A}_i{A}_{j}v=0$. Choose an orthonormal basis for each range and concatenate them. The resulting change-of-basis matrix is an orthogonal matrix. In these coordinates, each $Q_i$ is the sum of squares of the $r_i$ coordinates belonging to its range.

The transformed multivariate normal distribution still has density proportional to $\exp (-\Vert z{\Vert }^2/2)$, which factors into the standard normal densities of its coordinates. The coordinates are independent, so the disjoint groups of squared coordinates give independent chi-squared distributions of the asserted degrees of freedom. This proves the forward implication. Conversely, if the $Q_i$ have the asserted chi-squared laws, then $\mathbb{E}{Q}_i=r_i$, while ${\sum }_i{Q}_i=Z^{T}Z$ has expectation $n$. Taking expectations yields ${\sum }_i{r}_i=n$. This completes the equivalence and the proof of Cochran's theorem.

For the application, specify a normal linear modelwith fixed full-column-rank design matrix $X$ of size $n\times p$, $p<n$, unrestricted $\beta \in {\mathbb{R}}^p$, and unknown ${\sigma }^2>0$. Test $H_0:R\beta =r$ against $H_1:R\beta \ne r$, where $R$ has $q$ independent rows, $1\le q\le p$, and $r$ is specified. The null contains an affine space of means of dimension $p-q$, whereas the full mean space has dimension $p$.

Let $\hat{\beta }=(X^{T}X{)}^{-1}{X}^{T}Y$ be the ordinary least squares and maximum-likelihood estimate in the full model, and write $G=(X^{T}X{)}^{-1}$. The constrained estimate follows by minimizing the residual squared length under $R\beta =r$:Indeed, $R\tilde{\beta }=r$ and the correction is the required constrained projection in the $X^{T}X$ inner product. The least-squares normal equations give the Pythagorean decompositionConsequently the explicit residual quantities in the likelihood-ratio test statistic areHere $B$ is the full-model residual sum of squares, and $A=\Vert Y-X\tilde{\beta }{\Vert }^2-B$ is the increase caused by imposing the null restrictions.

To see the geometry behind Cochran's theorem, choose any ${\beta }_0$ with $R{\beta }_0=r$, and let $P$ project onto $\mathrm{col}X$ and $P_0$ onto $X\ker R$. For $z=Y-X{\beta }_0$,The projectors $P-P_0$ and $I-P$ are orthogonal and have ranks $q$ and $n-p$. Under $H_0$, the mean of $z$ belongs to $X\ker R$, so both of these projectors annihilate it. Thus Cochran's theorem givesThe nuisance variance cancels in the nested-model F-test statisticSince the likelihood-ratio test statistic isit is a strictly increasing function of $F$. Therefore an exact level-$\alpha$ likelihood-ratio test rejects when $F>F_{q,n-p;1-\alpha }$, where the right-hand side is the upper reference quantile of the F-distribution. This calibration is exact under the normal linear model, rather than an asymptotic chi-squared replacement.

Assume the explanatory variables are not all equal, so $S_{xx}={\sum }_i(x_i-\bar{x}{)}^2>0$. In the Gaussian linear model, the log-likelihood isFor fixed ${\sigma }^2$, maximum likelihood estimation therefore minimizes the residual sum of squares. Differentiating in $\alpha ,\beta$ gives the least-squares normal equationsThe first gives $\hat{\alpha }=\bar{Y}-\hat{\beta }\bar{x}$; substituting it into the second gives $\hat{\beta }=S_{xy}/S_{xx}$, with $S_{xy}={\sum }_i(x_i-\bar{x})(Y_i-\bar{Y})$. Let $\mathrm{SSE}={\sum }_i(Y_i-\hat{\alpha }-\hat{\beta }{x}_i{)}^2$. Differentiating the profiled log-likelihood in ${\sigma }^2$ givesFor the variance estimator this is the interior maximum when $\mathrm{SSE}>0$, which occurs almost surely under the model when $n>2$.