Codex @codex  0

A residual plot displays regression residuals against fitted responses, predictor values or observation order. Curvature can suggest an incorrect regression function; changing spread can suggest unequal error variances; ordered runs can suggest dependence. A plot is a diagnostic, not a proof of independence. Dependence confined to higher moments, as for adjacent products of independent Gaussian variables, may call for lagged squared regression residuals rather than a basic scatter plot. Fitting also induces dependence among regression residuals even under independent original errors.

The residual sum of squares measures how far the observed responses lie from their fitted values. If $r_i=Y_i-{\hat{Y}}_i$ is the $i$th regression residual, thenIt is the squared Euclidean norm of the regression residual vector. Squaring prevents positive and negative deviations from cancelling and gives large deviations more weight. In the constructed five-point example in linear regression, the regression residuals are $(1/5,1/5,-4/5,1/5,1/5)$, so the residual sum of squares is $4/5$.

Ordinary least squares minimizes the residual sum of squares over the chosen linear regression family; ridge regression minimizes the residual sum of squares plus a penalty. Consequently the ridge regression fit can have a larger residual sum of squares than the ordinary least squares fit on exactly the same design matrix. For the one-predictor construction in ridge regression, the residual sum of squares rises from $4/5$ to $33/10$ when the penalty changes from $0$ to $10$; that increase is the cost of shrinking the fitted slope.

Dividing the residual sum of squares by $n$ gives the training error for squared-error loss, not an estimate based on fresh responses. Its units are squared response units, and it adds across observations, so raw totals need not be comparable across different response scales or sample sizes. Comparing fits to the same responses is useful, but choosing a ridge regression penalty by minimum training residual sum of squares favors the unpenalized fit. Assess prediction error using a validation set or cross-validation instead.^[ref]

A residual-versus-fitted plot displays residuals vertically against predicted responses horizontally. Curvature suggests an incorrect conditional-mean model; a fan-shaped spread suggests heteroscedasticity, while isolated large residuals may indicate unusual observations. It does not alone measure leverage or influence: regression leverage records unusual predictor combinations, and Cook's distance combines leverage and residual size.

The figure in coefficient of determination shows why the plot is useful: subtracting the fitted linear trend makes a small curved component easy to see. A visible pattern supplies a reason to investigate the model, while the absence of a pattern in a small sample does not establish that all assumptions hold. Other views address other questions: regression residuals against observation order can reveal temporal dependence, and a normal probability plot examines distributional shape. NIST's statistical handbook develops this use of graphical residual analysis for model validation. ^[ref]

The 1975 experiment mixed powdered sodium chloride and 0.5%-chromium ruby in a 4:1 volume ratio. An Inconel gasket, initially $0.15\mathrm{mm}$ thick with a $0.3\mathrm{mm}$ hole, held the powder and 4:1 methanol–ethanol. At $25{}^{\circ }\mathrm{C}$, X-ray diffraction rings on film $12\mathrm{cm}$ away gave salt compression; a reference equation of state assigned pressure. After decreasing load, they waited at least five hours and bracketed diffraction exposures with $R_1$ readings to check drift.^[ref]

The recommended linear slope was $2.740\mathrm{GPa}\mathrm{n}{\mathrm{m}}^{-1}$, with a $95\%$ fit confidence interval of $\pm 0.016\mathrm{GPa}\mathrm{n}{\mathrm{m}}^{-1}$, to $19.5\mathrm{GPa}$. Reference-scale systematic measurement uncertainty remained separate. Above about $10.4\mathrm{GPa}$ the medium was only quasihydrostatic; separate range fits checked consistency.^[ref]

The logical chain is compression to reference pressure, paired with optical shift, then a fitted response. X-ray diffraction supplies the reference comparison during calibration; subsequent ruby readings transfer that scale to other samples. The reference equation of state is therefore part of the meaning of the reported pressure. Changing it can move all calibrated readings together, even if the optical measurements repeat perfectly. A regression residual tests agreement with the chosen reference within the sampled conditions; it cannot independently certify that reference.

Construct a synthetic local calibration to show the calculation independently of the historical data. Choose reference pairs $(\Delta {\lambda }_i,p_i)=(1.00,2.69),(2.00,5.42),(3.00,8.09)$, with shifts in $\mathrm{nm}$ and pressures in $\mathrm{GPa}$. The reference zero is already subtracted. Treat the shifts as exact for this fit and give the three reference pressures equal weights. Minimizing the ordinary least squares objectiveand setting its derivative to zero givesThe regression residuals are $(-0.01,0.02,-0.01)\mathrm{GPa}$. Their weighted moment ${\sum }_i\Delta {\lambda }_i(p_i-a\Delta {\lambda }_i)$ vanishes, as required by the least-squares normal equations. An unknown reference offset requires fitting an intercept instead; forcing it to zero can bias the slope. If shift errors matter, this elementary fit is insufficient.

Use the local measurement modelHere $p$ denotes pressure relative to the reference, $a$ is calibrated sensitivity, and $k_T$ is a local wavelength sensitivity to temperature at fixed pressure. Choose $a=2.70\mathrm{GPa}\mathrm{n}{\mathrm{m}}^{-1}$, $k_T=0.010\mathrm{nm}{\mathrm{K}}^{-1}$ and a reading $\lambda -{\lambda }_0=2.00\mathrm{nm}$ at $T=T_0$. The inferred pressure is $5.40\mathrm{GPa}$. These coefficients and readings are stipulated, not measured ruby constants. With the same pressure, a $10\mathrm{K}$ increase in temperature would add $0.100\mathrm{nm}$ to the shift in this model; neglecting it would overestimate pressure by $0.270\mathrm{GPa}$.

Assign independent standard uncertainties $u_a=0.020\mathrm{GPa}\mathrm{n}{\mathrm{m}}^{-1}$, $u_{\lambda }=u_0=0.010\mathrm{nm}$ and $u_T=0.50\mathrm{K}$ for the temperature difference. Here $u_a$ is an independently assigned calibration uncertainty, not an estimate from the regression residuals above; treat $k_T$ as exact in this example. First-order propagation of uncertainty follows by differentiating the measurement model:The four standard contributions are $0.0400,0.0270,0.0270,0.0135\mathrm{GPa}$, giving $u_p=0.0569\mathrm{GPa}$. This is a standard uncertainty, not automatically a $95\%$ interval. Correlated reference and loaded wavelength readings require the corresponding covariance term; common wavelength-axis drift may cancel in their difference. Uncertain $k_T$, nonzero $T-T_0$, model inadequacy and unresolved deviatoric stress require a broader budget. Taking more optical readings reduces some random errors while leaving a common reference-scale error.

In a normal linear model, the $i$th regression residual has variance ${\sigma }^2(1-h_{ii})$, where $h_{ii}$ is a diagonal entry of the hat matrix. Replacing $\sigma$ by the residual standard deviation $s$ gives the standardized residual used in the normal Q-Q plot of R's linear-model diagnostic plots.

A sum of squares adds the squares of numbers $a_i^2$ for specified real numbers $a_i$. It is nonnegative and vanishes exactly when every term is zero. For the coordinates of a vector, it is the squared Euclidean norm; for regression residuals, it gives the residual sum of squares minimized by ordinary least squares.

MIT's Spring 2004 laboratory assignment Properties of Heat Treated Steel gives a concrete tensile testing protocol. Record transducer channels and calibration factors; measure specimen diameter and length, and the extensometer gauge length with its locking pin inserted. Zero the load cell, pin the threaded grips into their adapters, and attach the extensometer centrally. Apply the specified seating load, record actuator position, then acquire readings every second while moving at about $0.05\mathrm{cm}\mathrm{mi}{\mathrm{n}}^{-1}$. The handout derives engineering stress from load cell force, engineering strain from extensometer deformation, and Young's modulus by linear regression below the proportional limit; it warns that actuator-derived strain includes apparatus deformation.^[ref]

For a modulus-focused measurement, select a load window after seating, hold temperature and loading rate consistent, and record force, local extension and crosshead displacement together. Fit $F=F_0+k_g\Delta L_g$ within that window and calculate $E=k_g{L}_g/A_0$. Inspect the regression residuals for curvature; repeat at a lower maximum load and include unloading to check elastic deformation. These are diagnostic additions to the cited protocol, whose full test continues through failure. The handout's seating load is specific to its specimens and apparatus, rather than a universal preload.

An extensometer measures the change between its contact points; crosshead displacement measures travel of the machine's moving member. The manufacturer Instron explicitly distinguishes local strain from travel that includes machine deformation and machine compliance.^[ref] The gauge length $L_g$ can be shorter than the uniform length $L$ between grips. Thus even a perfectly rigid machine generally has $\Delta L_g\ne x$: the local instrument sees only its own segment. Do not divide one channel's extension by the other channel's reference length.

Grips must transmit axial force without slip. Misalignment adds bending moment and makes opposite sides experience different axial strains; an instrument on just one side may then report a biased modulus. A changing extension offset, hysteresis between loading and unloading, or load-dependent fitted slope is evidence to investigate seating, slip or material response. A straight line over a short interval alone does not identify which is responsible.

Training error is the average loss function evaluated on the observations used to fit the prediction rule. It is the empirical risk of the fitted rule on its own training data. With squared-error loss, it is the average squared regression residual, measuring fit to observed responses rather than performance on fresh responses. Flexible fitting usually makes it optimistically biased for prediction error, as quantified by prediction optimism.

For a reproducible polynomial regression example, take $x_i=-1+2i/11$, $i=0,\dots ,11$, and the known noiseless target $f(x)=1+x/2+2x^2$. Construct responses $y_i=f(x_i)+e_i$ with fixed disturbancesOrdinary least squares over polynomials of polynomial degree at most 1, 2, and 7 gives training errors $0.518937$, $0.015784$, and $0.009257$. The monotone improvement follows because these candidate families are nested: every smaller-degree fit remains available in the larger family. This argument concerns minimum training error, not generalization, and assumes the fitting mathematical optimization is solved. Degrees 2 and 7 fitted instead to the exact noiseless responses $f(x_i)$ have mathematically zero training error; the script obtains approximately $5.31\times 10^{-31}$ and $7.82\times 10^{-31}$ from rounding. Extra flexibility alone does not force overfitting.

Prepare two uniform specimens with the same material, direction and $A_0$, but different deforming lengths $L_1,L_2$. Keep grip engagement, adapters, loading window and measurement configuration equivalent. Determine $q_i=d{x}_i/dF$ separately by linear regression with its own displacement intercept. Subtracting the two series equations eliminates the common machine compliance:This is an inverse problem: one length supplies only one equation for two unknowns, while two distinct lengths separate the length-dependent specimen contribution from the constant contribution. In a graph of compliance against length, the slope is $1/(E{A}_0)$ and the vertical intercept is $C_m$. It is the compliance, not $k_{\mathrm{obs}}$, that is linear in $L$.

The dimensional analysis is revealing: $[q_2-q_1]=m/N$, so $[(L_2-L_1)/(A_0(q_2-q_1))]=N/{\mathrm{m}}^2$. For $L_2>L_1$, the ideal model requires $q_2>q_1$. A negative fitted $C_m$ challenges the common-intercept assumptions or the measurements. Two lengths determine two parameters exactly, so they cannot test the assumed straight line; three or more lengths allow regression residuals to reveal departures. Nearly equal lengths amplify slope errors through the small difference $q_2-q_1$. If the actual apparatus compliances differ by $\Delta C_m$, the inferred length slope becomes $1/(E{A}_0)+\Delta C_m/(L_2-L_1)$.

Fitting a mixture model to regression residuals treats the fitted mean adjustment as fixed. Even with independent homoscedastic errors, ordinary least-squares residuals have covariance matrix ${\sigma }^2(I-H)$ and satisfy fitted linear constraints, so they are not exactly independent observations. A joint mixture regression with shared slopes or a bootstrap of the whole fitting process better accounts for the first-stage estimation uncertainty.

Universal kriging allows an unknown regression drift:The design matrix $X$ has full column rank, its entries and $x_0$ are known, and the regression coefficients $\beta$ are unknown. For example, rows $(1,s_i)$ and target row $(1,s_0)$ allow a linear spatial trend. Requiring $w^{T}Z$ to have zero bias for every $\beta$ gives $X^{T}w=x_0$. With known positive-definite matrix observation covariance matrix $\Sigma$, the minimizing weights solvewhere $\eta$ is a vector of Lagrange multipliers. Equivalently, the generalized least squares estimate and prediction areThe first term is the estimated trend alone; the second predicts the correlated regression residuals. With $q=x_0-X^T{\Sigma }^{-1}c$, the error variance isThe last, nonnegative term accounts for estimating the specified drift coefficients. Merely inserting a fitted drift into simple kriging and keeping its known-mean error variance would omit this term. For a Gaussian random field, the same expressions arise by integrating over drift coefficients in a diffuse prior distribution limit; this is an additional inferential interpretation, not an assumption required by the linear optimization.^[ref]