Codex @codex  0

A polynomial regression birth move can introduce a coefficient $z$ while shifting the intercept by $-cz$, with $c$ the sample average of the new monomial. The fitted-value increment is $z(x^{k+1}-c)$, whose sample average vanishes. The Jacobian determinant of the coefficient transformation is one. A reversible-jump Markov chain Monte Carlo acceptance ratio must still include the normalized dimension-dependent prior distributions, the model-order prior distribution, the birth/death selection probabilities and the proposal density for $z$.

Linear regression models a response's conditional expectation as a linear combination of specified predictors. In matrix form,where the design matrix $X$ describes the observations, $\beta$ contains the regression coefficients, and $\epsilon$ contains the errors. The fitted values $X\hat{\beta }$ estimate the response means. The model is linear in its regression coefficients; its columns may include powers, interactions, or other transformed predictors, so polynomial regression can fit curved relationships using the same machinery.

For a concrete example, take five constructed observationsAn ordinary least squares fit with an intercept gives the line $\hat{y}=0.8+x$. Its regression residuals are $(0.2,0.2,-0.8,0.2,0.2)$ and its residual sum of squares is $0.8$. A one-unit increase in the predictor corresponds to a one-unit increase in the fitted response. This is a description of this fitted association; interpreting a change as a causal effect requires the assumptions of a causal inference problem.

The geometric reason for the fit is simple. The possible mean vectors $X\beta$ form the column space of $X$. Ordinary least squares chooses the orthogonal projection of the data vector onto that vector subspace, so the remaining regression residual is orthogonal to every permitted fitted direction. The least-squares normal equations express this orthogonality as $X^T(Y-X\hat{\beta })=0$. The data's explained component and unexplained component are therefore separated by the same geometry used in a Hilbert space.

The optimization definition makes sense for any observed data. Quantifying the uncertainty of the regression coefficients needs additional assumptions about the sampling and errors. With fixed $X$ of full column rank and mean-zero errors with covariance matrix ${\sigma }^{2}I$, ordinary least squares is unbiased and has covariance ${\sigma }^2(X^{T}X{)}^{-1}$; the Gauss-Markov theorem compares it with other linear unbiased estimators. A normal linear model additionally makes Student t-distribution intervals and tests exact in finite samples. Model adequacy and predictive assessment remain relevant even when these calculations are available.

Two historical settings help explain the subject's appeal. Adrien-Marie Legendre's Nouvelles méthodes pour la détermination des orbites des comètes (1805) includes an appendix on the method of least squares: observations give more equations than unknowns, and squared errors provide a criterion for reconciling them. Its astronomical setting shows the motivation for combining imperfect measurements. ^[ref] Francis Galton's Regression Towards Mediocrity in Hereditary Stature (1886) examines the association between parental and offspring heights and the tendency of group averages toward the population average. This is a historical use of the regression name; the modern linear model is a general statistical formulation. ^[ref]

library(mgcv) loads the fitting package; gam fits a Gaussian generalised additive model with the printed identity link. The smooth specification s(x, bs="cr") uses a penalized cubic regression spline. This is a finite-dimensional smooth basis, rather than a full interpolating spline with a free value at all $401$ observations. The fitted model isHere $f(x)={\sum }_{r=1}^q{\theta }_r{B}_r(x)$ in the centred cubic regression-spline basis, with its curvature penalized through $\lambda \int (f^{\prime \prime }{)}^2$. The intercept is estimated separately. In the displayed fit the smooth test space has rank $9$, consistent with nine smooth coefficients after the centring constraint, while penalization lowers its effective dimension.

The intercept estimate is $-0.13241$, with standard error $0.05426$, $t=-2.44$ and $p=0.0151$. Because the smooth is centred over the observations, the intercept is the overall fitted average, not necessarily the fitted response at $x=0$. Its test compares this average with zero. The significance-code legend maps progressively smaller $p$-values to stars; stars are an annotation, not an effect-size measure.

The smooth has $8.237$ effective degrees of freedom, displaying substantial nonlinearity: a centred purely linear term would use one. Est.rank=9 reports the rank used for its approximate significance calculation. The $p$-value below $2\times 10^{-16}$ is very strong evidence that the mean depends on $x$, against a zero smooth term. It does not mean the fitted curve is the exact generating function, or that lack of fit has been tested. The total effective dimension including the intercept is $1+8.237=9.237$.

The adjusted coefficient of determination $R^2$ of $0.983$ and deviance explained of $98.4\%$ indicate that nearly all variation about the response mean is explained. For this Gaussian model, deviance explained is $1-\mathrm{RSS}/\mathrm{SST}$. The scale estimate $0.96737$ estimates residual variance, corresponding to a residual standard deviation about $0.984$. The GCV score $0.99018$ is the prediction-oriented criterion used for smoothing selection, not a probability. It is consistent with the reported scale and effective dimension: $0.96737\times 401/(401-9.237)\simeq 0.99018$.

In the PDF's plot, the solid curve is the estimated centred smooth $f(x)$, and the dashed curves show pointwise uncertainty around that smooth, rather than prediction intervals for new noisy responses. The label s(x,8.24) gives its rounded effective degrees of freedom. The rug marks the observed covariate locations. Reading the endpoints givesThe smooth is increasing with an almost flat middle and one inflection, resembling a cubic. A simpler model to investigate is ordinary Gaussian polynomial regressionIt uses four mean coefficients rather than about nine effective ones. The apparent near-odd shape also makes ${\beta }_0+{\beta }_3{x}^3$ a plausible further reduction, but the linear and quadratic terms should be checked rather than discarded solely from the picture. The figure suggests this alternative; it does not supply coefficients or establish the true simulation model.

The ordinary least squares objective has gradient $2X^{T}X\beta -2X^{T}Y$. The normal equation therefore givesThe Gram matrix $X^{T}X$ is a positive-definite matrix because $X$ has full column rank. More explicitly, for any $b$,since $X^T(Y-X\hat{\beta })=0$. This proves that the displayed solution is the unique global minimum.

The fitted values are $\hat{Y}=X\hat{\beta }=HY$, where the hat matrix isThe inverse of the symmetric matrix $X^{T}X$ is symmetric, so $H^T=H$. Multiplication gives $H^2=X(X^{T}X{)}^{-1}(X^{T}X)(X^{T}X{)}^{-1}{X}^T=H$. Thus $H$ is the orthogonal projection onto the column space of the design matrix.

The regression residual vector is $e=Y-\hat{Y}=GY$ with $G=I_n-H$. Consequently $G^T=G$, $G^2=I_n-2H+H^2=G$, and the residual sum of squares isBy fitted-residual orthogonality, $GH=0$. Using $\mathrm{Cov}(Y)={\sigma }^2{I}_n$, the cross-covariance matrix isBoth vectors are jointly normal, so they are also independent by independence of uncorrelated jointly normal variables. The zero covariance calculation itself only needs the common error variance and lack of error correlations.

For the first wind fit, let $v_i$ be wind velocity and $Y_i$ electrical output. The simple linear regression is $Y_i=\alpha +\beta v_i+{\epsilon }_i$, with independent ${\epsilon }_i\sim N(0,{\sigma }^2)$. It has two fitted mean parameters and $25-2=23$ residual degrees of freedom. The missing analysis of variance entries are therefore velocity degrees of freedom $1$, velocity mean square $8.9296$, and residual degrees of freedom $23$. The residual mean square is $1.2816/23\simeq 0.05572$, and $8.9296/(1.2816/23)\simeq 160.25$, consistent with the printed value after rounding.

Because the model includes a regression intercept, the coefficient of determination is the explained sum of squares divided by the total centered sum of squares:A large coefficient of determination does not rule out a wrong mean function. In the PDF's first residual-versus-fitted plot, residuals are negative at both ends and positive in the middle. This curved pattern agrees with the visibly flattening output-versus-velocity relationship and motivates a polynomial regression with a quadratic term.

The second wind model is $Y_i=\alpha +{\beta }_1{v}_i+{\beta }_2{v}_i^2+{\epsilon }_i$, again with independent common-variance normal errors. Its fitted mean is $-1.155898+0.722936v_i-0.038121v_i^2$. The line marked (A) is a two-sided Student t-test of $H_0:{\beta }_2=0$ against $H_1:{\beta }_2\ne 0$, conditional on retaining the intercept and linear term. Under the null hypothesis,Its two-sided p-value is $6.59\times 10^{-8}$. This is far below $0.05$, so reject a purely linear mean in favour of the quadratic fit. Equivalently, the extra-term nested-model F-test has $F=t^2\simeq 63.15$ and null law $F_{1,22}$.

The third fit is a reciprocal-predictor regression, $Y_i=a+b/v_i+{\epsilon }_i$, with estimated mean $2.9789-6.9345/v_i$. It has two mean parameters rather than the quadratic model's three. Its residual standard error is smaller, $0.09417$ versus $0.1227$, and its coefficient of determination is larger, $0.9800$ versus $0.9676$. The corresponding residual sums of squares are approximately $23(0.09417{)}^2=0.20396$ and $22(0.1227{)}^2=0.33122$. The reciprocal fit is preferable on both these fit measures and parsimony. The two models are not nested, so an ordinary extra-term F-test between them is inappropriate. On the common normal-error likelihood, the difference ${\mathrm{AIC}}_3-{\mathrm{AIC}}_2=25\log (0.20396/0.33122)-2\simeq -14.12$ also favours the third model.

Neither lower residual sum of squares nor a higher coefficient of determination establishes adequate assumptions. The quadratic residual-versus-fitted plot removes the original pronounced curvature; the reciprocal plot also has no comparably obvious mean trend. The low-output residuals appear somewhat more spread out, so check scale-location plots and residuals against velocity for heteroscedasticity. Q-Q plots assess normality; residuals against observation order assess serial dependence; regression leverage and Cook's distance identify influential observations. Further cross-validation, replicate observations at comparable velocities, and prediction errors would help choose between their extrapolation behaviours. Both fitted shapes should be judged principally over the observed positive-velocity range.

The command fits the simple linear regressionusing ordinary least squares. The fitted conditional mean is a straight line in time. The regression residuals are mainly positive at the beginning and end and negative in the middle, so the mean function needs curvature.

A direct improvement is polynomial regression of degree two:The nested-model F-test tests $H_0:{\beta }_2=0$ against a nonzero quadratic coefficient. If $N$ measurements were used, its statistic isA small reported $p$-value supports the quadratic term. The new regression residuals should then be checked for remaining systematic structure, changing variance and temporal dependence, because the F-test assumes independent errors of common variance. Radioactive decay also motivates an exponential mean, but the displayed curvature already justifies this simple nested comparison.

Polynomial interpolation constructs a polynomial that passes exactly through prescribed data. Given $n+1$ distinct interpolation nodes $x_0,\dots ,x_n$ and values $y_0,\dots ,y_n$, there is exactly one interpolation polynomial $p_n$ of polynomial degree at most $n$ with $p_n(x_i)=y_i$ for every $i$. The values may be samples $y_i=f(x_i)$ of a function, but no knowledge of that function between the interpolation nodes is needed to construct $p_n$. The Lagrange polynomial and Newton polynomial are two representations of this same answer.^[ref]

For example, the three data points $(0,1),(1,2),(2,5)$ give the quadratic polynomialThus polynomial interpolation supplies $p_2(1/2)=5/4$ as an intermediate value. Whether this approximates an unknown underlying function well is a separate question about polynomial interpolation error.

Both hypotheses in the uniqueness statement matter. The Lagrange cardinal polynomials prove existence constructively. If two polynomials of polynomial degree at most $n$ fit the data, their difference has $n+1$ distinct roots of a polynomial, so it must be zero. Repeated interpolation nodes with conflicting values have no solution, and repeated equal values provide no additional independent condition; prescribed derivatives instead lead to Hermite interpolation. Without the polynomial degree restriction there are infinitely many solutions:where $q$ is any polynomial. By the factor theorem, this describes every polynomial fitting the same data. In the example, $x^2+1+cx(x-1)(x-2)$ fits for every $c$, but at $x=1/2$ its value is $5/4+3c/8$; exact agreement at the three interpolation nodes alone says nothing about that intermediate value.

The usefulness of polynomial interpolation is that data become an explicit function whose values, derivatives, and integrals can be calculated. Differentiating or integrating the Lagrange cardinal polynomials produces differentiation weights from nodal evaluations or a quadrature rule. Exact passage through observations is the defining constraint. In polynomial regression, linear least squares instead chooses a polynomial degree and minimizes a sum of squared residuals, generally without passing through every observation. Interpolating noisy values reproduces their errors at the interpolation nodes and can amplify them between interpolation nodes; polynomial interpolation does not itself smooth measurement noise. A linear least squares problem with $n+1$ distinct interpolation nodes and polynomial degree at most $n$ does happen to attain zero residual at the same interpolation polynomial.

A polynomial regression of degree two is linear in its unknown regression coefficients despite being curved in the predictor. Its design matrix has columns $1,x,x^2$; three distinct predictor values give full column matrix rank.

Training error is the average loss function evaluated on the observations used to fit the prediction rule. It is the empirical risk of the fitted rule on its own training data. With squared-error loss, it is the average squared regression residual, measuring fit to observed responses rather than performance on fresh responses. Flexible fitting usually makes it optimistically biased for prediction error, as quantified by prediction optimism.

For a reproducible polynomial regression example, take $x_i=-1+2i/11$, $i=0,\dots ,11$, and the known noiseless target $f(x)=1+x/2+2x^2$. Construct responses $y_i=f(x_i)+e_i$ with fixed disturbancesOrdinary least squares over polynomials of polynomial degree at most 1, 2, and 7 gives training errors $0.518937$, $0.015784$, and $0.009257$. The monotone improvement follows because these candidate families are nested: every smaller-degree fit remains available in the larger family. This argument concerns minimum training error, not generalization, and assumes the fitting mathematical optimization is solved. Degrees 2 and 7 fitted instead to the exact noiseless responses $f(x_i)$ have mathematically zero training error; the script obtains approximately $5.31\times 10^{-31}$ and $7.82\times 10^{-31}$ from rounding. Extra flexibility alone does not force overfitting.

Underfitting occurs when a prediction rule fails to capture useful structure in the target relation. A straight-line polynomial regression fitted to a genuinely quadratic noiseless target illustrates a representational limitation. Poor prediction alone does not identify the cause: compare candidate rules using cross-validation or a validation set, while keeping the final test set reserved for assessment.