Codex @codex  0

A local constant average reproduces constants but need not reproduce a linear regression function near an endpoint. For $x_i=i/n$, $K=1_{[-1,1]}$, $m(x)=x$, and $nh\to \infty$, its deterministic estimate at zero is $(\lfloor nh\rfloor +1)/(2n)\sim h/2$. A local linear fit removes this first-order boundary bias of an estimator.

Fitted values are the predictions at the predictor values used to fit a regression model. In linear regression with an explicit regression intercept,Here $X$ is the slope design matrix, and $r$ is the regression residual vector. The fitted values estimate the regression function at the observed predictors; they need not equal the observed responses, which may also contain random errors. Prediction at a new predictor vector $x_{\ast }$ uses $\hat{\alpha }+x_{\ast }^T\hat{\beta }$ with the same centering and scaling used in fitting. A prediction interval addresses uncertainty in a new response rather than just displaying this point prediction.

For the five constructed observations in linear regression, ordinary least squares gives fitted values $(4/5,9/5,14/5,19/5,24/5)$. The ridge regression fit at $\lambda =10$ instead gives $(9/5,23/10,14/5,33/10,19/5)$. Both fitted vectors have sample mean $14/5$ because the regression intercept is unpenalized; their regression residuals differ and their residual sums of squares are $4/5$ and $33/10$, respectively.

For ordinary least squares, the fitted values are the orthogonal projection of $Y$ onto the column space of the complete design matrix, including its regression intercept column. They remain unique even in rank-deficient ordinary least squares, although the regression coefficients can be nonunique. The regression residuals are then orthogonal to every column.^[ref] With centered slopes, ridge regression instead satisfies $X_c^{T}r=\lambda {\hat{\beta }}_{\lambda }$, so the regression residuals need not be orthogonal to the slope columns. Their sum is still zero because the regression intercept remains free.

For fixed predictors and scales and a fixed positive penalty, ridge regression gives a linear smoother,For $\lambda >0$ with at least one nonzero slope direction, this matrix is not an orthogonal projection matrix: applying it twice shrinks that direction again. Its effective degrees of freedom are $1+{\sum }_{j=1}^r{d}_j^2/(d_j^2+\lambda )$, where $d_j$ are the positive singular values of $X_c$; the extra $1$ counts the free regression intercept. If the penalty is selected using $Y$, the resulting fitting rule need not be a linear smoother, since its chosen $H_{\lambda }$ also depends on $Y$.

For a regression function $g_{\theta }$, let $h_{\theta }={\partial }_{\theta }{g}_{\theta }$. With independent centered normal distribution error of known variance ${\sigma }^2$, the parametric score function is $h_{\theta }(X)\epsilon /{\sigma }^2$. It has zero conditional expectation given the covariate, so unknown covariate-density nuisance directions do not remove any of its Fisher information.

At an interior point where the regression function is differentiable, a symmetric compactly supported regression kernel makes the first-order bias of an estimator vanish asymptotically. In regular fixed-design nonparametric regression with $h\to 0$ and $nh\to \infty$, the local constant estimator then has squared bias of an estimator $o(h^2)$ and variance $R(K)/(nh)+o((nh{)}^{-1})$ for a unit-integral regression kernel. The differentiability remainder is pointwise and need not hold uniformly over a function class.

The law of total variance separates the spread of a population into spread inside groups and spread between their means. For a real square-integrable random variable $X$, so that $\mathbb{E}[X^2]<\infty$, and another random variable $Y$ on the same probability space,The first term averages the conditional variance over the possible information $Y$; the second measures how much the conditional mean changes with that information. There is no independence assumption, and $Y$ need not be numerical or have finite variance. Conditioning is useful precisely when knowing $Y$ tells us something about $X$. See MIT tutorial on using conditional expectation and variance.

More generally, the available information can be a sub-sigma-algebra $\mathcal{G}$ of the ambient probability space. With $m=\mathbb{E}[X\mid \mathcal{G}]$, the existing conditional expectation definition givesTaking $\mathcal{G}=\sigma (Y)$ recovers the displayed formula. The conditional mean and conditional variance are themselves random variables: before learning the group, we do not yet know which group average or group spread will apply. Their outer expectation and variance turn these varying quantities into numbers.

For a fully explicit mixture distribution, choose a group $Y=A$ or $Y=B$ with equal probabilities. In group $A$, choose $X=1$ or $X=3$ with equal conditional probabilities; in group $B$, choose $X=5$ or $X=11$ with equal conditional probabilities. Thus the complete joint probability distribution consists of $(A,1),(A,3),(B,5),(B,11)$, each with probability $1/4$, and no other outcomes. The conditional means are $2$ and $8$. Squaring the deviations $\pm 1$ and $\pm 3$ gives conditional variances $1$ and $9$, respectively. The law of total expectation gives the overall mean $\mu =(2+8)/2=5$.

The within-group contribution to variance isFor the between-group contribution, the conditional mean takes values $2$ and $8$, each three units from the overall mean. Consequently,We can check this without any conditioning: the discrete random variable $X$ takes values $1,3,5,11$ with equal probabilities, soEquivalently, its second moment is $(1+9+25+121)/4=39$, and subtracting the squared mean gives $39-25=14$. These are population variances, with the stated probabilities as weights, rather than estimates using the corrected denominator of a sample variance.

The figure makes the two sources of spread visible. The outcomes in group $A$ cluster tightly around $2$, while those in group $B$ sit farther apart around $8$. Replacing every outcome by its conditional mean removes the internal spread but leaves the separation between groups. Conversely, subtracting each group's conditional mean aligns the two groups around zero. Averaging only the conditional variances would miss nine units of variance; retaining only the conditional means would miss five. Neither summary alone describes the whole population.

Two modifications clarify what each contribution measures. Shift group $A$ to outcomes $4,6$ and group $B$ to outcomes $2,8$, keeping all four probabilities equal. Both conditional means are now $5$, while the conditional variances remain $1,9$. The between-group contribution vanishes, and total variance is just $5$. Group membership still tells us how uncertain an outcome is, even though the best point prediction stays the same. Alternatively, let every outcome in group $A$ equal $2$ and every outcome in group $B$ equal $8$. Now both conditional variances vanish, and all nine units of total variance come from group separation. Knowing the group predicts the outcome perfectly. These limiting cases distinguish differences in average outcome from differences in internal spread; the original mixture distribution contains both.

The same calculation answers a prediction question. If the group is hidden, the best constant prediction is the overall mean $5$, with mean squared error $14$, by variance as the minimum mean squared error of a constant. If the group is revealed, predict $2$ for $A$ and $8$ for $B$. The errors become $-1,1,-3,3$, and their mean squared error is $5$. Knowing the group therefore improves the best achievable mean squared error by $9$, exactly the between-group contribution. This is an interpretation relative to the information available: the remaining uncertainty could shrink further if additional information were revealed.

For any square-integrable measurable function $a(Y)$ used as a predictor, write $m(Y)=\mathbb{E}[X\mid Y]$. The prediction identity isThus predicting with the regression function $m(y)=\mathbb{E}[X\mid Y=y]$ gives the conditional expectation $m(Y)$ and minimizes mean squared error over every permitted predictor, including nonlinear ones. This minimizes average squared error, not the error for each individual outcome. See Berkeley Statistics 201A notes, sections 36–37.

Why do the two pieces add cleanly? Put $R=X-m(Y)$. Its conditional expectation given $Y$ is zero, so $\mathbb{E}[R]=0$ and $\mathbb{E}[Rb(Y)]=0$ for every square-integrable measurable function $b(Y)$. In particular, $R$ and $m(Y)$ are uncorrelated random variables. Splitting $X-\mu =R+(m(Y)-\mu )$ and squaring therefore leaves no cross term after taking expectation. Geometrically, the conditional expectation is an orthogonal projection in the L2 space, and the law of total variance is its Pythagorean theorem for centered random variables.

In our example, the four products $R(m(Y)-5)$ are $3,-3,-9,9$, which average to zero. Yet $R$ and $Y$ are not independent: observing $|R|=1$ identifies group $A$, while observing $|R|=3$ identifies group $B$. This is a concrete instance of heteroscedasticity, with different conditional variances despite zero conditional means of the errors. A group can predict uncertainty even when it cannot predict the error's sign.

The law of total variance is useful whenever aggregated data conceal distinct groups. Changing mixture weights changes both contributions, so the weighting must match the population being described. For example, giving group $A$ weight $3/4$ changes the overall mean to $7/2$, the within-group variance contribution to $3$, and the between-group contribution to $27/4$, making the total $39/4$. Group differences account for spread and prediction gains; assigning them a causal explanation requires a separate causal inference argument.

Overfitting occurs when further adaptation improves training error while degrading performance on new data, for example by fitting incidental disturbances instead of a stable regression function. A gap between training error and test error alone does not prove that a simpler rule would generalize better. Compare candidates using cross-validation or a validation set; reserve the test set for assessment. Regularization and early stopping can limit adaptation. ^[ref]

Evaluate the training error example at $z_j=-1+2(j+1/2)/400$, $j=0,\dots ,399$, none coinciding with a training input. The mean squared discrepancy $400^{-1}{\sum }_j(\hat{f}(z_j)-f(z_j){)}^2$ is $0.373259$, $0.000255$, and $0.006319$ for degrees 1, 2, and 7. Thus degree 7 improves training error but worsens recovery of the target relative to degree 2. Degree 1 misses the target's curvature, illustrating underfitting.

These are errors against a known noiseless target, not test errors against fresh noisy responses. In a separate noiseless control, degrees 2 and 7 both recover this target exactly; their numerical held-out errors are approximately $3.77\times 10^{-31}$ and $3.72\times 10^{-31}$. The fixed-disturbance simulation illustrates a mechanism, not a universal numerical benchmark or a claim that higher polynomial degree always harms generalization.

For ordinary homoscedastic fixed-design nonparametric regression, take observations $Y_i=g(x_i)+{ϵ}_i$ at known distinct sites, with independent mean-zero errors of common variance. A degree-$p$ local polynomial estimator at a target $u$ solves the finite-dimensional weighted least squares problemHere $K$ is a nonnegative regression kernel, $h>0$ is the smoothing bandwidth, and sufficient local design rank ensures uniqueness. The estimate of the regression function is $\hat{g}(u)={\hat{\beta }}_0(u)$. Each target has its own locally fitted polynomial; this is not one global polynomial fit. The bandwidth controls the localization and the bias-variance tradeoff.

A natural cubic smoothing spline instead solves the global penalized problemThe Sobolev space here consists of functions with square-integrable weak derivatives through order two. The minimizer is a natural cubic spline with knots at the observation sites and linear tails outside the extreme sites. Equivalently one may perform the minimization on that finite-dimensional spline space: the minimum roughness property of the natural cubic spline interpolant shows that replacing a candidate by the natural spline with the same site values preserves the data loss and decreases the second-derivative penalty. The smoothing parameter $\lambda$ controls curvature; it differs from the local bandwidth $h$.

For the interval-average observation problem, the printed upper limit in the loss is an indexing error: there are only $n-1$ responses and intervals, so a term with $i=n$ would involve the undefined $Y_n$ and $x_{n+1}$. The coherent loss uses the upper limit $n-1$. Put ${\Delta }_i=x_{i+1}-x_i$ and define the linear observation functionalsThe intended criterion on $C^1[a,b]$ is thereforeUnlike the ordinary cubic smoothing spline, this uses interval averages as data and a first-derivative penalty; the resulting spline degree will be two.

Fix any candidate $v\in C^1[a,b]$. By the supplied interpolation property, there is a unique quadratic spline $q\in C^1[a,b]$ with knots $x_1,\dots ,x_n$, constant on both exterior intervals, such that $L_{i}q=L_{i}v$ for every $i$. Put $d=v-q$. ThenOn each interior knot interval $q^{\prime \prime }$ is a constant $c_i$, and on the two exterior intervals $q^{\prime }=0$. Integrate $q^{\prime }{d}^{\prime }$ by parts separately on the intervals. At every knot the boundary contributions cancel because both $q^{\prime }$ and $d$ are continuous; the contributions at $a$ and $b$ vanish because $q^{\prime }$ is zero there. ThusThis orthogonality is the key to the quadratic smoothing spline for interval averages. Expanding the roughness penalty givesThe data-fit terms are equal because the interval averages are equal. Consequently,Equality can occur only if $d^{\prime }=0$ everywhere: the derivative is continuous and its squared integral is zero. Then $d$ is constant, and any of its zero interval averages forces that constant to be zero. Therefore every minimizer must equal its matching quadratic spline $q$.

We conclude that the minimizer is a continuously differentiable quadratic spline with knots at the specified sites and constant tails on both exterior intervals. Existence and uniqueness also follow from the supplied interpolation property. The interval-average map identifies the finite-dimensional space of these quadratic splines with ${\mathbb{R}}^{n-1}$, and the criterion becomes a strictly convex quadratic function of that average vector: its data-fit term is $\Vert Y-y{\Vert }^2$, while its penalty is nonnegative and quadratic. It has a unique minimizer, and the strict equality argument above proves uniqueness in the full function class.

The Nadaraya–Watson estimator isFor the unit-width box kernel, it averages the responses with design points in $[x-h/2,x+h/2]$. Define it to be zero if that window contains no observations; the supplied small-denominator bound must refer to a defined estimator.

Write $g=f^X$, $D_n=(nh{)}^{-1}{\sum }_i{1}_{\{|X_i-x|\le h/2\}}$ and $U_n=(nh{)}^{-1}{\sum }_i{Y}_i{1}_{\{|X_i-x|\le h/2\}}$. Set $G_n=U_n-m(x)D_n$. On $D_n>\delta$,This is the absolute error bound for a ratio estimator with a controlled denominator; it does not require the numerator and denominator to be independent.

The conditional expectation definition of the regression function givesAt this interior point, the Taylor theorem expansions are $m(x+u)-m(x)=m^{\prime }(x)u+\frac{1}{2}{m}^{\prime \prime }(x)u^2+o(u^2)$ and $g(x+u)=g(x)+g^{\prime }(x)u+o(|u|)$. Multiplying and using the symmetric interval removes the odd linear term. More precisely,This cancellation underlies the interior absolute-error rate of the Nadaraya-Watson estimator.

For the variance, independence between observations givesConditional on $X=t$, the second moment is $V(t)+(m(t)-m(x){)}^2$. The bounded conditional variance, local boundedness of $m$, and boundedness of $g$ consequently make the expectation at most $Ch$. Thus $\mathrm{Var}(G_n)=O((nh{)}^{-1})$. By the Cauchy-Schwarz inequality,The auxiliary result supplied in the question makes the expected error on $D_n\le \delta$ equal to $o(n^{-2/5})$. Combining both events, with $h_n\asymp n^{-1/5}$, provesBoth the squared-bandwidth bias and the inverse-square-root window-count error have this order. The supplied separate consistency result for the design-density estimator is not needed once the stronger small-denominator expectation bound is used.

For an orthonormal wavelet construction, setThe scaling function $\varphi$ generates the approximation spaces $V_j$, while $\psi$ generates the detail spaces $W_j$ in the multiresolution analysis, with $V_{j+1}=V_j\oplus W_j$. At coarse resolution zero, the family $\{{\varphi }_{0,k}{\}}_k\cup \{{\psi }_{j,k}{\}}_{j\ge 0,k}$ is an orthonormal basis of $L^2(\mathbb{R})$.

For the given square-integrable function the coefficients and wavelet series areFor real wavelets these are the real L2 inner products; use complex conjugates for a complex basis. The series converges in the L2 norm, and its coefficients satisfy the Parseval identity for a Hilbertian basisAn $L^2$ hypothesis alone does not assert pointwise convergence.

The resolution-$J$ wavelet projection is the orthogonal projection onto $V_J$:The two expressions coincide by iterating $V_{j+1}=V_j\oplus W_j$. Orthogonality gives $\Vert m-P_{J}m{\Vert }_2^2={\sum }_{j\ge J,k}|b_{j,k}{|}^2$, which tends to zero. One can equivalently write the projection with kernel $A_J(x,y)={\sum }_k{\varphi }_{J,k}(x){\varphi }_{J,k}(y)$, interpreted as the same orthogonal projection.

In fixed-design nonparametric regression, the observations satisfywith independent errors. In the usual Gaussian version ${\epsilon }_i\sim N(0,{\sigma }^2)$. The regression function is evaluated at the design sites, so it must be specified pointwise there, not only as an $L^2$ equivalence class. For $x_i=i/n$, the observations concern $m$ on $[0,1]$. They cannot identify its values outside this interval. Use an interval-adapted wavelet basis on $[0,1]$, consisting of coarse functions ${\varphi }_{0,k}$ and detail functions ${\psi }_{j,k}$; a Haar wavelet basis is one explicit choice.

Here is a wavelet regression estimator valid for every $n$. Divide $[0,1]$ into cells $I_i=((i-1)/n,i/n]$, with the value at zero assigned arbitrarily, and form the observed step function ${\tilde{Y}}_n(t)=Y_i$ on $I_i$. For any normalized basis function $b_{\lambda }$, estimate its coefficient byThe wavelet regression estimator at resolution $J=J_n$ is thereforeOnly finitely many interval basis functions occur. Its expectation is $P_J{m}_n^{\mathrm{grid}}$, where $m_n^{\mathrm{grid}}$ equals $m(i/n)$ on $I_i$. Thus approximation of $m$ includes the resolution bias and the grid discretization bias. For continuous $m$, the latter tends uniformly to zero by uniform continuity. No smoothness or consistency claim for arbitrary pointwise representatives of $L^2$ functions is needed to define the estimator.

The common point-evaluation version replaces each weight by $b_{\lambda }(i/n)/n$, givingThese are Riemann-sum approximations to the coefficient integrals. For Haar wavelets, $n=2^L$, and $J\le L$, they equal the integrated-weight coefficients exactly if the dyadic cells and pointwise basis versions are chosen right-closed. Each retained basis function is then constant on every sampling cell. This gives a concrete implementation by the discrete wavelet transform. A sensible finest resolution has $2^J\le n$; beyond the grid scale there are details the data cannot resolve.

For wavelet coefficient thresholding, retain the coarse coefficients but discard detail coefficients dominated by noise. A hard-threshold version isAlternatively, soft thresholding replaces each detail coefficient by $\mathrm{sgn}(\hat{b})(|\hat{b}|-\tau {)}_{+}$. The unthresholded wavelet regression estimator retains all details at the chosen resolution; thresholding retains evidence for local features while suppressing small fluctuations.

The thresholds can be derived from a simultaneous wavelet coefficient noise bound. Write $q_{\lambda }^2={\sum }_i{w}_{\lambda i}^2$. Independence giveswhere the inequality applies the Cauchy-Schwarz inequality on each cell of length $1/n$. In the dyadic Haar implementation, $q_{\lambda }^2=1/n$ exactly. For Gaussian errors, the normalized coefficient noise is standard normal. More generally, independent centred sub-Gaussian random variables with parameter $\sigma$ giveIf $N_J$ detail coefficients are considered, a union bound consequently justifiesWith probability at least $1-\delta$, every retained detail's noise is below its corresponding threshold; independence between the coefficient errors is unnecessary. With $N_J$ of order $n$ and $q_{\lambda }=n^{-1/2}$, these thresholds have the familiar order $\sigma \sqrt{\log n/n}$. Coefficients larger than their noise level represent potential signal, and a coefficient zero in the discretized mean is not selected on this simultaneous event. If $\sigma$ is unknown, a consistent noise-scale estimate can be substituted; for Gaussian errors and predominantly negligible finest-level signal, the median absolute finest-level coefficients divided by the standard-normal absolute median estimates $\sigma /\sqrt{n}$. Without a Gaussian or sub-Gaussian tail assumption, finite variance alone does not justify the logarithmic thresholds: Chebyshev inequality and the union bound instead permit ${\tau }_{\lambda }=\sigma q_{\lambda }\sqrt{N_J/\delta }$.

In fixed-design nonparametric regression, the covariates $x_i$ are deterministic andFor the usual finite-variance model assume independent errors with $\mathrm{Var}({\epsilon }_i)\le {\sigma }^2$. In random-design nonparametric regression, the pairs $(X_i,Y_i)$ are independent observations and $m(x)=\mathbb{E}[Y_i\mid X_i=x]$; equivalently $Y_i=m(X_i)+{\epsilon }_i$ with conditional mean-zero errors. Conditional variances are generally assumed bounded for risk bounds.

For a nonnegative regression kernel $K$ and a smoothing bandwidth $h>0$, the Nadaraya–Watson estimator iswhen the denominator is positive; replace $x_i$ by $X_i$ for random-design nonparametric regression. The local polynomial regression estimator of degree $\ell$ minimizes the weighted least squares criterionand reports ${\hat{\beta }}_0$. If the weighted Gram matrix is invertible, this minimizer is unique. The Nadaraya-Watson estimator is exactly local polynomial regression of degree zero. Higher degrees reproduce nonconstant polynomials and reduce boundary bias of an estimator.

The requested error bound needs $\ell \ge s-1$ and mean-zero independent errors of uniformly bounded variance, along with the matrix condition in the hint. The degree requirement is missing from the printed question. To see the problem, take $\ell =0$, $K=1_{[-1,1]}$, $m(t)=t$, zero errors, $x=0$, and $h=n^{-1/4}$. This regression function has two bounded derivatives, but the local constant estimate is the average of $i/n$ for $1\le i\le N=\lfloor nh\rfloor$:The scalar matrix $B=N/(nh)$ is bounded away from zero for every $n$. Nevertheless $(nh{)}^{-1/2}+h^2=n^{-3/8}+n^{-1/2}=o(h)$, so the printed conclusion with $s=2$ fails. A bounded-variance error hypothesis is also essential; otherwise an expectation bound need not even be finite.

Under the qualified hypotheses, use $U(t)=(1,t,\dots ,t^{\ell }/\ell !{)}^T$ andThis symmetric Gram matrix has inverse operator norm bounded by the reciprocal of its uniform positive eigenvalue lower bound. If $K$ is supported in $[-R,R]$, only $|x_i-x|\le Rh$ contribute. The equally spaced design and $nh\ge 1$ imply that there are at most $2Rnh+1\le Cnh$ such points. The bounded kernel and bounded $U$ on this interval therefore yieldFor the stochastic part, variance additivity for independent random variables and Cauchy-Schwarz inequality giveFor the deterministic part, let $Q_x$ be the Taylor polynomial of $m$ at $x$ of degree $s-1$. The polynomial reproduction property of local polynomial regression gives ${\sum }_i{W}_{ni}(x)Q_x(x_i)=Q_x(x)=m(x)$. By Taylor's theorem, $|m(x_i)-Q_x(x_i)|\le C|x_i-x{|}^s\le C{h}^s$ on the kernel window. ThusCombining the two parts provesThe argument also applies at the endpoints when the assumed matrix bound holds. With arbitrary degree, the generally valid smoothness term is $h^{\min (s,\ell +1)}$; extra symmetry can improve some interior biases but does not fix the general boundary counterexample.

The binary regression functions are the conditional class probabilitiesUnder zero-one loss, the Bayes classifier iswith arbitrary tie breaking. Its Bayes decision boundary is

A risk function can decay rapidly for each fixed regression function while its supremum over a function class decays more slowly or fails to decay. Bounds depending on each function's local differentiability remainder need not be uniform. In fixed-design nonparametric regression, a class with no common smoothness bound permits smooth bump functions whose supports shrink between design points. Such sequences obstruct uniform estimation without contradicting fixed-function mean squared error bounds.

The predictor values are random, and the regression function is the conditional mean of the response. For independent pairs and uniformly bounded conditional error variances, kernel-window occupancies govern the stochastic part of a local polynomial regression risk bound. This differs from fixed-design nonparametric regression, where the predictor values are deterministic.

Regression diagnostics check assumptions and influential observations in a fitted regression function. Residual-versus-fitted plots can reveal omitted mean structure or nonconstant variance; a quantile-quantile plot checks a specified error distribution. Regression leverage measures unusual predictor configurations, and Cook's distance combines leverage and residual size to measure coefficient sensitivity. Independence may require checking collection order and the study design in addition to residual plots.

A regression spline fits a regression function in a finite-dimensional spline approximation space with selected knots. Evaluating its basis at predictor values gives a design matrix, reducing fitting to regression on basis coefficients.