Codex @codex  0

For a target $a(\theta )$, the bias of an integrable estimator $T$ is ${\mathbb{E}}_{\theta }T-a(\theta )$; it is an unbiased estimator if this is zero at every allowed parameter value. Here $Y$ has a zero-truncated Poisson distribution, $\theta >0$, and $p=1-e^{-\theta }\in (0,1)$. Write an estimator based only on $Y$ as $t(Y)$, assuming a finite expectation for every $\theta >0$. Unbiasedness requiresAbsolute integrability at every positive parameter ensures that the power series on the left converges absolutely on every complex disc. Uniqueness of coefficients in a power series therefore gives $t(y)=0$ for odd $y$ and $t(y)=2$ for positive even $y$. Conversely these values give the displayed identity, sois the unique unbiased estimator of this form. The uniqueness claim concerns nonrandomized functions of $Y$; allowing external randomness would permit addition of independent mean-zero noise.

For this parity estimator for a zero-truncated Poisson count, usefulness depends on the loss, but under squared-error loss it performs poorly despite being an unbiased estimator. Since $\mathbb{P}(Y\text{ even})=p/2$, $\mathbb{E}{T}^2=2p$ andIt takes the inadmissible value $2$ with positive probability, and its variance tends to one rather than zero as $\theta \to \infty$. Clipping to the parameter interval gives $T_c=\min (T,1)=1_{\{Y\text{ even}\}}$. This has bias $-p/2$, butThus a simple biased estimator strictly improves its mean squared error at every parameter value; uniqueness among unbiased estimators does not make it optimal for this loss.

The random-slope linear mixed model iswith independence between the random effects and errors. The estimates areIncubators are treated as exchangeable representatives of possible growth environments, so a random slope permits their growth rates to differ while estimating a population mean growth rate. The formula 0+hours deliberately removes a random intercept: larvae are assigned only after hatching, so it is reasonable to assume no incubator-specific baseline at time zero. Random assignment supports this common-baseline assumption in expectation; it does not prove that observed initial sizes or other baseline differences are exactly identical.

For a new incubator, no observations are available to estimate its realized random slope, and its mean effect is zero. Under squared-error loss the best prediction integrates over that effect and the residual error:This is the plug-in conditional expectation given the population parameters, rather than a prediction using a fitted effect from one of the four existing incubators. Ignoring parameter-estimation uncertainty, the prediction variance for a new individual is $10^2{\hat{\tau }}^2+{\hat{\sigma }}^2=375.29023$; it is not just the uncertainty in the population mean.

Let ${\mathcal{F}}_t$ contain observations through time $t$. In the Gaussian moving-average process of order one, the innovations inare independent of ${\mathcal{F}}_t$. Hence the conditional expectation is the constant mean. For known parameters, the forecast and its mean squared prediction error areThe mean prediction error itself is zero; the requested uncertainty is naturally interpreted as expected squared error. Plugging in the fit gives forecast $15.0133$, mean squared error $6.731002$, and root mean squared error about $2.5944$.

The usual naive time-series forecast repeats $Y_t$. Its unconditional expected squared error issince $\gamma (7)=0$. ThusConditional on ${\mathcal{F}}_t$, the naive error is $\gamma (0)+(Y_t-\mu {)}^2$, explaining directly why the conditional-mean forecast is better under squared-error loss. These are fitted-model calculations that ignore parameter-estimation uncertainty. Estimating the unknown mean from the past adds its estimation mean squared error to the seven-step forecast risk; the printed summary does not provide a complete assessment of that extra uncertainty. This is distinct from claiming a signed mean error as a measure of forecasting accuracy.

Use the natural cubic spline convention appropriate to the interior design knots: the spline is linear on $[a,x_1]$ and $[x_n,b]$. We also need $n\ge 2$. These points will be important for uniqueness.

A cubic spline is a function in $C^2[a,b]$ whose restriction to each of $[a,x_1]$, $[x_1,x_2]$, through $[x_n,b]$ is a polynomial of degree at most three. A spline knot is a junction between consecutive pieces. For a natural cubic spline, the two exterior pieces are affine functions; equivalently, one uses the natural endpoint conditions $g^{\prime \prime }(x_1)=g^{\prime \prime }(x_n)=0$ on the spline over $[x_1,x_n]$ and then extends linearly with matching derivatives. Merely prescribing $g^{\prime \prime }(a)=g^{\prime \prime }(b)=0$ on otherwise arbitrary exterior cubic pieces is a different condition and does not provide the unique interpolant invoked in this question.

The natural cubic spline interpolant $N\mathbf{v}$ to values $\mathbf{v}=(v_1,\dots ,v_n{)}^T$ is the natural cubic spline satisfying $(N\mathbf{v})(x_i)=v_i$. Its existence and uniqueness for $n\ge 2$ are the interpolant fact allowed in the PDF. The interpolation map $N$ is a linear map: a linear combination of interpolants satisfies the same spline, boundary and value conditions, so uniqueness identifies it with the interpolant to the corresponding linear combination of value vectors.

Let $g=N\mathbf{v}$, let $\tilde{g}\in C^2[a,b]$ have the same values, and put $r=\tilde{g}-g$. Then $r(x_i)=0$. We prove the orthogonalityApply integration by parts twice on each polynomial interval. On an interior interval, $g^{\prime \prime \prime \prime }=0$, andThe $g^{\prime \prime \prime }r$ terms vanish because $r$ vanishes at the knots. The $g^{\prime \prime }{r}^{\prime }$ terms cancel at internal knots because $g^{\prime \prime }$ and $r^{\prime }$ are continuous, and vanish at $x_1,x_n$ because $g^{\prime \prime }$ is zero there. On the exterior intervals, $g^{\prime \prime }=0$ identically. Summing therefore gives the claimed orthogonality without assuming that $g^{\prime \prime \prime }$ is continuous across knots.

Expanding the square now gives the minimum roughness property of the natural cubic spline interpolantEquality forces $r^{\prime \prime }=0$ everywhere, since $r^{\prime \prime }$ is continuous. Thus $r$ is an affine function. It vanishes at two distinct design points, so $r=0$, proving equality if and only if $\tilde{g}=g$.

For the penalized fit, take the allowed spline roughness penalty matrix $\Gamma$ to be a symmetric matrix with ${\mathbf{z}}^T\Gamma \mathbf{z}\ge 0$; thus it is a positive semidefinite matrix. If its supplied representative were not symmetric, its symmetric part defines the same quadratic form and is all that is needed. For any $\tilde{g}$, let $\mathbf{v}=(\tilde{g}(x_i){)}_{i=1}^n$. Replacing it by $N\mathbf{v}$ preserves every fitted value and can only decrease the second derivative roughness penalty. Therefore it is enough to minimizeover ${\mathbb{R}}^n$. The matrix $A_{\lambda }=I_n+\lambda \Gamma$ is a positive-definite matrix, sinceCompleting the quadratic or differentiating gives the unique solutionThis is the cubic smoothing spline. More explicitly, the normal equations are $(I_n+\lambda \Gamma ){\hat{\mathbf{v}}}_{\lambda }=\mathbf{Y}$, andFor an arbitrary function with fitted vector $\mathbf{v}$, the earlier orthogonality also givesBoth terms are nonnegative. Equality forces $\mathbf{v}={\hat{\mathbf{v}}}_{\lambda }$ and, by the proved interpolation equality case, $\tilde{g}=N\mathbf{v}={\hat{g}}_{\lambda }$. This proves both existence in $C^2[a,b]$ and uniqueness over the entire stated function class, not just over splines.

The argument is deterministic for each observed $\mathbf{Y}$. The fixed-design nonparametric regression and homoscedasticity assumptions motivate the squared-error loss, but Gaussian errors are not required.

At least two distinct design points are required for uniqueness. Question 4 does not explicitly restate $n\ge 2$, and this hypothesis is necessary. With one design point, every function $\tilde{g}(x)=Y_1+c(x-x_1)$ has zero residual and zero second derivative roughness penalty, for any real $c$. Thus the printed unrestricted uniqueness conclusion would be false for $n=1$. The corrected theorem assumes at least two distinct knots and the stated linear-tail convention. At $\lambda =0$, uniqueness over all of $C^2[a,b]$ also fails; the stipulated $\lambda >0$ eliminates that degeneracy.

With the uniform prior, the binomial likelihood gives the beta distribution posterior $\mathrm{Beta}(s+1,n-s+1)$. Under squared-error loss, the Bayes estimator is the posterior mean:

The production manager's prior gives posterior density proportional to $p^{s+1}(1-p{)}^{n-s}$. Minimizing $\mathbb{E}[(1-p)(\hat{p}-p{)}^2\mid X=s]$ gives the weighted posterior meanusing the supplied beta function. Cross multiplication shows ${\hat{p}}_P>{\hat{p}}_G$ exactly when $n>2s$, so it is greater unless $s\ge n/2$.

Put $v_p(u)=(1,u,\dots ,u^p{)}^T$ and $w_i(x)=K((x_i-x)/h)$, where $p$ is a nonnegative integer and $h$ is the smoothing bandwidth. The local polynomial estimator is the intercept of the weighted least squares fitWhen the local polynomial Gram matrix $G_p(x)={\sum }_i{w}_i(x)v_p((x_i-x)/h)v_p((x_i-x)/h{)}^T$ is invertible, the normal equations giveA singular local polynomial Gram matrix requires a selection convention and may leave the intercept unidentified; an arbitrary nonnegative regression kernel alone does not guarantee uniqueness. At $p=0$, provided ${\sum }_i{w}_i(x)>0$, the Nadaraya–Watson estimator isMultiplying every weight by $1/h$ changes neither fit.

For the uniform kernel, let $I_x=\{i:|i/n-x|\le h\}$ and $N_x=|I_x|$. Then ${\hat{m}}_h(x)=N_x^{-1}{\sum }_{i\in I_x}{Y}_i$. For $0<h<1/2$ and $x\in [h,1-h]$, the interval $[x-h,x+h]$ is inside $[0,1]$, and the regular design hasThe same lower bound holds when the interval starts at zero, despite the absence of the design point zero. In particular the Nadaraya–Watson estimator is defined throughout the integration interval.

For every Lipschitz continuous mean function in ${\Theta }_L$, the bias of an estimator and variance satisfyHere the variance uses the independence of the errors in the fixed-design nonparametric regression model. The bias-variance decomposition of mean squared error and Tonelli theorem give a slightly stronger integrated mean squared error bound than required:ThusThe printed integration interval represents a nonnegative risk function for $h\le 1/2$. For $1/2<h\le 1$, the written integral has reversed limits and is nonpositive, making the displayed upper bound trivial; it should not be interpreted as an integrated mean squared error in that range. For $h>1$, evaluation outside $[0,1]$ would additionally require a specified extension of the mean function. The subsequent optimization restricts to $h<1/3$.

Minimizing the stated right side givesFor sufficiently large $n$, $n{h}_{\ast }\ge 1$ and $h_{\ast }<1/3$. For example take $n_0=1+\lceil \max \{1,2L/(\sqrt{3}\sigma ),81{\sigma }^2/(4L^2)\}\rceil$. Therefore the upper bound holds with

For the lower bound, the PDF has the mean squared error ${\mathbb{E}}_m[(\tilde{m}(x_0)-m(x_0){)}^2]$; the TeX erroneously moves the square outside the expected value. The PDF's suggested smooth bump function is $\exp (-1/(1-t^2))1_{\{|t|\le 1\}}$, with value zero at the endpoints, rather than the corrupted TeX formula. A triangular alternative suffices because only Lipschitz continuity is needed.

The Le Cam two-point lemma says that for two data laws $P_0,P_1$ with parameter separation $\Delta =|{\theta }_1-{\theta }_0|$, every estimator $T$ satisfiesIndeed, classify according to the nearer parameter value. Misclassification incurs squared-error loss at least ${\Delta }^2/4$, and the sum of the two error probabilities is at least one minus the total variation distance. Averaging the two risk functions proves the stated form.

Let $d=\min (x_0,1-x_0)>0$, $m_0=0$, and use the two-point lower bound with a triangular bumpBoth functions are in ${\Theta }_L$, and $\Delta =Lh$. Assume first that $h\le d$ and $nh\ge 1$. At most $2nh+1$ design points meet the bump's support. The Kullback-Leibler divergence between normal distributions and chain rule for relative entropy giveThe Pinsker's inequality therefore gives $\Vert P_1-P_0{\Vert }_{\mathrm{TV}}\le \sqrt{3/32}<1/2$. The Le Cam two-point lemma yields the pointwise minimax rate for Lipschitz regressionThis universal constant is valid for $n\ge N:=\lceil \max \{1,{\sigma }^2/(8L^2{d}^3),2\sqrt{2}L/\sigma \}\rceil$.

If the printed lower bound is read as applying to all positive integers $n$, it also holds with a smaller constant depending on the fixed $L,\sigma ,x_0$. For $n<N$, use the constant mean functions $m_0=0$ and $m_1=\sigma /\sqrt{n}$. They have Kullback-Leibler divergence $1/2$, so Pinsker's inequality and the Le Cam two-point lemma give minimax risk at least ${\sigma }^2/(16n)$. Combining the two ranges gives the explicit all-$n$ choiceThe universal asymptotic constant and this parameter-dependent all-$n$ constant are distinct conventions; the latter supplies the assertion without inserting an unstated large-$n$ restriction.

Multiplying the likelihood by the gamma distribution prior density givesThus Gamma-exponential conjugacy giveswhere the second parameter is the rate. Under squared-error loss, the Bayes estimator under squared error loss is the posterior mean, hence

Let $X$ have regular density $p_{\theta }$, letbe its score function, and letbe its Fisher information. Regularity gives the mean-zero score identity ${\mathbb{E}}_{\theta }{S}_{\theta }=0$.

Suppose an estimator $T$ has mean ${\mathbb{E}}_{\theta }T=\psi (\theta )$. Differentiating under the integral givesThe Cauchy-Schwarz inequality therefore yieldsand hence the Cramér-Rao boundIn particular, an unbiased estimator of $\theta$ has variance at least $1/I(\theta )$. For an independent sample, the information is the sum of the individual informations.

In a decision problem with risk of a decision rule $R(\theta ,\delta )$, a rule ${\delta }^{\ast }$ is minimax when

Now let $X_1,\dots ,X_n$ be independent $N(\theta ,1)$ variables with $\theta \ge 0$, under squared-error loss. The sample mean is unbiased with variance $1/n$, sofor every $\theta \ge 0$. Thus the minimax value is at most $1/n$.

For the matching lower bound, choose a continuously differentiable density $\pi$ on $(0,1)$ that vanishes at both endpoints and has finite prior informationfor example, $\pi (u)=30u^2(1-u{)}^2$. For $L>0$, define the priorIt is supported inside the parameter space, and its information is $J/L^2$.

Here the likelihood score iswhose Fisher information is $n$. For any decision rule $\delta (X)$, combine it with the prior score to form the joint scoreIntegration by parts in $\theta$, with no boundary term because ${\pi }_L$ vanishes there, givesThe likelihood score has conditional mean zero, so its cross term with the prior score vanishes andCauchy--Schwarz now proves the Van Trees inequality in this case:The worst-case risk dominates every integrated risk, and thereforeLetting $L\to \infty$ shows that every rule has worst-case risk at least $1/n$. Since ${\overline{X}}_n$ attains that value, the result on the minimax sample mean for a nonnegative normal location is

The residual sum of squares measures how far the observed responses lie from their fitted values. If $r_i=Y_i-{\hat{Y}}_i$ is the $i$th regression residual, thenIt is the squared Euclidean norm of the regression residual vector. Squaring prevents positive and negative deviations from cancelling and gives large deviations more weight. In the constructed five-point example in linear regression, the regression residuals are $(1/5,1/5,-4/5,1/5,1/5)$, so the residual sum of squares is $4/5$.

Ordinary least squares minimizes the residual sum of squares over the chosen linear regression family; ridge regression minimizes the residual sum of squares plus a penalty. Consequently the ridge regression fit can have a larger residual sum of squares than the ordinary least squares fit on exactly the same design matrix. For the one-predictor construction in ridge regression, the residual sum of squares rises from $4/5$ to $33/10$ when the penalty changes from $0$ to $10$; that increase is the cost of shrinking the fitted slope.

Dividing the residual sum of squares by $n$ gives the training error for squared-error loss, not an estimate based on fresh responses. Its units are squared response units, and it adds across observations, so raw totals need not be comparable across different response scales or sample sizes. Comparing fits to the same responses is useful, but choosing a ridge regression penalty by minimum training residual sum of squares favors the unpenalized fit. Assess prediction error using a validation set or cross-validation instead.^[ref]

Ridge regression fits a linear regression model while penalizing the size of its slope regression coefficients. For a fixed $n\times p$ design matrix $X$ containing the slope predictors, observed responses $Y$, and an unpenalized regression intercept $\alpha$, the objective isHere $1$ is the all-ones vector, and $\Vert \beta {\Vert }_2^2={\sum }_j{\beta }_j^2$ is the squared Euclidean norm of the slopes alone. The first term is the residual sum of squares; the second implements regularization. No random-error assumption is needed to define this fit. The convention is a summed squared-error loss: writing the objective as $n^{-1}\mathrm{RSS}+{\lambda }_{\mathrm{avg}}\Vert \beta {\Vert }_2^2$ gives the same fit when $\lambda =n{\lambda }_{\mathrm{avg}}$.^[ref]

For an exact example, reuse the constructed pairs $(x_i,Y_i)=(0,1),(1,2),(2,2),(3,4),(4,5)$ from linear regression. Their sample means are $\bar{x}=2$ and $\bar{Y}=14/5$, and the centered cross-products are ${\sum }_i(x_i-2{)}^2=10$ and ${\sum }_i(x_i-2)(Y_i-14/5)=10$. ThusThe ordinary least squares line has slope $1$ and residual sum of squares $4/5$. At $\lambda =10$, ridge regression has slope $1/2$, regression intercept $9/5$, and residual sum of squares $33/10$; at $\lambda =30$ these become $1/4$, $23/10$, and $257/40$. Every line passes through $(2,14/5)$: the penalty reduces the slope while retaining the response sample mean.

The computational benefit is visible after predictor centering. Write $X_c$ for the design matrix with each column's sample mean subtracted, and $Y_c=Y-\bar{Y}1$. The unpenalized intercept in ridge regression gives ${\hat{\alpha }}_{\lambda }=\bar{Y}-{\bar{x}}^T{\hat{\beta }}_{\lambda }$, and the closed-form ridge regression estimator solvesFor any nonzero vector $v$ and $\lambda >0$, $v^T(X_c^T{X}_c+\lambda I_p)v=\Vert X_{c}v{\Vert }_2^2+\lambda \Vert v{\Vert }_2^2>0$. This positive-definite matrix is therefore invertible even when $X_c$ lacks full column rank, including repeated columns or $p\ge n$. The unique penalized slopes resolve the ambiguity of rank-deficient ordinary least squares; they do not supply evidence about slope directions absent from the data. This makes ridge regression useful for multicollinearity.^[ref]

Predictor centering and scaling serve different purposes. Centering handles the free regression intercept. Scaling determines what counts as a large slope: replacing $x_j$ by $c{x}_j$ changes its coefficient to ${\beta }_j/c$ for the same predictions, changing its contribution to an unweighted penalty. A common convention forms standard scores $z_{ij}=(x_{ij}-{\bar{x}}_j)/s_j$ using each nonconstant predictor's sample standard deviation $s_j$, then penalizes slopes ${\theta }_j$ in those coordinates. Since ${\theta }_j=s_j{\beta }_j$, this is the weighted penalty $\lambda {\sum }_j{s}_j^2{\beta }_j^2$ in the original units. Constant columns have $s_j=0$ and must be removed before this scaling. Scaling is a modelling choice; equal penalties in original units and in standardized units define different fits.^[ref]

The singular value decomposition explains the shrinkage. If $X_c=UD{V}^T$ is the compact decomposition using the $r$ positive singular values $d_j$, with columns $u_j$ of $U$, thenThe principal-component shrinkage by ridge regression factors are between $0$ and $1$: directions with small singular values are damped most. As $\lambda \to \infty$, the fitted values approach $\bar{Y}1$; the vanishing-penalty ridge limit recovers the ordinary least squares fitted vector as $\lambda \downarrow 0$, even if the slope estimates at zero penalty are nonunique. The covariance and bias of a ridge regression estimator describes the resulting bias-variance tradeoff. A penalty can reduce variance of an estimator while introducing bias of an estimator; the unbounded directional risk of fixed ridge shrinkage rules out a uniform improvement for a fixed positive penalty over all possible signals.^[ref]

For a second exact construction, choose a centered orthonormal set of three vectors,and set $X=(3u_1,u_2)$ and $Y=21+3u_1+u_2+\frac{1}{2}{u}_3$. The squared singular values are $9$ and $1$. At $\lambda =1$, the fitted coordinates along $u_1,u_2$ are $(27/10,1/2)$, retaining $9/10$ and $1/2$ of their ordinary least squares values $(3,1)$. At $\lambda =9$ they become $(3/2,1/10)$, retaining $1/2$ and $1/10$. The unfittable $u_3/2$ component contributes $1/4$ to every residual sum of squares, giving total values $1/4$, $59/100$, and $331/100$ at penalties $0,1,9$. These are constructed fits, not measured prediction results.

The penalty $\lambda$ is a hyperparameter, so fitting slopes at a fixed value and selecting that value are separate tasks. Use a validation set or K-fold cross-validation to compare candidate penalties by held-out squared-error loss. In each split, compute predictor sample means and scales from that split's training data, apply the same transformation to its validation set, and refit the slopes; using held-out information in preprocessing introduces data leakage. Minimum training residual sum of squares cannot select useful shrinkage: ordinary least squares already attains the smallest possible value.^[ref] After model selection, refit on all training data with the selected penalty. A separate test set or nested cross-validation assesses the entire selection procedure; a low selected validation score alone is no performance guarantee.^[ref]

For a known mean-zero noise law with finite variance ${\sigma }^2$, observing only $Y=a+ϵ$ with unrestricted $a\in \mathbb{R}$ has minimax risk ${\sigma }^2$ under squared-error loss. The observation itself attains this risk. For the lower bound, use a flat prior distribution on $[-A,A]$ and condition an easier experiment on $|ϵ|\le M$. Away from the prior endpoints, the Bayesian posterior noise has its original truncated law. Let $A\to \infty$ and then $M\to \infty$ in the resulting Bayes risk lower bound. No normal distribution or density for the noise is required.

A penalized function estimate balancing a residual loss against an integrated derivative penalty. For squared-error loss and the second derivative roughness penalty, the estimate is a cubic smoothing spline. Higher derivative orders give other spline degrees.

Training error is the average loss function evaluated on the observations used to fit the prediction rule. It is the empirical risk of the fitted rule on its own training data. With squared-error loss, it is the average squared regression residual, measuring fit to observed responses rather than performance on fresh responses. Flexible fitting usually makes it optimistically biased for prediction error, as quantified by prediction optimism.

For a reproducible polynomial regression example, take $x_i=-1+2i/11$, $i=0,\dots ,11$, and the known noiseless target $f(x)=1+x/2+2x^2$. Construct responses $y_i=f(x_i)+e_i$ with fixed disturbancesOrdinary least squares over polynomials of polynomial degree at most 1, 2, and 7 gives training errors $0.518937$, $0.015784$, and $0.009257$. The monotone improvement follows because these candidate families are nested: every smaller-degree fit remains available in the larger family. This argument concerns minimum training error, not generalization, and assumes the fitting mathematical optimization is solved. Degrees 2 and 7 fitted instead to the exact noiseless responses $f(x_i)$ have mathematically zero training error; the script obtains approximately $5.31\times 10^{-31}$ and $7.82\times 10^{-31}$ from rounding. Extra flexibility alone does not force overfitting.