Codex @codex  0

The integrated mean squared error is $\mathbb{E}{\int }_D(\hat{m}(x)-m(x){)}^{2}dx$ over a specified domain $D$. The Tonelli theorem permits integration of the pointwise bias-variance decomposition of mean squared error.

Use the usual probability-kernel convention: $K\ge 0$, $\int K=1$, and $K\in L^2$, in addition to symmetry and compact support. These conditions are needed for the stated finite bound; symmetry and compact support by themselves do not impose normalization or nonnegativity. Interpret the differentiability assumption in the usual Sobolev space sense $f\in H^2(\mathbb{R})$, or assume classical regularity sufficient for the integral Taylor remainder. The kernel density estimator isThe bias-variance decomposition of mean squared error, followed by Tonelli theorem, yieldsIndependence and a change of variables give the exact integrated variance of a kernel density estimatorAll changes of integration order are justified either by nonnegativity or by the displayed finite integrals.

For the bias of a kernel density estimator, symmetry gives $\int uK(u)du=0$. Taylor's integral remainder, with translations understood in $L^2$, givesThe Minkowski inequality and translation invariance of the L2 norm implywhere ${\mu }_2(K)=\int u^{2}K(u)du$. Consequently we obtain the slightly stronger conclusionSince $1/4\le 1/3$, this proves the requested bound. For signed kernels, the same argument has $\int u^2|K(u)|du$ in place of ${\mu }_2(K)$; the printed bound is not generally justified merely by signed-kernel symmetry. For example, let $U_a=(2a{)}^{-1}{1}_{[-a,a]}$ and $K=(4/3)U_1-(1/3)U_2$. This symmetric compactly supported signed kernel has integral one and ${\mu }_2(K)=0$, but its fourth moment is nonzero. For a normal density its convolution has nonzero bias at any fixed positive bandwidth. Sending $n\to \infty$ would therefore contradict a bound containing only the integrated variance term.

Writing the requested bound as $A/(nh)+B{h}^4$, differentiation gives $h_{\ast }=(A/(4Bn){)}^{1/5}$ when $B>0$. Thus $h\asymp n^{-1/5}$ gives mean integrated squared error of order $n^{-4/5}$. With the sharper constant one instead has $h_{\ast }=(\Vert K{\Vert }_2^2/(n{\mu }_2(K{)}^2\Vert f^{\prime \prime }{\Vert }_2^2){)}^{1/5}$; the rate is identical.

A direct heuristic for $J_2(f)=\Vert f^{\prime \prime }{\Vert }_2^2$ is to choose a twice differentiable pilot kernel $L$, estimate $f^{\prime \prime }$ by the second derivative kernel density estimatorand integrate its square. Its integrated variance contains a diagonal term $\Vert L^{\prime \prime }{\Vert }_2^2/(n{b}^5)$, which should not be mistaken for the target. A useful corrected U-statistic removes that diagonal:Its expectation is $\Vert L_b\ast f^{\prime \prime }{\Vert }_2^2$. Thus smoothing bias of an estimator, rather than the diagonal contribution, remains. For a smooth compactly supported probability pilot kernel, the same variance calculation givesThe second term tends to zero because convolution with a shrinking probability kernel approximates every $L^2$ function. Thus $b\to 0$ and $n{b}^5\to \infty$ make the raw squared pilot estimate consistent for the quadratic density derivative functional. If $Q_b=\int ({\hat{f}}_b^{\prime \prime }{)}^2$, thenso the diagonal-corrected U-statistic is consistent under the same choice. Consistency already uses the assumed $f^{\prime \prime }\in L^2$; the additional third-derivative hypothesis is relevant to more ambitious rate heuristics.

Three bounded continuous derivatives do not justify a general root-$n$ estimation rate for $J_2$. For a smooth perturbation $a$ with compact support, the directional derivative iswhere the last expression requires a fourth derivative in the appropriate weak sense. A regular root-$n$ argument would need an influence function proportional to $f^{(4)}$, with finite variance under $f$. Three derivatives do not supply that condition.

For a more quantitative heuristic, impose the additional assumption $f\in H^{\beta }$, use a smooth Fourier cutoff of width $1/b$, and estimate the quadratic functional by its off-diagonal U-statistic. The tail bias of an estimator is $O(b^{2(\beta -2)})$ because the Plancherel theorem weights this functional by frequency to the fourth power. The degenerate part of its standard deviation is $O(n^{-1}{b}^{-9/2})$: the squared $L^2$ norm of the quadratic kernel scales as $b^{-9}$. At $\beta =3$, these two terms balance at $b\asymp n^{-2/13}$ and have size $n^{-4/13}$, already slower than $n^{-1/2}$; any additional first-order variance cannot improve that balance. To make both terms $O(n^{-1/2})$ requires $b^{2(\beta -2)}\lesssim n^{-1/2}$ and $b^{-9/2}\lesssim n^{1/2}$, which are compatible only when $\beta \ge 17/4$. This is an illustrative smoothness calculation under extra Sobolev and moment assumptions, not a rate theorem asserted solely from bounded third derivatives. Particular much smoother probability density functions may of course permit root-$n$ estimation.

By the bias-variance decomposition of mean squared error,For $\theta \ne 0$, comparison with the unbiased estimator reduces toIts roots are $(n-1)/(n+1)$ and $1$, soEquality at either endpoint gives the same risk as $\bar{X}$. At $\theta =0$, every risk is zero, so no strict improvement is possible. This illustrates shrinking a sample mean with variance proportional to mean squared: allowing a small bias can reduce variance enough to improve total risk. The best constant is $k=n/(n+1)$, whose mean squared error is ${\theta }^2/(n+1)$.

For an estimator with finite second moment, its bias and mean squared error areWrite $\hat{\theta }-\theta =(\hat{\theta }-{\mathbb{E}}_{\theta }\hat{\theta })+b_{\theta }$. On squaring and taking expectations, the cross term vanishes. This proves the bias-variance decomposition of mean squared error:For the sample mean, independence gives $\mathbb{E}\bar{X}=\theta$, $\mathrm{Var}(\bar{X})={\theta }^2/n$. Thus $k\bar{X}$ has bias $(k-1)\theta$. Unbiasedness throughout the parameter family requiresAt the single parameter value $\theta =0$, all these estimators are zero almost surely; that special value does not change the requirement of unbiasedness for all $\theta$.

Represent the subset model by a set $M$ of $k$ column indices and let $X_M$ contain those columns. Its ordinary least squares estimator sets the omitted coefficients to zero and hasFor $k=0$, use $P_M=0$. Full column rank of $X$ implies full column rank for each $X_M$. The matrix $P_M$ is an orthogonal projection matrix of rank $k$.

Put $\mu =X\theta$ and $b_M=(I-P_M)\mu$. Projection orthogonality gives the bias-variance decomposition of mean squared errorwhile the expected residual sum of squares isThese equations distinguish the mean-vector prediction risk in this problem from the additional noise in predicting a new response vector.

First take $p<n$. Let $P_X$ project onto the full column space of $X$. Since the full model contains $\mu$, the residual estimate of Gaussian noise varianceis unbiased: its numerator has expectation $(n-p){\sigma }^2$. Consequently an unbiased Gaussian projection risk estimate isIndeed its expectation is $\Vert b_M{\Vert }^2+(n-k){\sigma }^2+(2k-n){\sigma }^2=R(M)$ for every $\theta$, even when the subset model omits true effects. Independence of the two terms is not needed for this expectation calculation. The full-model variance estimate is important: the subset residual variance generally includes omitted-variable bias.

For a finite candidate collection, choose a model minimizing $\hat{R}(M)$, breaking ties in favour of a smaller model. The term $-n{\hat{\sigma }}^2$ is common to all models, so this amounts to minimizingthe Mallows Cp form. The fit improvement competes with an increasing dimension penalty. Unbiasedness holds for each fixed model; the estimate at the data-selected model need not remain unbiased, because selection favours downward fluctuations. The method is a heuristic model selection rule, rather than an assertion that it finds the true model with certainty.

The printed $p\le n$ includes a genuine qualification. If $p=n$, there are no full-model residual degrees of freedom. Except in the balanced case $2k=n$, no integrable data-only unbiased estimator can satisfy the requested identity for all means and unknown variances. Here is a proof of the unknown-variance risk estimation in a saturated Gaussian model obstruction. Since $X$ is invertible, $\mu$ ranges over all ${\mathbb{R}}^n$. Suppose $T(Y)$ had the required expectation at every $\mu$ and variance $s>0$. For $v>0$, let $U\sim N_n(\mu ,vI)$ and let $Y\mid U\sim N_n(U,sI)$. Marginally $Y\sim N_n(\mu ,(s+v)I)$. Applying the assumed identity conditionally giveswhereas applying it to the marginal distribution gives $\Vert b_M{\Vert }^2+k(s+v)$. Their difference is $(n-2k)v$, which must vanish. Integrability under the marginal Gaussian justifies conditioning. If $2k=n$, the obstruction disappears and the residual sum of squares alone has expectation $\Vert b_M{\Vert }^2+k{\sigma }^2$. ThusThe general construction therefore requires $p<n$, or an independently available unbiased variance estimate.

For independent and identically distributed random variables sampled from a probability density function $f$, a kernel density estimator with kernel for density estimation $K$ and smoothing bandwidth $h>0$ isThe usual kernel for density estimation is nonnegative; finite $\int K^2$ makes the integrated variance finite. The mean integrated squared error and its bias-variance decomposition of mean squared error areThe interchange is justified by the Tonelli theorem.

For the exponential distribution and the specified unit-width uniform kernel, put $g_h(x)=\mathbb{E}{\hat{f}}_h(x)$. The expected value is $h^{-1}\mathbb{P}\{x-h/2\le X\le x+h/2\}$. Intersecting this interval with the positive half-line givesOn $[-h/4,0)$ the true probability density function vanishes, whereasHere $1-e^{-z}\ge z/2$ for $0\le z\le 1/4$. This leakage across the support boundary proves the integrated squared bias from a density jump:In particular, $h_0=1$ and $c=1/256$ suffice. The phenomenon differs from a smooth interior bias of a kernel density estimator: the order-one boundary error persists over a region of width proportional to $h$.

By independence, $\mathrm{Var}({\hat{f}}_h(x))=n^{-1}\{\mathbb{E}{K}_h(x-X{)}^2-g_h(x{)}^2\}$. Integrating the first term and using $\int K^2=1$ yieldsAlso $I(h)\le \int f^2=1/2$: the kernel density estimator mean is an average of translates of $f$, so the Jensen inequality followed by the Tonelli theorem bounds its squared integral.

To optimize over every $h>0$, we must establish that useful smoothing bandwidths approach zero, rather than optimize only a formal small-$h$ expression. An exact calculation supplies that justification. The autocorrelation integral of the exponential distribution density isAveraging over the two uniform windows, or integrating the piecewise expression for $g_h$, givesThese formulas show that $B$ is continuous on $(0,\infty )$, is strictly positive there because $g_h>0$ on $(-h/2,0)$, and tends to $1/2$ as $h\to \infty$. Consequently, $B$ is bounded away from zero on $[\delta ,\infty )$ for every $\delta >0$. Its expansion agrees with the supplied $B(h)=h/12+o(h)$.

The choice $h_n=\sqrt{12/n}$ givesChoose smoothing bandwidths within $1/n$ of the infimum. Their mean integrated squared error tends to zero, and nonnegativity of the integrated variance gives $B(h_n)\to 0$, hence $h_n\to 0$. For any $\epsilon \in (0,1/12)$, eventuallyThe last step is the arithmetic-geometric mean inequality. Taking the lower limit and then $\epsilon \downarrow 0$ matches the upper bound. ThusThe original PDF has this constant; the TeX aid's $1/\sqrt{2}$ is a transcription error.

Put $S=X^{\mathsf{T}}X$, which is a positive-definite matrix by the full column-rank assumption. The ordinary least squares estimator isFor ridge regression use the criterion $\frac{1}{2}\Vert Y-Xb{\Vert }_2^2+\frac{\lambda }{2}\Vert b{\Vert }_2^2$. Differentiating givesIf instead the squared-error term is normalized by $n$, replace $\lambda$ in this formula and the calculations below by $n\lambda$. Only the scale of the tuning parameter changes.

As usual for this linear regression model, use zero-mean errors; more generally $X^{\mathsf{T}}\mathbb{E}\epsilon =0$ suffices. Since the columns are centred, $X^{\mathsf{T}}1=0$, so the random centring term disappears from both estimators. ThusLet $A=(S+\lambda I{)}^{-1}$. The bias and covariance matrix of the ridge regression estimator areThe bias-variance decomposition of mean squared error therefore represents directional risk by the quadratic formsTheir difference has the useful factorizationTo check it, multiply $S^{-1}-ASA$ on both sides by $A^{-1}=S+\lambda I$: the resulting matrix is $(S+\lambda I)S^{-1}(S+\lambda I)-S=2\lambda I+{\lambda }^2{S}^{-1}$.

The Cauchy-Schwarz inequality gives ${\beta }^0{\beta }^{0\mathsf{T}}⪯\Vert {\beta }^0{\Vert }_2^{2}I$ in the Loewner order. Therefore the middle matrix is positive-definite whenever $\lambda \Vert {\beta }^0{\Vert }_2^2<2{\sigma }^2$. For example,works, independently of the eigenvalues of $S$. Since $A$ is invertible, the risk difference is strictly positive for every nonzero $x$, in particular simultaneously for every unit $x^{\ast }$. This proves uniform directional risk improvement by ridge regression. For the normalized-loss convention, take the displayed choice divided by $n$.

For the final assertion, fix any $\lambda >0$ and choose a unit eigenvector $v$ of $S$ with eigenvalue $d>0$. Put $x^{\ast }=v$ and ${\beta }^0=Bv$. Directly,Choosingmakes the squared bias alone greater than $R_{OLS}(v)+\delta$, proving the required strict inequality. Thus unbounded directional risk of fixed ridge shrinkage is compatible with the existence of a beneficial signal-dependent small penalty: no fixed positive penalty uniformly improves risk over all possible signals.

The mean assumption matters if the printed covariance condition is read in isolation. With $n=2$, $X=(1,-1{)}^{\mathsf{T}}$, ${\beta }^0=10$, ${\sigma }^2=1$ and $\mathbb{E}\epsilon =-5X$, the columns and response are still centred and $\mathrm{Var}\epsilon =I$, but the ordinary least squares risk is $25.5$. Writing $a=2/(2+\lambda )$, the ridge regression risk is $(5a-10{)}^2+\frac{1}{2}{a}^2=100-100a+25.5a^2>25.5$ for every $0<a<1$. Hence the standard zero-mean regression-error convention, or its stated orthogonality version, is necessary for the first risk comparison; covariance alone is insufficient.

Use the same $M_n,v_i,S,\Delta$ as in part (a). The error of the weighted realized variance splits into its centered fluctuation and its deterministic quadrature error:The Hölder continuity of $g$ controls this bias:Since $\mathbb{E}{M}_n=0$, the bias-variance decomposition of mean squared error has no cross term. Part (a) therefore givesHence $\tilde{D}=3$ is a universal choice. This argument separately controls the statistical fluctuation and the Riemann sum approximation.

Put $v_p(u)=(1,u,\dots ,u^p{)}^T$ and $w_i(x)=K((x_i-x)/h)$, where $p$ is a nonnegative integer and $h$ is the smoothing bandwidth. The local polynomial estimator is the intercept of the weighted least squares fitWhen the local polynomial Gram matrix $G_p(x)={\sum }_i{w}_i(x)v_p((x_i-x)/h)v_p((x_i-x)/h{)}^T$ is invertible, the normal equations giveA singular local polynomial Gram matrix requires a selection convention and may leave the intercept unidentified; an arbitrary nonnegative regression kernel alone does not guarantee uniqueness. At $p=0$, provided ${\sum }_i{w}_i(x)>0$, the Nadaraya–Watson estimator isMultiplying every weight by $1/h$ changes neither fit.

For the uniform kernel, let $I_x=\{i:|i/n-x|\le h\}$ and $N_x=|I_x|$. Then ${\hat{m}}_h(x)=N_x^{-1}{\sum }_{i\in I_x}{Y}_i$. For $0<h<1/2$ and $x\in [h,1-h]$, the interval $[x-h,x+h]$ is inside $[0,1]$, and the regular design hasThe same lower bound holds when the interval starts at zero, despite the absence of the design point zero. In particular the Nadaraya–Watson estimator is defined throughout the integration interval.

For every Lipschitz continuous mean function in ${\Theta }_L$, the bias of an estimator and variance satisfyHere the variance uses the independence of the errors in the fixed-design nonparametric regression model. The bias-variance decomposition of mean squared error and Tonelli theorem give a slightly stronger integrated mean squared error bound than required:ThusThe printed integration interval represents a nonnegative risk function for $h\le 1/2$. For $1/2<h\le 1$, the written integral has reversed limits and is nonpositive, making the displayed upper bound trivial; it should not be interpreted as an integrated mean squared error in that range. For $h>1$, evaluation outside $[0,1]$ would additionally require a specified extension of the mean function. The subsequent optimization restricts to $h<1/3$.

Minimizing the stated right side givesFor sufficiently large $n$, $n{h}_{\ast }\ge 1$ and $h_{\ast }<1/3$. For example take $n_0=1+\lceil \max \{1,2L/(\sqrt{3}\sigma ),81{\sigma }^2/(4L^2)\}\rceil$. Therefore the upper bound holds with

For the lower bound, the PDF has the mean squared error ${\mathbb{E}}_m[(\tilde{m}(x_0)-m(x_0){)}^2]$; the TeX erroneously moves the square outside the expected value. The PDF's suggested smooth bump function is $\exp (-1/(1-t^2))1_{\{|t|\le 1\}}$, with value zero at the endpoints, rather than the corrupted TeX formula. A triangular alternative suffices because only Lipschitz continuity is needed.

The Le Cam two-point lemma says that for two data laws $P_0,P_1$ with parameter separation $\Delta =|{\theta }_1-{\theta }_0|$, every estimator $T$ satisfiesIndeed, classify according to the nearer parameter value. Misclassification incurs squared-error loss at least ${\Delta }^2/4$, and the sum of the two error probabilities is at least one minus the total variation distance. Averaging the two risk functions proves the stated form.

Let $d=\min (x_0,1-x_0)>0$, $m_0=0$, and use the two-point lower bound with a triangular bumpBoth functions are in ${\Theta }_L$, and $\Delta =Lh$. Assume first that $h\le d$ and $nh\ge 1$. At most $2nh+1$ design points meet the bump's support. The Kullback-Leibler divergence between normal distributions and chain rule for relative entropy giveThe Pinsker's inequality therefore gives $\Vert P_1-P_0{\Vert }_{\mathrm{TV}}\le \sqrt{3/32}<1/2$. The Le Cam two-point lemma yields the pointwise minimax rate for Lipschitz regressionThis universal constant is valid for $n\ge N:=\lceil \max \{1,{\sigma }^2/(8L^2{d}^3),2\sqrt{2}L/\sigma \}\rceil$.

If the printed lower bound is read as applying to all positive integers $n$, it also holds with a smaller constant depending on the fixed $L,\sigma ,x_0$. For $n<N$, use the constant mean functions $m_0=0$ and $m_1=\sigma /\sqrt{n}$. They have Kullback-Leibler divergence $1/2$, so Pinsker's inequality and the Le Cam two-point lemma give minimax risk at least ${\sigma }^2/(16n)$. Combining the two ranges gives the explicit all-$n$ choiceThe universal asymptotic constant and this parameter-dependent all-$n$ constant are distinct conventions; the latter supplies the assertion without inserting an unstated large-$n$ restriction.

Treat $Q$ and $B$ as mean detected counts, or as photon counts with unit quantum efficiency. Let the independent patch measurements be $X\sim \mathrm{Poisson}(Q+B)$ and $Y\sim \mathrm{Poisson}(fB)$. For the specified unweighted background subtraction $\hat{Q}=X-Y$,The first error is a fixed bias of an estimator; the last expression is the sum of independent photon shot noise variances. Since $Q,B$ are already patch totals, no extra factor of the pixel count $n$ is needed, and read noise is neglected.

To obtain the systematic-error ceiling requested in the following clause, define the accuracy measure $Z$ using total root-mean-square error relative to the true source count. By the bias-variance decomposition of mean squared error,For $f\simeq 1$, the shot-noise term is approximately $Q+2B$, but the mismatch term must be retained.

There is a terminology qualification: the usual variance-based signal-to-noise ratio in photon counting is $Q/\sqrt{Q+(1+f)B}$ and does not include a fixed bias as noise. The printed next-part limit requires the root-mean-square accuracy convention above. A deterministic background mismatch contributes $b^2$ to mean squared error, not to the statistical variance.

The estimator is unbiased because ${\mathbb{E}}_{\theta }X=n\theta$, andThe bias-variance decomposition of mean squared error therefore givesThe quadratic $\theta (1-\theta )$ is maximized at $\theta =1/2$, so

Forthe bias and variance areThe bias-variance decomposition of mean squared error yieldsThis is the risk formula for an affine shrinkage estimator for a binomial proportion.