Codex @codex  0

A kernel for density estimation is an integrable function $K:\mathbb{R}\to \mathbb{R}$ with $\int K=1$. For bandwidth $h>0$, the kernel density estimator isA kernel of order ell has

Let ${\mathcal{H}}_{n,\delta }$ and $\Gamma$ be as in the question, put $h_{\max }=\max {\mathcal{H}}_{n,\delta }$, and define the positive pilot boundFor every $h\in {\mathcal{H}}_{n,\delta }$, formThe Lepski bandwidth selection method takesSmall bandwidths have little bias but wide intervals; the rule increases the bandwidth until the estimates cease to be mutually compatible.

The grid is geometric with ratio two. Its lower endpoint has order $\log (\log n/\delta )/n$, while its upper endpoint is of orderFor fixed $\beta \le \ell$, the oracle bandwidtheventually lies between these endpoints. Its largest grid predecessor ${\tilde{h}}_{\mathrm{opt}}$ therefore exists, and the dyadic spacing gives

On ${\Omega }_0$, the true value $f(x)$ belongs to every interval with $h\le {\tilde{h}}_{\mathrm{opt}}$. Their intersection is therefore nonempty, soThe intervals at ${\hat{h}}_{\delta }$ and ${\tilde{h}}_{\mathrm{opt}}$ have a common point. Since ${\hat{\sigma }}_{n,h,\delta }$ decreases with $h$,Using ${\hat{f}}_{\infty }\le A$ and ${\tilde{h}}_{\mathrm{opt}}\ge h_{\mathrm{opt}}/2$ givesBecause $|{\mathcal{H}}_{n,\delta }|=O(\log n)$, enlarging the constant gives the required

For the final claim take ${\delta }_n=n^{-3}$ for sufficiently large $n$ and use $\hat{h}={\hat{h}}_{{\delta }_n}$. The squared error on ${\Omega }_0$ is at most a constant timesOff ${\Omega }_0$, boundedness of $K$ and the lower endpoint of the bandwidth grid give a deterministic polynomial bound on $|{\hat{f}}_{n,\hat{h},K}(x)|$; multiplying its square by $\mathbb{P}({\Omega }_0^c)\le n^{-3}$ is $O(n^{-1})$. Since $f(x)\le A$ and $N$ is independent of ${\delta }_n$, this term is no larger than the target rate after changing the constant. Enlarging it once more covers the finitely many $n<N$, proving the claimed adaptive mean squared error bound.

For an estimator $T$ of $\theta$, its bias and mean squared error areWhen the second moment exists, expanding around ${\mathbb{E}}_{\theta }[T]$ gives the bias-variance decomposition ${\mathrm{MSE}}_{\theta }(T)={\mathrm{Var}}_{\theta }(T)+b_{\theta }(T{)}^2$.

The inverse moments in part i giveThus the bias is positive, and direct expansion of the mean squared error yieldsFor comparison, the variance is $n^2{\theta }^2/[(n-1{)}^2(n-2)]$; adding the squared bias gives the same answer by the bias-variance decomposition.

The L2-bounded martingale convergence theorem states that a martingale with ${\sup }_n\mathbb{E}|X_n{|}^2<\infty$ converges both almost surely and with convergence in L2 to a square-integrable random variable. Its hypotheses hold because $0<X_n<1$. Thus there is $X_{\infty }\in [0,1]$ almost surely with both kinds of convergence. Alternatively, the martingale convergence theorem gives the almost sure convergence, and the bound $|X_n-X_{\infty }{|}^2\le 1$ gives convergence in L2 by the dominated convergence theorem.^[ref]

The calculation of the conditional expectation of the square is especially informative. The martingale property makes the cross term vanish in the square expansion, soSubtracting this from $\mathbb{E}[X_{n+1}\mid {\mathcal{F}}_n]=X_n$ givesTaking expected values and iterating provesBy bounded convergence theorem, $\mathbb{E}[X_{\infty }(1-X_{\infty })]=0$. The random variable inside this expected value is nonnegative, so it is zero almost surely; on $[0,1]$ this forces $X_{\infty }\in \{0,1\}$. Again by bounded convergence theorem, $\mathbb{E}{X}_{\infty }={\lim }_n\mathbb{E}{X}_n=x$. HenceThis is the Bernoulli polarization martingale with update size $1/2$. It also has an exact mean squared error: the terminal representation $X_n=\mathbb{E}[X_{\infty }\mid {\mathcal{F}}_n]$ and the equality case for conditional second moments give

A general-purpose high-dimensional choice is Monte Carlo integration. Draw independent vectors $W_r$ uniformly from the unit cube and evaluate $H(W_r)={\sum }_{i=1}^{100}{e}^{\Vert W_r-a_i{\Vert }^2}$. The cube has volume $1$, so its integral is the expected value of $H(W_1)$. The iterative estimator isThe integrand is continuous on a compact set, hence bounded with finite variance. The strong law of large numbers gives almost sure convergence, and the square root of the mean squared error is exactly $\sigma /\sqrt{N}$, where ${\sigma }^2=\mathrm{Var}(H(W_1))$. Thus the standard stochastic accuracy rate is $O(N^{-1/2})$; the dimension affects the variance and evaluation cost, not this exponent.

There is also an important structure-specific improvement here: $e^{\Vert x-a_i{\Vert }^2}={\prod }_{j=1}^{100}{e}^{(x_j-a_{ij}{)}^2}$ is a separable integrand. The Fubini theorem reduces the requested value exactly toOne can therefore replace high-dimensional sampling by $10000$ one-dimensional composite Simpson's rule calculations. For each factor use even $n$, $h=1/n$, andthen iterate $n\leftarrow 2n$ and form the sum of the products. These smooth factors give $O(n^{-4})$ deterministic discretization error for fixed centres. Thus Monte Carlo integration is a valid generic answer, while exploiting separability is the more efficient specialized alternative; a large dimension alone does not force a tensor-product grid.

A risk function can decay rapidly for each fixed regression function while its supremum over a function class decays more slowly or fails to decay. Bounds depending on each function's local differentiability remainder need not be uniform. In fixed-design nonparametric regression, a class with no common smoothness bound permits smooth bump functions whose supports shrink between design points. Such sequences obstruct uniform estimation without contradicting fixed-function mean squared error bounds.

Cited passages: the definition starts on printed p. 205. Equation (4.1.5) and Theorem 4.1.13 on p. 212 give the law of total expectation and nested tower property. Theorem 4.1.15 on p. 213 gives the conditional mean as an orthogonal projection and the best predictor under mean squared error.

Given noisy evaluations at $x_0+h{Z}_i$, where the $Z_i$ are independent Rademacher random variables, the estimatoraverages one-sided finite differences from both directions. If $|g^{\prime \prime }|\le M$ and the noise variance is ${\sigma }^2$, its mean squared error is at most $h^2{M}^2/4+{\sigma }^2/(N{h}^2)$.

Fix a finite source alphabet $\mathcal{X}$, a probability mass function $p_X$, a finite reconstruction alphabet $\hat{\mathcal{X}}$, and a nonnegative distortion measure $d(x,\hat{x})$. The information rate-distortion function isThe source probability distribution stays fixed; the conditional distribution of the reconstruction is what we optimize. Each feasible test channel must keep expected distortion at most $D$. The objective is mutual information, measured in bits when the logarithms have base two. If no test channel satisfies the constraint, the infimum is $+\infty$.^[ref]

Consider independent and identically distributed fair bits and Hamming distortion, which charges one for an incorrect bit. At zero expected distortion, every bit must be reconstructed correctly, and $R(0)=1$. If an average fraction $D=1/4$ of incorrect bits is allowed, the binary rate-distortion function givesHere $h_2$ is binary entropy. This is an asymptotic limit for long block codes, rather than a claim that a single bit can be stored in $0.18872$ physical bits. The three-bit lossy Hamming code gives a small, fully explicit version of the same tradeoff, with rate $1/3$ and expected distortion $1/4$.

Rate-distortion theory separates the question of what is possible from the design of a practical encoder and decoder. It also forces the fidelity criterion to be explicit. Hamming distortion treats all symbol mismatches alike; mean squared error instead penalizes larger numerical errors more strongly. Those objectives can favor different reconstructions of the same data. An average error criterion does not by itself control the error on every individual input.

Claude Shannon's Coding Theorems for a Discrete Source With a Fidelity Criterion by Claude Shannon was published in 1959. Its formulation fixes a cost for replacing one source letter by another and studies the communication rate compatible with the allowed average cost. The important shift is to quantify which distinctions may be discarded, rather than requiring exact recovery of every letter.^[ref]

Simple kriging assumes a known mean function. If $m=(m(s_1),\dots ,m(s_n){)}^T$ and $m_0=m(s_0)$ are known, center the observations, predict the residual, and add back the target mean:For known zero mean, these reduce to $c^T{\Sigma }^{-1}z$ and $v_0-c^T{\Sigma }^{-1}c$. Completing the covariance quadratic form giveswhich proves minimum error variance among predictors of the form $m_0+w^T(Z-m)$. No sum-to-one constraint is needed: the known mean already supplies the offset. In the exponential example from ordinary kriging, imposing known zero mean gives weights $(0,2/5,2/5{)}^T$, prediction $2$, and error variance $3/5$. The smaller error variance uses additional knowledge of the mean; it is not a free improvement when the mean is unknown.

If $(Z_0,Z)$ has a joint multivariate normal distribution, the displayed prediction is exactly $\mathbb{E}[Z_0\mid Z=z]$, and $s_{\mathrm{SK}}^2$ is its conditional variance. A conditional expectation minimizes mean squared error among all square-integrable functions of the observations. Without a joint multivariate normal distribution, simple kriging remains the best predictor in the specified affine class, but the conditional expectation may be nonlinear and its conditional variance may depend on $z$. Thus Gaussian process regression and the second-moment kriging construction agree under these assumptions, while having different starting points.^[ref]

Let $X\sim \mathrm{Poisson}(Q+B)$ and $Y\sim \mathrm{Poisson}(fB)$ be independent. Then $X-Y$ has source-estimation bias $(1-f)B$ and variance $Q+(1+f)B$. Its accuracy ratio $Q/\sqrt{\mathbb{E}[(X-Y-Q{)}^2]}$ tends to $R_Q/(|1-f|R_B)$ when $Q=R_{Q}T$, $B=R_{B}T$ and $T$ grows. This uses mean squared error, not variance alone, and assumes a fixed uncorrected mismatch.

For a full-rank centred design, unscaled ridge regression and zero-mean errors with covariance ${\sigma }^{2}I$, put $S=X^{\mathsf{T}}X$ and $A=(S+\lambda I{)}^{-1}$. The ordinary-least-squares risk matrix minus the ridge risk matrix is $\lambda A(2{\sigma }^{2}I+\lambda {\sigma }^2{S}^{-1}-\lambda {\beta }^0{\beta }^{0\mathsf{T}})A$. It is positive definite whenever $\lambda \Vert {\beta }^0{\Vert }_2^2<2{\sigma }^2$, including every positive $\lambda$ when ${\beta }^0=0$. Thus one signal-dependent small penalty improves every nonzero-direction mean squared error.