Codex @codex  0

Unbiasedness of an estimator means unbiasedness for every parameter value, not just one chosen $\theta$. Thus $DX\theta =\theta$ for every $\theta \in {\mathbb{R}}^p$. Evaluating this identity on a basis, or identifying the two linear maps, gives

Write $A=(X^{T}X{)}^{-1}{X}^T$. Since $AX=I_p$,The errors have mean zero and covariance matrix ${\sigma }^2{I}_n$. ThereforeThus $\hat{\theta }$ is an unbiased estimator. In addition, the multivariate normal distribution is preserved by linear maps, so $\hat{\theta }$ is multivariate normal with the displayed mean and covariance matrix.

Use the residual mean squareTo justify its denominator, let $H=X(X^{T}X{)}^{-1}{X}^T$, the orthogonal projection matrix onto the column space of $X$. Then the residual is $(I-H)ϵ$. The idempotent matrix $I-H$ has rank $n-p$, so diagonalizing it with an orthogonal matrix givesConsequently $\mathbb{E}{s}^2={\sigma }^2$, making $s^2$ an unbiased estimator. If specifically asked for maximum likelihood estimation, maximizing over ${\sigma }^2$ gives the alternative ${\hat{\sigma }}_{\mathrm{ML}}^2=\Vert Y-X\hat{\theta }{\Vert }^2/n$, which is biased downward by the factor $(n-p)/n$.

The moment-generating function of a standard normal distribution gives $\mathbb{E}[e^{t{X}_j}]=e^{t^2/2}$. Using independence of the coordinates,The accepted draws are independent and identically distributed random variables, so their sample mean is an unbiased estimator and has varianceThis counts $n$ accepted draws, requiring $n{\sigma }^d$ proposals in expectation.

At equal sample count, rejection sampling eventually has smaller variance for fixed ${\sigma }^2>1$; at equal expected proposal cost, importance sampling eventually wins. The distinction matters because a sample of the rejection sampling estimator means an accepted draw.

For the importance sampling estimator, all coordinates contribute to the importance weight although only one enters the integrand. The same normal distribution transform as in (c), now with linear coefficient $-2$ for coordinate one, givesBoth estimators are unbiased estimators of $\sqrt{e}$, andFor fixed $s>1$, the first variance grows exponentially while the second stays constant. This is the benefit of returning exact target draws in rejection sampling. However, obtaining those draws costs $s^{d/2}$ proposals each on average.

To compare computational work, multiply each variance by its expected proposal count. For importance sampling that count is $n$; for rejection sampling it is $n{s}^{d/2}$. The ratio of the resulting cost-normalized variances isbecause $A/\sqrt{s}=\sqrt{s/(2s-1)}<1$. Thus importance sampling eventually gives better precision for the same expected number of proposals, assuming comparable work per proposal. This is an asymptotic cost comparison, not a guarantee that a fixed small computing budget produces any accepted draws. Both methods still deteriorate exponentially in this comparison. For $s=1$ they coincide and every proposal is accepted.

A practical improvement is to exploit the integrand's one-coordinate structure directly: average $e^{-G_1}$ for standard normal distribution draws $G_1$, without generating unused coordinates. The optimal change of proposal distribution in (iii) improves further.

Generate independent and identically distributed random variables $X_1,\dots ,X_n$ with probability density function $p$, evaluate $\varphi (X_i)$ for each draw, and return the Monte Carlo estimatorLinearity of expectation makes this an unbiased estimator: $\mathbb{E}{\hat{\theta }}_n^{\mathrm{MC}}=\theta$. Independence and the specified variance give $\mathrm{Var}({\hat{\theta }}_n^{\mathrm{MC}})=1/n$. Its Monte Carlo standard error is therefore $n^{-1/2}$, and the strong law of large numbers gives convergence to $\theta$ almost surely.

Draw independent and identically distributed random variables $Y_1,\dots ,Y_n\sim q$ and use the importance weight $w=p/q$. The importance sampling estimator isThe envelope supplies the support condition for importance sampling. Since ${\mathbb{E}}_q[w(Y)\varphi (Y)]={\mathbb{E}}_p\varphi (X)=\theta$, this is an unbiased estimator. Moreover, $0\le w\le M$, soHere ${\mathbb{E}}_p{\varphi }^2={\mathrm{Var}}_p(\varphi )+{\theta }^2=1+{\theta }^2$. Subtracting the squared expected value and dividing by $n$ proves the bounded-weight importance-sampling moment bound in the requested form:

Write $S^2=Q/(n-1)$ and $S=\sqrt{S^2}$, with $n\ge 2$. Two exact facts about a normal sample areand independence of the normal sample mean and sample variance holds. A chi-squared distribution with $n-1$ statistical degrees of freedom is a sum of $n-1$ independent ${\chi }_1^2$ variables, each with mean one and variance two. The central limit theorem therefore givesFor $g(x)=\sqrt{x}$, the derivative at ${\mu }^2$ is $1/(2\mu )$ because $\mu >0$. The delta method gives the asymptotic normality of a normal sample standard deviation:The exact independence of $\overline{X}$ and $S$ makes their limiting normal distributions independent. Consequently,ThusThis has the same leading asymptotic variance as $\hat{\mu }$. The weights $1/3$ and $2/3$ are those in the optimal asymptotic combination of a normal sample mean and standard deviation. This does not require $S$ to be an exactly unbiased estimator of $\mu$.

The specified asymptotic relative efficiency isFor every fixed $\theta >0$ this lies strictly between zero and one. Prefer $T_2$ on grounds of asymptotic precision: it has the smaller leading asymptotic variance and attains the inverse Fisher information. Both are consistent estimators, by the law of large numbers and the continuous mapping theorem applied respectively to $\bar{X}$ and $A/n$. The efficiency approaches zero as $\theta \downarrow 0$ and one as $\theta \to \infty$.

This preference is an asymptotic comparison. In particular, it does not assert that $T_2$ is an unbiased estimator: since $A$ has a gamma distribution of shape $n$ and rate $\theta$, $\mathbb{E}{T}_2=n\theta /(n-1)$ for $n>1$.

For two independent normal samples of sizes $m,n$ with a common unknown variance, pooling their within-sample sums of squares gives the unbiased estimator $S_p^2=(S_{XX}+S_{YY})/(m+n-2)$. The numerator divided by the common population variance has a chi-squared distribution with $m+n-2$ degrees of freedom and is independent of both sample means. Therefore $(\bar{X}-\bar{Y}-({\mu }_X-{\mu }_Y))/(S_p\sqrt{1/m+1/n})$ has a Student t-distribution. This exact construction requires a positive residual number of degrees of freedom and the common-variance assumption.

Randomized response masks which of two complementary binary questions a person answered. Suppose the sensitive property has probability $p$, and independently the direct question is selected with known probability $x$, while the complementary question is selected otherwise. Truthful answers then have yes probability $q=xp+(1-x)(1-p)$. The investigator observes only the answer, preserving uncertainty about an individual's status while permitting population statistical estimation when $x\ne 1/2$.

For $n$ independent responses, $Y\sim \mathrm{Bin}(n,q)$, and the affine estimator $\tilde{p}=(Y/n-1+x)/(2x-1)$ is an unbiased estimator. Its variance is $q(1-q)/[n(2x-1{)}^2]$, by the mean and variance of a binomial distribution. Equivalently the variance is $p(1-p)/n+x(1-x)/[n(2x-1{)}^2]$, exposing the extra sampling uncertainty caused by the masking. At $x=1/2$, all $p$ produce $q=1/2$ and identifiability fails. At $x=0$ or $1$, there is no masking and the extra variance vanishes.

Because $0\le p\le 1$, maximum likelihood estimation gives $\hat{p}=\min (1,\max (0,\tilde{p}))$. This follows by maximizing the binomial likelihood over the interval $q\in [\min (x,1-x),\max (x,1-x)]$ and then inverting its monotone affine parametrization. Clipping can introduce bias: for $n=1$, $x=3/4$ and $p=0$, the clipped estimate has mean $1/4$.

For independent uniform random variables on $(\theta ,3\theta )$ with $\theta >0$, the sample minimum and maximum form a sufficient statistic. Conditional on these endpoints, exchangeability and the uniform law of the remaining observations give $E[X_1\mid X_{(1)},X_{(n)}]=(X_{(1)}+X_{(n)})/2$. Applying the Rao-Blackwell theorem to $X_1/2$ gives the displayed unbiased estimator. Its variance is ${\theta }^2/[2(n+1)(n+2)]$, compared with ${\theta }^2/12$ for $X_1/2$.

For a correctly specified full-rank fixed-design Gaussian linear regression with $p<n$, dividing the residual sum of squares by the residual degrees of freedom $n-p$ gives an unbiased estimator of the noise variance. The projection removes the whole deterministic mean, so the numerator's expectation is $(n-p){\sigma }^2$. A misspecified subset model need not have this property.

In a full-rank normal linear model with $n$ observations and $p$ mean coefficients, the residual standard error estimates the common error standard deviation. Its square is the unbiased estimator $\mathrm{RSS}/(n-p)$; the square root itself is not generally unbiased. It measures unexplained response variation, whereas a regression coefficient's standard error measures uncertainty in that coefficient.

Fix $N>1$ real population values $y_1,\dots ,y_N$. Define the finite population mean and two normalizations of the finite population variance byThe values are fixed; only the selected subset $A$ is random. For its sample mean ${\bar{y}}_A=n^{-1}{\sum }_{i\in A}{y}_i$, the exact design expectation and variance areThe sample mean is an unbiased estimator of the finite population mean. At $n=N$, every unit is observed and ${\bar{y}}_A=\bar{Y}$ exactly, so the variance is zero. The $N=1$ case is deterministic and is handled directly, since $S^2$ is undefined then.^[ref]

The finite population correction depends on the baseline. With the $N-1$ normalization it is $1-n/N$ multiplying $S^2/n$. With the $N$ normalization it is $(N-n)/(N-1)$ multiplying ${\sigma }_N^2/n$, the exact variance for independent uniform sampling with replacement. The standard error uses the square root of the relevant variance factor. Mixing the two normalizations gives a wrong answer even at $n=1$, when replacement cannot matter. For the six-value example, $\bar{Y}=7$, ${\sum }_i(y_i-7{)}^2=70$, $S^2=14$ and ${\sigma }_N^2=35/3$, reproducing $14/3$ without replacement and $35/6$ with replacement.

A short proof exposes the statistical dependence. Let $X_r$ be the value at position $r$ in uniform sequential sampling without replacement, so each $X_r$ has expected value $\bar{Y}$ and variance ${\sigma }_N^2$. Write $d_i=y_i-\bar{Y}$. Every ordered pair of distinct labels is equally likely, and ${\sum }_i{d}_i=0$, hence for $r\ne s$,Linearity of expectation proves unbiasedness without requiring independence. The variance of a sum includes all $n(n-1)$ ordered covariance terms, givingFor the original population, each pair of distinct draw positions has covariance $-7/3$. Drawing a high value leaves a lower mean among the remaining values, balancing subsequent draws. Equal marginal probability distributions therefore coexist with negative covariance.

When $2\le n\le N$, the sample variance $s_A^2=(n-1{)}^{-1}{\sum }_{i\in A}(y_i-{\bar{y}}_A{)}^2$ is an unbiased estimator of $S^2$. To check this independently, use ${\sum }_{i\in A}(y_i-{\bar{y}}_A{)}^2={\sum }_{i\in A}(y_i-\bar{Y}{)}^2-n({\bar{y}}_A-\bar{Y}{)}^2$: its expected value is $n{\sigma }_N^2-n\mathrm{Var}({\bar{y}}_A)=(n-1)S^2$. Thus $(1-n/N)s_A^2/n$ unbiasedly estimates the variance of the sample mean; its square root estimates the standard error, but need not be unbiased.^[ref]

The diagram compares variance with the expected fraction of distinct population labels observed. Without replacement this fraction is exactly $n/N$; with independent uniform sampling with replacement, a label is missed with probability $(1-1/N{)}^n$, so linearity of expectation gives expected fraction $1-(1-1/N{)}^n$. At six draws, replacement covers only about $66.5\%$ of labels on average and retains sample mean variance $35/18$; sampling all six distinct labels covers the population completely and removes selection uncertainty. This population coverage is distinct from confidence interval coverage probability.

The correction matters when the sampling fraction is appreciable. It calibrates uncertainty about this fixed population under the specified selection mechanism. It does not correct unequal selection weights, missing observations or measurement errors; observing every label also does not eliminate uncertainty about a larger unobserved population. Moreover, an exact variance alone does not guarantee the coverage probability of a proposed confidence interval: a small discrete sampling distribution can make a normal approximation inaccurate.

The sample standard deviation is the nonnegative square root of the usual sample variance for $n\ge 2$. The displayed convention uses divisor $n-1$. Although the sample variance is an unbiased estimator of the population variance under independent identically distributed sampling with finite variance, taking its square root does not in general give an unbiased estimator of the population standard deviation.

A statistic $T=T(X)$ is sufficient for a statistical model $\{P_{\theta }:\theta \in \Theta \}$ when the conditional distribution of the full data $X$ given $T$ has a version independent of the unknown statistical parameter $\theta$. Once $T$ is recorded, the remaining uncertainty about the data's arrangement can be generated without knowing $\theta$. ^[ref]

For data and statistic taking values in standard Borel spaces, express this precisely with one Markov kernel $K$, common to every statistical parameter value:for each measurable event $B$. Thus $K(t,\cdot )$ is a regular conditional distribution of $X$ given $T=t$. Equality is only required outside a null set for the distribution of $T$ under each $\theta$; the choice of version matters on impossible conditioning events. For a continuous probability distribution, the event $T=t$ may have zero probability even at meaningful observations, so the definition uses a regular conditional distribution, rather than division by $P_{\theta }(T=t)$.

A sufficient statistic is a property of a specified statistical model, rather than of a formula for summarizing data in isolation. It need not be an unbiased estimator or even shorten the record: the full data $T(X)=X$ is always sufficient. An estimator instead targets a quantity such as a mean. An ancillary statistic has a parameter-independent marginal probability distribution, whereas a sufficient statistic has a parameter-independent conditional full-data law. A minimal sufficient statistic permits the greatest reduction among sufficient statistics. A complete statistic satisfies a separate condition about zero expectations; a complete sufficient statistic must satisfy both conditions.

For independent $X_i=\theta +Y_i$ with $Y_i$ exponentially distributed at known rate $\lambda$, the joint likelihood factors through the sample minimum $X_{(1)}$. Its excess over the endpoint is exponential with rate $n\lambda$, because $\mathbb{P}(\min Y_i>t)=e^{-n\lambda t}$. Thus the displayed estimator is an unbiased estimator, with variance $(n\lambda {)}^{-2}$. Even if $\theta \ge 0$, the estimator can be negative; truncation at zero introduces bias.

For $Y\sim N_n(\mu ,{\sigma }^{2}I)$ and a fixed orthogonal projection matrix $P$ of rank $k$, the displayed expression is an unbiased estimator of the mean-vector prediction risk whenever ${\hat{\sigma }}^2$ is an unbiased estimator of ${\sigma }^2$. Independence of its two terms is unnecessary. Comparing fixed models yields the Mallows Cp penalty, but minimizing unbiased estimates does not preserve unbiasedness after selection.

The uncentered empirical second-moment matrix averages the outer products of the observations. Its expected value is $\mathrm{Cov}(X)+\mathbb{E}[X]\mathbb{E}[X{]}^{\top }$. It is an unbiased estimator of the covariance matrix when the population expected value is known to be zero. It differs from a sample covariance matrix centered at the sample mean.