Codex @codex  0

First, the absolute values in the printed limit are an error: the correct long-run variance of a stationary process is the signed sum of autocovariances. Directly,soEach coefficient tends to $1$ and has absolute value at most $1$. Absolute summability and the dominated convergence theorem therefore giveThe sum of absolute values is an upper bound, not the general limit. For a concrete counterexample to the printed assertion, take $X_t={\eta }_t-{\eta }_{t-1}$ with unit-variance iid noise. Then $\gamma (0)=2$, $\gamma (\pm 1)=-1$, and other covariances vanish. The absolute sum is $4$, but ${\overline{X}}_T=({\eta }_T-{\eta }_0)/T$, so $T\mathrm{Var}({\overline{X}}_T)=2/T\to 0$.

For correlated data the central limit theorem can still hold under suitable strong mixing of a stationary process and moment conditions, but the limiting variance is the long-run variance of a stationary process, rather than the one-observation variance. When it is positive,Positive serial dependence usually increases the standard error, while negative dependence can reduce it. The approximate effective sample size of a stationary sample is $T{\gamma }_X(0)/{\sum }_h{\gamma }_X(h)$ when that denominator is positive. A long-memory time series can require a different normalization or a different limit law. Absolute covariance summability alone does not prove a CLT: if $X_t=V{\eta }_t$ with a common independent random scale $V$ taking values $1$ and $2$ with equal probabilities and iid ${\eta }_t$ with the standard normal distribution, the off-diagonal covariances are zero, yet $\sqrt{T}{\overline{X}}_T$ has the nonnormal scale-mixture law $VN(0,1)$. The example is not an ergodic stationary process. If the long-run variance of a stationary process is zero, the usual nondegenerate square-root-$T$ CLT is unavailable.

For the causal AR(1), $|\varphi |<1$ andThe conditional distribution follows because the current innovation is independent of the past.

For a fixed known $\varphi$, condition on the observed $X_1$ and use the $n=T-1$ transitions $t=2,\dots ,T$. Their conditional maximum likelihood criterion is, up to constants,Differentiating in $\mu$ yieldsSince $X_t-\varphi X_{t-1}=(1-\varphi )\mu +{\epsilon }_t$,Thus it is an unbiased estimator, even conditionally on $X_1$, andIt has statistical consistency with mean-square convergence, and the iid-noise strong law of large numbers also gives almost-sure statistical consistency. Its conditional distribution is exactly normal with the displayed mean and variance. If $X_0$ is also observed and all $T$ transitions are used, replace $T-1$ by $T$.

The fixed-$\varphi$ qualification is necessary for the exact finite-sample claims. If $\varphi$ is jointly estimated, conditional likelihood function is linear regression with an intercept: writing ${\bar{X}}_{-}=n^{-1}{\sum }_{t=2}^T{X}_{t-1}$ and ${\bar{X}}_{+}=n^{-1}{\sum }_{t=2}^T{X}_t$, the unconstrained estimators areThis ratio is not generally an unbiased estimator and does not have the preceding finite-sample variance. Under the usual stationary regression conditions it has statistical consistency; its asymptotic variance is ${\sigma }_{\epsilon }^2/(1-\varphi {)}^2$. Indeed, it differs from ${\bar{X}}_{+}$ by $\hat{\varphi }(X_T-X_1)/(n(1-\hat{\varphi }))$, an asymptotically negligible endpoint term. Profiling an unknown innovation variance does not change the fixed-$\varphi$ estimate of $\mu$.

Let $p=\Phi (1)-\Phi (0)$, and write $\phi$ for the standard normal distribution density. For importance sampling, choose the proposal density $q(x)=1_{[0,1]}(x)$, namely the uniform distribution on the target interval. Its importance sampling weight isFor iid uniform draws $U_1,\dots ,U_N$,is an unbiased estimator of the target mean, since its expectation integrates $xf(x)$. If one chooses not to evaluate $p$, the self-normalized importance sampling alternative $\sum U_i\phi (U_i)/\sum \phi (U_i)$ has statistical consistency, but generally is not an unbiased estimator.

Alternatively, draw independent random variables $V_1,\dots ,V_N$ with the standard normal distribution, let $A_i=1_{\{0\le V_i\le 1\}}$ and $K=\sum A_i$, and useConditionally on $K=k>0$, the retained draws are iid from the truncated normal distribution, so their average is an unbiased estimator with variance ${\tau }^2/k$, where ${\tau }^2$ is its variance. If $K=0$ the ratio is undefined; one can continue drawing until a prescribed number of retained observations is obtained.

Here importance sampling uses every draw inside the interval, whereas only $p\approx 0.3413$ of the normal proposals are retained. The advantage can also be quantified. Integration using ${\phi }^{\prime }(x)=-x\phi (x)$ givesFor uniform importance sampling,For the retained-draw ratio with a fixed total proposal budget, its asymptotic variance coefficient is ${\tau }^2/p\approx 0.233347$. Thus this importance choice has substantially smaller error for the same proposal count; it also avoids wasting about two thirds of the standard normal distribution draws. This comparison is specific to this well-matched importance sampling proposal; arbitrary proposal choices do not share this guarantee.

The support condition for importance sampling is missing from the printed assertion. Assume $M>1$, $E_f|h|<\infty$, and $Mg(z)>f(z)$ for $f$-almost every $z$. For a proposal $Z\sim g$, rejection has conditional probability $1-f(Z)/(Mg(Z))$. Its unconditional probability is $(M-1)/M$, so the conditional density of a rejected draw isThus the rejected-draw weight is precisely $f(z)/r(z)$, andConditionally on any positive rejection count $n$ among $N$ independent proposals, the rejected locations have this iid conditional density. Henceis an unbiased estimator conditionally on $n$, and therefore also conditionally on the event $0<n<N$. This is importance sampling from the rejected-proposal distribution.

The nonstrict assumption $f\le Mg$ alone does not suffice: the rejected distribution may completely miss part of the target support. For example, let $g$ be uniform on $[0,1]$, $f(z)=21_{[0,1/2]}(z)$, $M=2$, and $h=1$. All rejected proposals lie in $(1/2,1]$, where $f=0$, so the displayed estimator is always $0$, whereas $E_{f}h=1$. This remains a counterexample after conditioning on $0<n<N$. A strict envelope, such as choosing $M$ larger than the essential supremum of $f/g$, repairs the missing support condition. Near-equality can still give high-variance weights, even when it remains an unbiased estimator.

The printed identity needs an expectation on its right-hand side. For fixed $r\ge 1$, condition on $X_2,\dots ,X_r$ and use independence of $X_1$:The expression inside this expectation is generally random and cannot equal the unconditional probability by itself.

For each independent Monte Carlo method replicate, draw $R_j$ from its given distribution. If $R_j=r\ge 1$, draw $r-1$ independent $F$-variables, form $W_j={\sum }_{i=2}^r{X}_{j,i}$ and set $H_j=F(x-W_j)$. For $r=1$ the empty sum is $0$. If $R$ may equal zero, use $H_j=1_{\{x\ge 0\}}$ when $R_j=0$, since $S_0=0$. The law of total expectation givesSince $0\le H_j\le 1$, the estimator has statistical consistency by the strong law of large numbers. It is the conditional Monte Carlo estimator obtained by Rao-Blackwellization of the direct indicator $1_{\{S_R\le x\}}$, and the law of total variance gives $\mathrm{Var}(H_j)\le l(1-l)$.

This version assumes that the cumulative distribution function $F$ can be evaluated. An easy sampler alone does not automatically supply an easy cumulative distribution function evaluation. If only sampling is available, use the direct indicator estimator, or replace each conditional cumulative distribution function by the average of several independent indicators $1_{\{X_{j,1,k}\le x-W_j\}}$; the nested version remains an unbiased estimator. With $L$ such inner draws its variance per outer replicate is $\mathrm{Var}(H_j)+L^{-1}E[H_j(1-H_j)]$.

Because the scalar signal and the noise vector are independent random variables with normal distributions, their sum is a Gaussian random vector. Its expected value is zero, and the covariance matrix isThus this is a rank-one covariance spike. Each summand $X_i{X}_i^{\top }$ has expected value ${\Sigma }_S$, so $\mathbb{E}\hat{\Sigma }={\Sigma }_S$. Here the empirical matrix is an uncentered empirical second-moment matrix; because the population expected value is known to be zero, it is an unbiased estimator of the covariance matrix. The TeX's apparent factorial denominator is a transcription error.

A sufficient statistic has a conditional distribution of the full data given its value that is independent of the unknown parameter. A minimal sufficient statistic is sufficient and is a function of every other sufficient statistic, up to null sets. For a dominated family with common positive support, the likelihood-ratio criterion for minimal sufficiency states that $T$ is minimal sufficient if $T(x)=T(y)$ holds exactly when $p_{\theta }(x)/p_{\theta }(y)$ is independent of $\theta$.

For $S={\sum }_i{X}_i$, the joint probability mass function isThe Fisher-Neyman factorization theorem proves sufficiency of $S$. For two samples, the likelihood ratio is $(1-1/\theta {)}^{S(x)-S(y)}$, independent of $\theta >1$ exactly when their sums agree. Thus $S$ is minimal sufficient. Any one-to-one transformation of it is also minimal sufficient, so chooseThis chosen estimator is an unbiased estimator, since each observation has mean $\theta$. If instead the statistic is reported as $S$, its expectation is $n\theta$, and as an estimator of $\theta$ it has bias $(n-1)\theta$ (zero only when $n=1$). Being a minimal sufficient statistic is unchanged by this rescaling; being an unbiased estimator is not.

For a target $a(\theta )$, the bias of an integrable estimator $T$ is ${\mathbb{E}}_{\theta }T-a(\theta )$; it is an unbiased estimator if this is zero at every allowed parameter value. Here $Y$ has a zero-truncated Poisson distribution, $\theta >0$, and $p=1-e^{-\theta }\in (0,1)$. Write an estimator based only on $Y$ as $t(Y)$, assuming a finite expectation for every $\theta >0$. Unbiasedness requiresAbsolute integrability at every positive parameter ensures that the power series on the left converges absolutely on every complex disc. Uniqueness of coefficients in a power series therefore gives $t(y)=0$ for odd $y$ and $t(y)=2$ for positive even $y$. Conversely these values give the displayed identity, sois the unique unbiased estimator of this form. The uniqueness claim concerns nonrandomized functions of $Y$; allowing external randomness would permit addition of independent mean-zero noise.

For this parity estimator for a zero-truncated Poisson count, usefulness depends on the loss, but under squared-error loss it performs poorly despite being an unbiased estimator. Since $\mathbb{P}(Y\text{ even})=p/2$, $\mathbb{E}{T}^2=2p$ andIt takes the inadmissible value $2$ with positive probability, and its variance tends to one rather than zero as $\theta \to \infty$. Clipping to the parameter interval gives $T_c=\min (T,1)=1_{\{Y\text{ even}\}}$. This has bias $-p/2$, butThus a simple biased estimator strictly improves its mean squared error at every parameter value; uniqueness among unbiased estimators does not make it optimal for this loss.

The Rao-Blackwell theorem applies to an estimator $Y$ with finite second moment and a sufficient statistic $T$. The conditional estimator $Y^{\ast }={\mathbb{E}}_{\theta }[Y\mid T]$ can be chosen as a function of the observed $T$ that does not depend on the unknown parameter: this uses the parameter-independent conditional data distribution in the definition of sufficient statistic. The tower property of conditional expectation gives ${\mathbb{E}}_{\theta }{Y}^{\ast }={\mathbb{E}}_{\theta }Y$, so their bias agrees. The law of total variance givesFor any target $a(\theta )$ it follows thatEquality holds exactly when $Y=Y^{\ast }$ almost surely under that parameter value. This proves variance reduction and reduction of mean squared error, whether or not the original estimator is an unbiased estimator. More generally, conditional Jensen inequality proves the corresponding inequality for any convex loss in the estimate, whenever the expectations exist.

Since $\mathbb{P}(X_1=0)=e^{-\lambda }$, the indicator $Y$ is an unbiased estimator of $\theta$. The joint probability mass function isThe Fisher-Neyman factorization theorem therefore makes $T={\sum }_i{X}_i$ a sufficient statistic. More explicitly, conditioning on $T=t$ gives a multinomial distribution with $t$ trials and cell probabilities $1/n$, so $X_1\mid T=t$ has the binomial distribution $\mathrm{Bin}(t,1/n)$. Consequently conditioning by the Rao-Blackwell theorem givesFor $n=1$, interpret the value at $T=0$ as $1$ and at $T>0$ as $0$; thus the same expression uses $0^0=1$ in this probability convention. For $\lambda =0$ all observations and $T$ are zero almost surely. For $n>1$ and $\lambda >0$, the reduction in variance is strict: given $T=1$, the event $X_1=0$ still has a nondegenerate conditional probability. Directly, using the probability generating function of $T\sim \mathrm{Poisson}(n\lambda )$, the new variance is $e^{-2\lambda }(e^{\lambda /n}-1)$.

For a weakly stationary process, every observation has the same finite expectation $\mu$. Linearity of expectation, without any assumption of independence, givesThe variance is a sum of all pairwise covariances:For a real-valued weakly stationary process, the autocovariance is even. There are $n$ pairs at lag zero and $n-h$ pairs in each direction at positive lag $h$. ThereforePositive autocovariances inflate the uncertainty relative to the independent-observation formula; negative terms may reduce it. An unbiased estimator need not have variance tending to zero: a process consisting of the same random variable at every time is weakly stationary, but averaging cannot reduce its variance.

Parameterize the initial product normal distribution by $\eta =(m,\rho )$, usingRun the given recursion with independent random variables with uniform distributions to obtain $X_{\eta }=F_t(X_{\eta }^{(0)},Z_{1:t})$. Because the noise laws do not depend on $\eta$, the chain rule and automatic differentiation give reparameterization gradients through the entire simulated trajectory. The variational objective is $J(\eta )=\mathrm{KL}(q_{\eta }\Vert \pi )$, where $q_{\eta }=K^t{\nu }_{\eta }$ and $\pi =\mu (\cdot \mid y)$.

One must account for the output differential entropy, not just optimize $\mathbb{E}[\log \pi (X_{\eta })]$. Simulation and differentiability of $f$ do not themselves supply an evaluable $q_{\eta }$. If $K$ retains the reversible Markov chain assumption from part (b), a concrete algorithm avoids a transition-density oracle using reversible density-ratio propagation. Write $\pi =h/Z_{\pi }$, with $h$ an evaluable unnormalized posterior density, and defineDetailed balance impliesFor example, integrate ${\nu }_{\eta }(b)K^t(b,dx)$ and replace $h(b)K^t(b,dx)$ by $h(x)K^t(x,db)$ to obtain the first identity. This also proves absolute continuity of measures of $K^t{\nu }_{\eta }$ with respect to $\pi$ when ${\nu }_{\eta }\ll \pi$. An everywhere finite differentiable log-posterior has positive density, so an ordinary product normal distribution has this domination.

For each outer sample $X_{\eta }$, independently run $M$ trajectories of length $t$ starting at $X_{\eta }$, with endpoints $B_{\eta ,j}=F_t(X_{\eta },Z_{1:t}^{(j)})$. These are reverse trajectories because of detailed balance; the same recursion simulates them. Estimate $A_{\eta }(X_{\eta })$ byAverage $\log \hat{A}$ over an outer batch. Compute it stably by a logarithmic sum of exponentials. Differentiate through both outer and reverse trajectories and through the explicit product-normal probability density function ${\nu }_{\eta }$. Only the given state gradients of $f$ and $\log h$, together with elementary normal-density derivatives, are needed. Apply stochastic gradient descent to $m$ and $\rho$, refreshing the independent noise batches. The logarithmic scale parameters keep all variances positive.

This is nested Monte Carlo optimization of the exact objective in the limit of adequately increasing inner and outer sample sizes. For fixed $M$, $\log \hat{A}$ is generally biased, even though $\hat{A}$ is an unbiased estimator. Finite logarithmic moments, suitable uniform integrability, and justified differentiation under the integral sign are needed for consistency and gradient interchange. Under those conditions increasing $M$ and controlling the optimization error yields a justified numerical approximation; nonconvexity prevents a generic guarantee of a global optimum.

There is a real assumption issue: part (c) redefines $K$ without explicitly restating reversibility. The algorithm above uses the reversible interpretation inherited from part (b). For an arbitrary differentiable recursion, one additionally needs access to output densities or a simulatable reverse kernel and regularity of the objective. Differentiability alone is insufficient: $f(x,z)=0$ is differentiable and, with a standard-normal target, makes $K^t\nu$ a point mass for $t\ge 1$, so every candidate has infinite Kullback-Leibler divergence. Thus a universal finite-objective algorithm cannot be justified solely from the literal differentiability assumptions. An invertible recursion with computable density transformations offers a different sufficient route, but invertibility is not stated here.

Put $r=1-\alpha \in (0,1)$. The variable $Y=X_1-\theta$ has density $r{y}^{r-1}$ on $(0,1)$, so $\mathbb{E}{Y}^j=r/(r+j)$. ThusThe unbiased estimator isTaking expectations verifies that its bias is zero.

The support restricts $\theta$ to $\theta >0$, $\theta \le X_{(1)}$ and $\theta \ge X_{(n)}-1$. Within its feasible interior, the likelihood function is proportional to ${\prod }_i(X_i-\theta {)}^{-\alpha }$ and is strictly increasing in $\theta$, diverging as $\theta \uparrow X_{(1)}$. Under the printed infinite-endpoint-density, or extended-likelihood convention, the maximum-likelihood estimator isA density is defined only almost everywhere. If its value at the singular endpoint is reassigned a finite value, a finite maximum need not exist; the sample-minimum answer uses the stated endpoint/supremum convention.

Independence and $\mathbb{P}(Y\le t)=t^r$ on $[0,1]$ giveThe error is positive almost surely, so the estimator is not an unbiased estimator. More explicitly its bias is

An unbiased estimator $T$ of a parameter $\theta$ satisfies ${\mathbb{E}}_{\theta }[T]=\theta$ for every permissible value of $\theta$. Since the sample mean is linear,so $\hat{\mu }$ is unbiased.

Use the identityThe two expectations on the right are $n{\sigma }^2$ and $n\mathrm{Var}(\overline{X})={\sigma }^2$, respectively. HenceThusThe unbiased sample variance instead divides the same sum of squares by $n-1$.