Importance sampling

To estimate ${\mathbb{E}}_f[g(X)]$, sample $Y_1,\dots ,Y_m$ independently from a reference density $h$ whose support covers that of $gf$, and useThe summands have expectation $\int g(x)f(x)dx$, and the strong law of large numbers gives almost-sure convergence under integrability.

To estimate $\mathbb{P}(Z>z)$ for a standard normal variable using proposal $N(\mu ,1)$, the weighted indicator has second moment $e^{{\mu }^2}\bar{\Phi }(z+\mu )$. Differentiating it at a minimum gives the displayed equation. Since $\phi (a)/\bar{\Phi }(a)=E[Z\mid Z>a]$, the minimizing shift is half the target conditional mean above the shifted threshold. The resulting estimator remains unbiased with the exact likelihood-ratio weight.

For target $p=N(0,{\sigma }^2)$ and proposal $q=N(0,1)$, this is the importance weight. Assuming only $E_p{f}^2<\infty$, the weighted estimator has guaranteed finite variance for $0<\sigma \le 1$, since $E_q(w_{\sigma }^2{f}^2)=E_p(w_{\sigma }{f}^2)\le {\sigma }^{-1}{E}_p{f}^2$. For every $\sigma >1$, choosing $f^2=e^{a{x}^2}$ with $\max (0,{\sigma }^{-2}-1/2)<a<(2{\sigma }^2{)}^{-1}$ gives finite target second moment but infinite weighted second moment. With the additional assumption that $f$ is bounded, the sufficient range is instead $\sigma <\sqrt{2}$. The integrand hypothesis therefore matters.

Likelihood weighting uses observation likelihoods as importance weights. If parameters are drawn independently from their prior distribution $\rho$, weight a draw by $L(\theta )$ and estimate a posterior expectation of $h$ by ${\sum }_{j}L({\theta }_j)h({\theta }_j)/{\sum }_{j}L({\theta }_j)$. With positive finite evidence and integrable weighted quantities, the strong law of large numbers proves consistency of this self-normalized importance sampling ratio. Finite-sample bias and concentration of the weights remain possible. A different proposal requires the additional prior density-to-proposal density ratio.

Let $r$ be an unnormalized density, $Z=\int r$ with $0<Z<\infty$, and $\pi =r/Z$. Given independent draws ${\theta }_t\sim \pi$ and an evaluable normalized density $q$ supported where $r>0$, reciprocal importance sampling estimates $Z^{-1}$ by $T^{-1}{\sum }_{t}q({\theta }_t)/r({\theta }_t)$. Its expectation is $Z^{-1}\int q=Z^{-1}$ and its variance is $(T{Z}^2{)}^{-1}(\int q^2/\pi -1)$. Broad tails of $q$ relative to $\pi$ can make this infinite; reversing the importance sampling ratio reverses the dangerous tail mismatch. Taking the reciprocal of the estimate is generally biased for $Z$.

For $U$ uniform on $(0,1)$, $Y$ has density $q(y)=2y^{-2}$ on $y\ge 2$. The importance sampling weight for the standard Cauchy distribution tail is $W=f(Y)/q(Y)$, and its expectation is the target tail probability. Writing $A=\arctan (1/2)$ gives $\mathbb{E}W=A/\pi$ and $\mathbb{E}{W}^2=(A+2/5)/(4{\pi }^2)$ by integrating $y^2/(1+y^2{)}^2$. The weight varies only between $2/(5\pi )$ and $1/(2\pi )$, making its variance much smaller than that of a rare-event indicator. This is a comparison of sampling variances, not a deterministic ordering of every pair of empirical variance estimates.

For rejection indicators with conditional acceptance $f(X_i)/(Mg(X_i))$, the conditional expectation of $(M/n){\sum }_i{I}_i\varphi (X_i)$ given all proposals is the unnormalized importance estimator. The Rao-Blackwell theorem reduces variance relative to that fixed-denominator estimator. It gives no such automatic comparison with the accepted-sample mean, whose denominator is the random count.

If $f\le Mg$, the importance weight $w=f/g$ is bounded by $M$, and ${\mathbb{E}}_g[{\varphi }^2{w}^2]\le M{\mathbb{E}}_f{\varphi }^2$. A finite target second moment therefore guarantees finite variance and the iid central limit theorem for the unnormalized importance-sampling average. Its proposal-budget variance is ${\mathbb{E}}_g[{\varphi }^2{w}^2]-({\mathbb{E}}_f\varphi {)}^2$.

For $\mu =\int hf$ and $0<A=\int |h|f<\infty$, the proposal $q_{\ast }=|h|f/A$ minimizes the variance of the ordinary unbiased importance sampling estimator. The Cauchy-Schwarz inequality gives $\int h^2{f}^2/q\ge A^2$, so the minimum variance for $n$ independent samples is $(A^2-{\mu }^2)/n$. The proposal need only cover the nonzero integrand.

An importance sampling proposal $q$ must be positive wherever the target-weighted integrand can contribute. The usual sufficient condition is that the target probability is absolutely continuous with respect to $q$. Otherwise multiplying observed draws by $f/q$ cannot recover a missing part of the integral. A strict rejection envelope supplies this condition for the rejected-proposal distribution.

For target density $f$ and proposal density $h$, the importance weight is the density ratio $w(x)=f(x)/h(x)$ wherever the target is supported.

For an unnormalized target density, self-normalized importance sampling uses weights $w_i=f(Y_i)/h(Y_i)$ and estimates expectations by ${\sum }_i{w}_{i}g(Y_i)/{\sum }_i{w}_i$. Resampling a point $Y_i$ with probability proportional to $w_i$ produces the associated weighted empirical distribution.

For estimating $I=\int g(x)f(x)dx$ by the unbiased importance estimator $g(Y)f(Y)/q(Y)$, the proposal minimizing variance isThis follows from the Cauchy-Schwarz inequality. Its normalizing constant and dependence on the particular integrand often make it impractical.

For normalized importance weights $w_i$, the standard effective sample size is $K_{\mathrm{eff}}=1/{\sum }_i{w}_i^2$.