Likelihood weighting Created 2026-10-25 Updated 2026-10-31
Likelihood weighting uses observation likelihoods as importance weights. If parameters are drawn independently from their prior distribution , weight a draw by and estimate a posterior expectation of by . With positive finite evidence and integrable weighted quantities, the strong law of large numbers proves consistency of this self-normalized importance sampling ratio. Finite-sample bias and concentration of the weights remain possible. A different proposal requires the additional prior density-to-proposal density ratio.
Write and . If the constants needed to evaluate are unavailable, use the self-normalized importance sampling estimator
The unknown common factor cancels. Under full target probability support and integrability, the numerator and denominator sample means converge to and , respectively, whose ratio is . The same construction applies if only one normalization is unknown by taking the other probability density function already normalized. Sampling from itself is assumed available, even when its normalization is not explicitly known.
This ratio estimator is generally biased at finite , unlike the ordinary normalized-weight estimator. Also, its variance is not the formula in the next part: that formula concerns the ordinary unbiased estimator. Full probability support for is needed here because the denominator estimates the entire target normalizing integral, not just the probability support of .
Write and , where are known nonnegative kernels. If both constants are known, the ordinary weight is . If their ratio is unknown, use self-normalized importance sampling:
Indeed, under the normalized proposal law,
The strong law of large numbers makes their empirical ratio converge to , provided the numerator is absolutely integrable and . Multiplicative constants cancel, whether the unknown one belongs to the target, proposal, or both. Unlike the estimator in part (i), this ratio is generally biased at finite sample size. Exact sampling from is still presumed; knowing only an unnormalized proposal does not itself supply a sampler.
Let , and write for the standard normal distribution density. For importance sampling, choose the proposal density , namely the uniform distribution on the target interval. Its importance sampling weight is
For iid uniform draws ,
is an unbiased estimator of the target mean, since its expectation integrates . If one chooses not to evaluate , the self-normalized importance sampling alternative has statistical consistency, but generally is not an unbiased estimator.
Alternatively, draw independent random variables with the standard normal distribution, let and , and use
Conditionally on , the retained draws are iid from the truncated normal distribution, so their average is an unbiased estimator with variance , where is its variance. If the ratio is undefined; one can continue drawing until a prescribed number of retained observations is obtained.
Here importance sampling uses every draw inside the interval, whereas only of the normal proposals are retained. The advantage can also be quantified. Integration using gives
For uniform importance sampling,
For the retained-draw ratio with a fixed total proposal budget, its asymptotic variance coefficient is . Thus this importance choice has substantially smaller error for the same proposal count; it also avoids wasting about two thirds of the standard normal distribution draws. This comparison is specific to this well-matched importance sampling proposal; arbitrary proposal choices do not share this guarantee.
Conditionally on , the cumulative distribution function of is the weighted empirical distribution
For fixed , the strong law of large numbers gives, almost surely,
while
The ratio therefore converges almost surely to . This is precisely the claimed convergence in distribution of the self-normalized importance sampling resample.
The simulation catalog is a sample from the joint prior. Self-normalized importance sampling therefore estimates the posterior mean by
The likelihood weights update each simulated system according to how well its satellite resembles the measured one.
I would first collapse the duration noises as in part (c), then use self-normalized importance sampling on the two shared parameters. Draw independent , give each the importance weight , and return
This is likelihood weighting: the observations affect the weights rather than forcing repeated simulation of twenty latent inequalities. Integrating out both the constraints' auxiliary randomness and the future Boolean query is Rao-Blackwellization; at fixed parameters it removes randomness without changing the relevant conditional expectations.
The weights and weighted queries lie in , and the evidence is positive. The strong law of large numbers therefore makes the numerator and denominator averages converge to their corresponding integrals, proving almost sure convergence of their ratio to part (d). The finite- ratio is generally biased; the assertion concerns consistency rather than a finite-sample unbiased estimator. Use logarithms of importance weights to avoid numerical underflow:
def weighted_answer(n, rng):
    if n <= 0:
        raise ValueError("n must be positive")
    samples = []
    for _ in range(n):
        mu = rng.uniform(0.0, 60.0)
        sigma = rng.uniform(1.0, 20.0)
        samples.append((journey_loglik(mu, sigma),
                        math.exp(5.0 * normal_logcdf((mu - 30.0) / sigma))))
    offset = max(logw for logw, h in samples)
    if not math.isfinite(offset):
        raise ArithmeticError("all weights underflowed")
    scaled = [(math.exp(logw - offset), h) for logw, h in samples]
    den = math.fsum(w for w, h in scaled)
    answer = math.fsum(w * h for w, h in scaled) / den
    ess = den * den / math.fsum(w * w for w, h in scaled)
    return answer, ess
The returned effective sample size of importance sampling is a concentration diagnostic, not a proof of accuracy. Repeat runs and inspect the weights. If the prior is much broader than the posterior, improve the proposal with the corresponding importance sampling density correction, or use the Metropolis–Hastings algorithm targeting . Because this particular collapsed problem is only two-dimensional, deterministic numerical integration is also an excellent independent accuracy check; it was used for the figure and numerical illustration. Straight rejection of the original traces is correct but wastes most proposals here.
For unknown normalization constants, use self-normalized importance sampling with :
Under the same support condition, and . Moreover , by Cauchy-Schwarz inequality and the supplied . The strong law of large numbers applies to numerator and denominator sample averages. Their ratio tends almost surely, hence in probability, to . If a finite sample happens to give zero denominator, it can be assigned an arbitrary value; this does not affect the eventual almost sure limit. The ratio estimator is generally biased at finite .