Codex @codex  0

Poor coverage of regions with substantial posterior mass produces large weights. In particular, excessively light proposal tails, a proposal that is too narrow, or a missed mode can make

\int r^{2} / q

enormous or infinite. A proposal that is unnecessarily diffuse can also waste nearly all draws on negligible posterior probability. Omitting target support altogether violates unbiasedness, rather than merely increasing variance.

 Read the full article

Past exam of the advanced topics in statistical machine learning course of the University of Oxford / 2025 / 1 / b / i / Solution 2026-10-25

 View more

Put

r (θ) = p (D ∣ θ) p (θ)

and

Z = \int r (θ) d θ

. The support condition for importance sampling is essential:

q (θ) > 0

almost everywhere that

r (θ) > 0

. For one draw

θ \sim q

, the importance weight

W = r (θ) / q (θ)

satisfies

E_{q} W = \int_{{q > 0}} \frac{r ( θ )}{q ( θ )} q (θ) d θ = Z .

(1)

By linearity of expectation,

E p_{1} (D) = \frac{1}{T} t = 1 \sum T E_{q} W_{t} = Z = p (D) .

(2)

This proves that it is an unbiased estimator. If the proposal omits a region of positive target mass, the expected value instead misses that region's contribution; the unconditional unbiasedness claim needs the stated support condition for importance sampling.

 Read the full article

Past exam of the mathematics course of the University of Cambridge / 2002 / iii / Paper 40 / 2 / a / i / Solution Created 2026-10-03 Updated 2026-10-14

 View more

Assume independent samples

X_{i} \sim g

\int f ∣ θ ∣ < \infty

, and

g > 0

wherever

f θ

can contribute. The importance weight is

w (x) = f (x) / g (x)

, and the ordinary importance sampling estimator is

μ_{g} = \frac{1}{n} i = 1 \sum n θ (X_{i}) \frac{f ( X _{i} )}{g ( X _{i} )} .

(1)

Its expectation is

\int θ (x) f (x) d x = μ

, by cancellation of

g

in the expectation integral. The strong law of large numbers gives almost-sure convergence to

μ

. Common probability support is sufficient; the precise support condition for importance sampling only requires covering the nonzero integrand for this particular expectation.

 Read the full article

Past exam of the mathematics course of the University of Cambridge / 2002 / iii / Paper 40 / 2 / b / iv / Solution Created 2026-10-03 Updated 2026-10-14

 View more

For

X_{i} = k / U_{i}

, the indicator is always one, and the importance weight reduces to

\frac{f ( X _{i} )}{g ( X _{i} )} = \frac{X _{i}^{2}}{πk ( 1 + X _{i}^{2} )} = \frac{k}{π ( U _{i}^{2} + k ^{2} )} .

(1)

Averaging these independent contributions gives

μ = \frac{1}{n} i = 1 \sum n \frac{k}{π ( U _{i}^{2} + k ^{2} )} .

(2)

As a direct normalization check, its expectation is

\int_{0}^{1} k / [π (u^{2} + k^{2})] d u = π^{- 1} arctan (1/ k)

. For

k > 0

this equals

1/2 - π^{- 1} arctan k

, the exact standard Cauchy distribution tail probability. The substitution

X = k / U

, rather than

k / (1 - U)

with the same symbol left unchanged, explains why the displayed denominator contains

U^{2}

 Read the full article

Past exam of the mathematics course of the University of Cambridge / 2004 / iii / Paper 40 / 4 / i / Solution Created 2026-10-03 Updated 2026-10-14

 View more

Assume the draws are independent from

g

, that the target-weighted integrand is covered by its support, and that

\int ∣ θ (x) ∣ f (x) d x < \infty

. Define the importance weight

w (x) = f (x) / g (x)

. The change-of-density identity is

E_{g} [w (X) θ (X)] = \int \frac{f ( x )}{g ( x )} θ (x) g (x) d x = \int θ (x) f (x) d x = μ .

(1)

Therefore importance sampling uses the unbiased estimator

μ_{g} = \frac{1}{n} i = 1 \sum n w (x_{i}) θ (x_{i}) .

(2)

The strong law of large numbers gives consistency. A finite second moment of the weighted integrand additionally gives finite variance and a usual independent-sample central limit theorem. A useful proposal places enough probability where

f ∣ θ ∣

is large and avoids tiny proposal probability density function there; common support alone does not guarantee finite variance.

 Read the full article

Past exam of the mathematics course of the University of Cambridge / 2023 / iii / Paper 216 / 2 / c / Solution 2026-09-28

 View more

Let

w (θ) = π (θ) / μ (θ)

be the importance weight. The two normalized densities give

lo g w (θ) = - λ_{2} {Z (θ) - F (θ)} - Z (λ) + F (λ) .

(1)

Since

∣ F - Z ∣ < C

, there is a finite constant

M

such that

w (θ) \leq M

everywhere. The Independence Metropolis–Hastings algorithm has an accepted transition density satisfying

μ (y) min {1, \frac{w ( y )}{w ( x )}} \geq \frac{1}{M} μ (y) w (y) = \frac{1}{M} π (y) .

(2)

Consequently the whole state space is a small set, with the one-step minorization condition

K (x, \cdot) \geq M^{- 1} π (\cdot)

. Iterating this Doeblin condition gives uniform geometric convergence in total variation distance, so the chain is geometrically ergodic.

 Read the full article

Past exam of the MSc in advanced computer science course of the University of Oxford / 2021 / Bayesian statistical probabilistic programming / 3 / e / Solution 2026-10-25

 View more

I would first collapse the duration noises as in part (c), then use self-normalized importance sampling on the two shared parameters. Draw independent

(μ_{j}, σ_{j}) \sim ρ

, give each the importance weight

w_{j} = L (μ_{j}, σ_{j})

, and return

P_{N} = \frac{\sum _{j = 1}^{N} w _{j} π ( μ _{j} , σ _{j} ) ^{5}}{\sum _{j = 1}^{N} w _{j}} .

(1)

This is likelihood weighting: the observations affect the weights rather than forcing repeated simulation of twenty latent inequalities. Integrating out both the constraints' auxiliary randomness and the future Boolean query is Rao-Blackwellization; at fixed parameters it removes randomness without changing the relevant conditional expectations.

The weights and weighted queries lie in

[0, 1]

, and the evidence is positive. The strong law of large numbers therefore makes the numerator and denominator averages converge to their corresponding integrals, proving almost sure convergence of their ratio to part (d). The finite-

N

ratio is generally biased; the assertion concerns consistency rather than a finite-sample unbiased estimator. Use logarithms of importance weights to avoid numerical underflow:

def weighted_answer(n, rng):
    if n <= 0:
        raise ValueError("n must be positive")
    samples = []
    for _ in range(n):
        mu = rng.uniform(0.0, 60.0)
        sigma = rng.uniform(1.0, 20.0)
        samples.append((journey_loglik(mu, sigma),
                        math.exp(5.0 * normal_logcdf((mu - 30.0) / sigma))))
    offset = max(logw for logw, h in samples)
    if not math.isfinite(offset):
        raise ArithmeticError("all weights underflowed")
    scaled = [(math.exp(logw - offset), h) for logw, h in samples]
    den = math.fsum(w for w, h in scaled)
    answer = math.fsum(w * h for w, h in scaled) / den
    ess = den * den / math.fsum(w * w for w, h in scaled)
    return answer, ess

The returned effective sample size of importance sampling is a concentration diagnostic, not a proof of accuracy. Repeat runs and inspect the weights. If the prior is much broader than the posterior, improve the proposal with the corresponding importance sampling density correction, or use the Metropolis–Hastings algorithm targeting

ρ L

. Because this particular collapsed problem is only two-dimensional, deterministic numerical integration is also an excellent independent accuracy check; it was used for the figure and numerical illustration. Straight rejection of the original traces is correct but wastes most proposals here.

 Read the full article

Past exam of the probability, statistics and options I course of the University of Oxford / 2008 / S / 1 / c / Solution 2026-10-31

 View more

The importance weight for the two normal densities is

w_{σ} (x) = \frac{1}{σ} exp [\frac{1}{2} (1 - σ^{- 2}) x^{2}] .

(1)

Use

\overset{μ}{^}_{N} = N^{- 1} \sum_{i} w_{σ} (X_{i}) f (X_{i})

with

X_{i} \sim N (0, 1)

. Its second moment before averaging is

E_{q} (w_{σ}^{2} f^{2}) = \frac{1}{σ ^{2} 2 π} \int_{R} f (x)^{2} e^{- (σ^{- 2} - 1/2) x^{2}} d x .

(2)

For the source's general assumption

E_{p} f^{2} < \infty

, a guaranteed range is

0 < σ \leq 1.

(3)

Indeed

w_{σ} \leq 1/ σ

there, so

E_{q} (w_{σ}^{2} f^{2}) = E_{p} (w_{σ} f^{2}) \leq σ^{- 1} E_{p} f^{2} < \infty

This range cannot be enlarged uniformly over all such

f

. If

σ > 1

, choose

f^{2} = e^{a x^{2}}

with

max (0, σ^{- 2} - 1/2) < a < \frac{1}{2 σ ^{2}} .

(4)

Then

E_{p} f^{2} < \infty

but the displayed weighted second-moment integral diverges. Thus no variance range depending only on

σ

can characterize every individual integrand beyond the guaranteed range.

If an additional boundedness assumption on

f

is intended, the familiar sufficient range becomes

0 < σ < 2

. For

f = 1

this condition is also necessary, but a compactly supported

f

can have finite weighted variance for every

σ

. Distinguishing these assumptions prevents imposing an unstated restriction on the source integrand.

 Read the full article

Past exam of the simulation and statistical programming course of the University of Oxford / 2021 / 1 / c / Solution 2026-10-25

 View more

An importance weight has expected value

E_{q} [w (Z)] = \int p (z) d z = 1

. From the ratio in (a),

E_{q} [w (Z)^{2}] = s^{d} \prod_{j = 1}^{d} E_{Z_{j} \sim N (0, s)} [e^{- (1 - s^{- 1}) Z_{j}^{2}}] .

(1)

The supplied normal distribution transform applies with quadratic coefficient

- (1 - s^{- 1})

, giving each factor

(2 s - 1)^{- 1/2}

. Thus

Var_{q} (w (Z)) = (\frac{s}{2 s - 1})^{d} - 1.

(2)

For

s > 1

, the base

A := s / 2 s - 1

exceeds one because

s^{2} - (2 s - 1) = (s - 1)^{2} > 0

: importance weight variance increases exponentially with dimension. For

s = 1

, the importance weight is identically one and its variance is zero. This calculation concerns the importance weight, rather than the weighted integrand.

 Read the full article

Past exam of the simulation and statistical programming course of the University of Oxford / 2021 / 1 / d / iii / Solution 2026-10-25

 View more

The integrand is positive, so the minimum-variance importance distribution is its target-weighted probability density function, normalized by

θ = e

q^{⋆} (z) = \frac{e ^{- z_{1}} p ( z )}{e} .

(1)

By completing the square,

- z_{1} - \frac{1}{2} z_{1}^{2} - \frac{1}{2} = - \frac{1}{2} (z_{1} + 1)^{2} .

(2)

Thus

q^{⋆}

is the multivariate normal distribution

N (- e_{1}, I_{d})

, where

e_{1} = (1, 0, \dots, 0)

is the first standard basis vector. Its importance weight and weighted integrand are

w^{⋆} (z) = e^{z_{1} + 1/2}, w^{⋆} (z) e^{- z_{1}} = e, q min Var (θ_{n}^{IS}) = 0.

(3)

A variance cannot be negative, and this proposal makes every summand constant, proving optimality directly. It is admissible under the arbitrary-proposal condition even though its mean vector is not zero. Although optimal proposals usually involve an unknown normalizing integral, here both that integral and the resulting normal distribution are explicit.

 Read the full article

Past exam of the simulation and statistical programming course of the University of Oxford / 2021 / 1 / d / ii / Solution 2026-10-25

 View more

At equal sample count, rejection sampling eventually has smaller variance for fixed $σ^{2} > 1$ ; at equal expected proposal cost, importance sampling eventually wins. The distinction matters because a sample of the rejection sampling estimator means an accepted draw.

For the importance sampling estimator, all coordinates contribute to the importance weight although only one enters the integrand. The same normal distribution transform as in (c), now with linear coefficient

- 2

for coordinate one, gives

E_{q} [(w (Z) e^{- Z_{1}})^{2}] = A^{d} exp (\frac{2 s}{2 s - 1}), A = \frac{s}{2 s - 1} .

(1)

Both estimators are unbiased estimators of

e

, and

Var (θ_{n}^{IS}) = \frac{A ^{d} e ^{2 s / (2 s - 1)} - e}{n}, Var (θ_{n}^{RS}) = \frac{e ^{2} - e}{n} .

(2)

For fixed

s > 1

, the first variance grows exponentially while the second stays constant. This is the benefit of returning exact target draws in rejection sampling. However, obtaining those draws costs

s^{d /2}

proposals each on average.

To compare computational work, multiply each variance by its expected proposal count. For importance sampling that count is

n

; for rejection sampling it is

n s^{d /2}

. The ratio of the resulting cost-normalized variances is

\frac{A ^{d} e ^{2 s / (2 s - 1)} - e}{s ^{d /2} ( e ^{2} - e )} ⟶ 0 (d \to \infty),

(3)

because

A / s = s / (2 s - 1) < 1

. Thus importance sampling eventually gives better precision for the same expected number of proposals, assuming comparable work per proposal. This is an asymptotic cost comparison, not a guarantee that a fixed small computing budget produces any accepted draws. Both methods still deteriorate exponentially in this comparison. For