Use the Gaussian likelihood Gibbs annealing target relative to , with . Completing the normal square and recognizing an inverse-gamma kernel give
The shape is , because the probability density function in is . The base measure matters: inserting a prior factor would change this shape.
Integrating out contributes a factor proportional to , so the marginal is . Thus the joint target is proper for . Choose a decreasing sequence tending to zero and a finite starting value . At iteration , sample from the first conditional using , then sample from the second using . This is an explicit annealing algorithm entirely using Gibbs sampling, with all its inverse-gamma moments finite for the convergence proof.
At temperature , target a probability density function proportional to on a parameter region where it is normalizable, possibly with a specified proper base measure. A Metropolis–Hastings algorithm proposal has acceptance probability
For a symmetric proposal, improving moves are accepted and a loss is accepted with probability . Run updates while gradually lowering toward zero. This simulated annealing scheme initially permits exploration and increasingly favors high objective values. Appropriate cooling and mixing conditions are needed for a general global convergence theorem; a finite arbitrary schedule does not guarantee finding a global maximum.
To maximize a positive likelihood function, take , which has the same maximizers. The resulting likelihood-power annealing target is , and available Gibbs sampler conditional distributions can replace Metropolis proposals.
Put , and . The likelihood function and log-likelihood are
For every fixed , the unique maximum in is . At that value, changes sign from positive to negative at . Thus, for ,
This is the global maximum-likelihood estimate, since the first maximization was global for each and the second has a unique peak. If , the likelihood function is unbounded at , ; there is no maximizer in the parameter space . The freezing argument below uses the nondegenerate case .
For the recovered of the preceding part, compute the forward ratio from part (i). The reverse probability density function ratio and reverse Jacobian determinant invert every factor, so
Equivalently, with denoting the Gaussian regression likelihood,
The factor is present even though no random variable is generated during the death proposal: it is the probability density function of the auxiliary variable required by the reverse birth. These reciprocal rules give detailed balance on the paired dimension-matching moves.
From order , set , remove that final coefficient, retain for , and restore the intercept by
Keep fixed. This deterministically recovers the lower-order state and the auxiliary variable that would reproduce the higher-order state under a birth. The inverse map also has absolute Jacobian determinant one. In particular, simply deleting the highest coefficient without restoring the intercept is not the reverse of the supplied move.
The order prior and probabilities of choosing upward or downward moves are not specified, so include them explicitly. Let be the model-order prior probability, the birth-selection probability at order , and the reverse death-selection probability. Denote the correctly normalized coefficient prior by and the probability density function of by . Common variance-prior factors cancel since is unchanged.
Put , , and . The centered polynomial birth move in reversible-jump sampling gives , so the residual-sum-of-squares change is
The map is triangular with determinant one. Thus the reversible-jump Markov chain Monte Carlo ratio is
For an entirely explicit Gaussian expression, define and . Dividing the two normalized Gaussian priors and the proposal probability density function gives
Equal model-order priors and equal birth/death selection probabilities remove the first ratio, if that is the intended convention. Dropping the order-dependent normal-prior constants instead would give the wrong acceptance rule.
Start with and any coefficient vector. At sweep , draw from the multivariate normal distribution above using , then draw from the inverse-gamma distribution using the newly drawn coefficients. For the second step one may generate from a gamma distribution with the same shape and rate equal to the displayed inverse-gamma scale.
Each block update uses an exact conditional distribution, so it preserves the joint posterior distribution. The resulting pairs form a dependent Gibbs sampler sample. With proper priors, positive scale and positive-definite coefficient covariance, these conditionals have full probability support on and the usual posterior ergodicity applies. A burn-in reduces dependence on initialization; retained sweeps are still serially dependent and Monte Carlo errors should account for that dependence.
Write , , , , and , with positive-definite. For fixed , the terms involving in the posterior exponent are
Complete the square. With and , this gives
For fixed , collect the powers and exponential terms involving :
Hence, in the paper's shape-scale inverse-gamma distribution convention,
There is no extra in this shape, because the normal coefficient prior is independent of . These are the independent normal and inverse-gamma regression priors conditionals.
The Gaussian normalization printed in the PDF is incorrect. The proper -dimensional prior probability density function is . For fixed this correction is constant in and leaves the two conditionals unchanged, but it must be retained when comparing different model orders in part (b).
For a target probability density function , one systematic sweep of the Gibbs sampler updates coordinates successively:
Each update leaves the target invariant: keeping the other coordinates with their target marginal and replacing one coordinate by its conditional distribution reconstructs the target joint probability density function. Therefore composition of the coordinate updates also preserves . It defines a Markov chain whose consecutive draws are generally dependent. Under irreducibility, aperiodicity and the usual recurrence conditions it converges to the target law from an arbitrary valid start; a stationary start gives exact target marginals immediately. Invariance alone does not prove these ergodicity conditions, and a systematic full sweep need not itself be reversible.
For , the indicator is always one, and the importance weight reduces to
Averaging these independent contributions gives
As a direct normalization check, its expectation is . For this equals , the exact standard Cauchy distribution tail probability. The substitution , rather than with the same symbol left unchanged, explains why the displayed denominator contains .
For , the proposal CDF is . Therefore inverse transform sampling with gives
Equivalently, put , which is also uniform, and use . This equivalent parametrization matches the uniforms used in the final estimator.
For , . Thus the unnormalized integral is and the multiplicative normalization is :
Although this proposal does not cover the whole Cauchy distribution probability support, it covers the entire nonzero integrand , which is the required support condition for importance sampling.
Take the indicator function . Its expectation under the standard Cauchy distribution is the desired upper-tail probability. To use the proposed tail probability density function in the remaining parts, assume : a probability density function proportional to on is not normalizable when that interval contains zero.
Put , with . The Cauchy-Schwarz inequality applied to and gives
Equality holds exactly when is proportional to on the contributing probability support. Therefore the minimum-variance importance distribution is
If has one sign, and the ideal proposal gives zero variance. Its normalization or exact sampling may nevertheless be impractical. If , the integrand vanishes almost everywhere and the estimator is identically zero.
The minimizing probability density function may vanish where even though there. This is valid for the ordinary estimator, which needs probability support only for its integrand. If one insists on exactly the same positive probability support as , the bound is instead an infimum whenever this vanishing occurs, approached by as : because , the second moment lies between and , proving convergence to the bound.
Let . The independent summands have expectation and second moment
When this integral is finite, independence gives
If the second-moment integral diverges, the estimator can still be unbiased and consistent, but its variance is infinite; the finite-variance assumption cannot be omitted.
Write and . If the constants needed to evaluate are unavailable, use the self-normalized importance sampling estimator
The unknown common factor cancels. Under full target probability support and integrability, the numerator and denominator sample means converge to and , respectively, whose ratio is . The same construction applies if only one normalization is unknown by taking the other probability density function already normalized. Sampling from itself is assumed available, even when its normalization is not explicitly known.
This ratio estimator is generally biased at finite , unlike the ordinary normalized-weight estimator. Also, its variance is not the formula in the next part: that formula concerns the ordinary unbiased estimator. Full probability support for is needed here because the denominator estimates the entire target normalizing integral, not just the probability support of .
Assume independent samples , , and wherever can contribute. The importance weight is , and the ordinary importance sampling estimator is
Its expectation is , by cancellation of in the expectation integral. The strong law of large numbers gives almost-sure convergence to . Common probability support is sufficient; the precise support condition for importance sampling only requires covering the nonzero integrand for this particular expectation.
Integration of the Laplace distribution probability density function gives
Solving these two equations for gives
Generate one uniform and return this piecewise expression. The branches agree at and the inverse transform sampling proof establishes the target law without any rejection step.
Use the quantile function for , so a strictly increasing CDF is not required. For any real , the quantile event agrees, up to endpoint null sets, with . Hence
Thus has distribution function and probability density function . This proves inverse transform sampling, including a probability density function with intervals on which the CDF is constant.
Independently generate uniform on and uniform on , using the bounds above. Accept the proposal if
otherwise generate a fresh pair. On acceptance return . The bounding rectangle contains the entire ratio region: , and lies between the extrema of . Conditioning a uniform rectangle point on acceptance makes it uniform on , so the preceding marginal distribution calculation proves that the returned value has the desired Laplace distribution.
For this target , so the region has area two. The rejection sampling acceptance probability is
Endpoints have zero probability and can be excluded to avoid divisions by zero in implementation.

Pinned article: Introduction to the OurBigBook Project

Welcome to the OurBigBook Project! Our goal is to create the perfect publishing platform for STEM subjects, and get university-level students to write the best free STEM tutorials ever.
Everyone is welcome to create an account and play with the site: ourbigbook.com/go/register. We belive that students themselves can write amazing tutorials, but teachers are welcome too. You can write about anything you want, it doesn't have to be STEM or even educational. Silly test content is very welcome and you won't be penalized in any way. Just keep it legal!
We have two killer features:
  1. topics: topics group articles by different users with the same title, e.g. here is the topic for the "Fundamental Theorem of Calculus" ourbigbook.com/go/topic/fundamental-theorem-of-calculus
    Articles of different users are sorted by upvote within each article page. This feature is a bit like:
    • a Wikipedia where each user can have their own version of each article
    • a Q&A website like Stack Overflow, where multiple people can give their views on a given topic, and the best ones are sorted by upvote. Except you don't need to wait for someone to ask first, and any topic goes, no matter how narrow or broad
    This feature makes it possible for readers to find better explanations of any topic created by other writers. And it allows writers to create an explanation in a place that readers might actually find it.
    Figure 1.
    Screenshot of the "Derivative" topic page
    . View it live at: ourbigbook.com/go/topic/derivative
  2. local editing: you can store all your personal knowledge base content locally in a plaintext markup format that can be edited locally and published either:
    This way you can be sure that even if OurBigBook.com were to go down one day (which we have no plans to do as it is quite cheap to host!), your content will still be perfectly readable as a static site.
    Figure 5. . You can also edit articles on the Web editor without installing anything locally.
    Video 3.
    Edit locally and publish demo
    . Source. This shows editing OurBigBook Markup and publishing it using the Visual Studio Code extension.
  3. https://raw.githubusercontent.com/ourbigbook/ourbigbook-media/master/feature/x/hilbert-space-arrow.png
  4. Infinitely deep tables of contents:
    Figure 6.
    Dynamic article tree with infinitely deep table of contents
    .
    Descendant pages can also show up as toplevel e.g.: ourbigbook.com/cirosantilli/chordate-subclade
All our software is open source and hosted at: github.com/ourbigbook/ourbigbook
Further documentation can be found at: docs.ourbigbook.com
Feel free to reach our to us for any help or suggestions: docs.ourbigbook.com/#contact