For inverse-gamma distribution observations of known shape and scale , their likelihood function in is proportional to . A shape-, rate- gamma distribution conjugate prior therefore gives the displayed Bayesian posterior. The Bayes estimator under squared error loss of the conditional claim expected value is . It depends on reciprocals, rather than the sample mean of the claims.
Gamma scale inverse-gamma conjugacy Created 2026-10-06 Updated 2026-10-07
For conditionally independent gamma distribution observations of known shape and unknown scale , an inverse-gamma distribution prior with shape and scale gives an inverse-gamma distribution posterior with shape and scale . The posterior mean of the conditional claim expected value is when the denominator is positive.
Take independent proper normal prior distributions for the two segment means and a proper inverse-gamma distribution for the common variance. A uniform prior distribution over may then retain the empty second segment. The posterior distribution is proper: its likelihood function is bounded above by , whose integral against is finite for . At the second mean has its unchanged proper prior distribution as its full conditional distribution. The configuration's posterior probability now has the meaningful interpretation of no change inside the observed sample.
Let and . With Lebesgue base measure for , the likelihood-power law has conditionals
The joint law is proper for , since its marginal is inverse-gamma distribution with shape and scale . There is also a direct convergence proof for successive Gibbs sampling updates with deterministic and . Update from , then from . Inverse-gamma distribution moments and conditional normal moments give
The coefficients of the preceding moments are bounded by and , so these moments stay bounded from a finite deterministic initial variance. Letting gives and . Also . Thus the iterates converge in mean square to the normal maximum-likelihood estimate.
In a finite normal distribution mixture with allocation labels , independently assign , and weights a Dirichlet distribution. Let count allocations to component and sum their observations. The conditional mean precision is , giving . The conditional variance is the displayed inverse-gamma distribution; weights have Dirichlet parameters increased by allocation counts. Allocation probabilities are proportional to . Independence of the mean prior from is important: its density does not add an extra inverse-gamma shape term.
With independent prior distributions and in a normal distribution regression model, the conditional posterior distribution of has covariance matrix and mean . The conditional posterior distribution of is inverse-gamma distribution . The shape contains no half-dimension contribution from , since its prior distribution is independent of . These conditionals give a two-block Gibbs sampler.
Start with and any coefficient vector. At sweep , draw from the multivariate normal distribution above using , then draw from the inverse-gamma distribution using the newly drawn coefficients. For the second step one may generate from a gamma distribution with the same shape and rate equal to the displayed inverse-gamma scale.
Each block update uses an exact conditional distribution, so it preserves the joint posterior distribution. The resulting pairs form a dependent Gibbs sampler sample. With proper priors, positive scale and positive-definite coefficient covariance, these conditionals have full probability support on and the usual posterior ergodicity applies. A burn-in reduces dependence on initialization; retained sweeps are still serially dependent and Monte Carlo errors should account for that dependence.
Write , , , , and , with positive-definite. For fixed , the terms involving in the posterior exponent are
Complete the square. With and , this gives
For fixed , collect the powers and exponential terms involving :
Hence, in the paper's shape-scale inverse-gamma distribution convention,
There is no extra in this shape, because the normal coefficient prior is independent of . These are the independent normal and inverse-gamma regression priors conditionals.
The Gaussian normalization printed in the PDF is incorrect. The proper -dimensional prior probability density function is . For fixed this correction is constant in and leaves the two conditionals unchanged, but it must be retained when comparing different model orders in part (b).
We prove convergence of the actual sequential updates above, not just concentration of ideal equilibrium distributions. Write and . Given , the centered normal draw satisfies
The conditional inverse-gamma distribution moments therefore give
For , the coefficient multiplying is at most , so stays bounded by a geometric-recursion argument. The coefficient multiplying is at most , and the other terms are bounded using the first-moment bound, so also stays bounded. These bounds start from the finite deterministic .
Since , the boundedness makes every term containing or vanish. Hence and . Consequently
Thus
This establishes mean-square convergence for the specified Gaussian annealer. It does not assert that every implementation or cooling rule for every objective has the same guarantee, nor that mean-square convergence alone is almost-sure convergence.
Write for each component variance and for the fixed prior variance denoted in the source; these must not be confused. Let be the normal distribution probability density function and . Independent observations and independent parameter priors give the posterior distribution
Its support has , and , with . The component means have independent fixed-variance priors with a normal distribution, not means with variances proportional to . The factor follows from the stated inverse-gamma distribution. The later printed augmented expression instead has and an undefined prior mean; those are inconsistent with this specification. The intended model uses and prior mean zero throughout.
For minimization of an energy , simulated annealing at temperature targets the Boltzmann distribution relative to a specified parameter reference measure. With proposal distribution , accept with
At each temperature run the chain long enough to explore that law, then cool and retain the best state seen. For symmetric proposals every improvement is accepted, while a worsening move is accepted with probability . Equilibrium laws concentrate near global minima when their normalizing integrals are finite and the minima are suitably isolated. A particular rapid cooling schedule does not automatically inherit this guarantee.
Likelihood maximization uses energy , not . The source's later wording “ equal to the log-likelihood” conflicts with its preceding minimization convention. The intended maximum-likelihood Boltzmann distribution is proportional to . With the opposite sign, likelihood maxima would be disfavored; for binomial data with both successes and failures the density often would not even be integrable near the endpoints as decreases.
For the maximum-likelihood estimates, write , and . In the binomial distribution model with known positive integer , the log-likelihood is
Differentiating gives , hence . The second derivative is negative in the interior. If or , the maximum is at the corresponding endpoint. This model requires integer observations in ; otherwise its likelihood is zero.
For the normal distribution model write . Its log-likelihood is
For fixed , the unique maximizing mean is . Substituting it and differentiating in gives , so, assuming ,
If , the normal density likelihood grows without bound as with ; there is no finite interior maximum or ordinary finite AIC for that fit.
For Gaussian likelihood Gibbs annealing, use relative to . Completing the square and identifying an inverse-gamma distribution gives the Gibbs sampler conditionals
Integrating out contributes a factor proportional to , so the marginal is . The joint target is proper for when ; starting the Lebesgue-based annealing above this range would require a proper reference factor or bounded parameter domain. For sufficiently small , its moments satisfy
Thus equilibrium draws concentrate at as .
There is also a direct convergence proof for the actual successive Gibbs sampler updates here. Choose deterministic with , start at a finite deterministic , draw from the first conditional using , then draw from the second using . Conditional normal moments give and . The permitted inverse-gamma moments therefore yield
The coefficient on the preceding first moment is at most , and that on the preceding second moment is at most . The remaining coefficients are uniformly bounded, so both moment sequences are bounded by iteration. Since , the recursions then give and . Also . Hence
This proves convergence in mean square, and thus in probability, to the normal maximum-likelihood estimates for this explicit cooling/update scheme, rather than merely assuming equilibration.
For binomial likelihood-power annealing, normalizing on gives
Its mean is , and its variance is
The mean tends to and the variance tends to zero, proving concentration at the binomial maximum-likelihood estimate, including the endpoint cases. Independent exact draws from these beta laws are a simple annealing implementation within this model.
For model selection, the Akaike information criterion is with and . Thus
Use the endpoint convention . For a trans-dimensional annealing for penalized likelihood construction, target the disjoint model spaces with unnormalized densities
relative to and , respectively. The energies are half the unoptimized AIC, ; their joint minimizer is the fitted model with smaller AIC. Keep constants such as and the normal density normalization, which do not cancel across models. The within-model conditional draws are unchanged by these constant penalties.
Here is a complete dimension-changing proposal. Select a binomial-to-normal move with probability and a normal-to-binomial move with probability , both fixed; other steps can use the within-model updates. From , draw with density and set
This bijection from onto has inverse , . Its absolute Jacobian determinant is . The Binomial-normal reversible-jump annealing acceptance ratio is therefore
For a reverse proposal from , compute and deterministically and accept with
These ratios are reciprocal at inverse states and enforce detailed balance on the dimension-matched augmented spaces. They specify the proposal density, model-selection probabilities and Jacobian explicitly. The map is a valid example, not a claim of optimal mixing. Positive-temperature beta draws stay in the open interval even when the limiting estimate is an endpoint. At each temperature ensure adequate model switching; retain the best fitted penalized likelihood encountered. If minima tie, limiting concentration need not choose a unique model.
There is a material measurement qualification. A binomial probability mass and an unrounded normal density use different observation reference measures. Their raw AIC values are not an invariant, scientifically calibrated comparison of the same recorded data. The expressions and jump algorithm above implement the formal density comparison suggested by the question, with its units fixed. For integer-recorded measurements, a compatible normal model instead assigns probabilities , suitably adjusting boundaries or truncation if required. Then both models describe the same recorded outcomes relative to counting measure. Use those masses in the normal likelihood and the same reversible-jump formula; its simple normal MLE and conjugate Gibbs updates will generally no longer apply. This observation-model correction cannot be silently combined with the continuous-normal formulas.
For minimizing an energy , simulated annealing at temperature uses the Boltzmann distribution . At each temperature run a Metropolis–Hastings algorithm or Gibbs transition preserving that law, then reduce . With a symmetric proposal, accept with probability : downhill moves are accepted and uphill moves remain possible. Keep the best state encountered. As temperature decreases, equilibrium mass favors global minima, provided the target is proper and the relevant concentration hypotheses hold. An arbitrary finite cooling run does not by itself prove global convergence.
Let , and . For the Poisson distribution model the observations must be nonnegative integers. Its log-likelihood is
For , differentiation gives and . If , the likelihood is decreasing and the maximizer is the boundary value zero. For the normal distribution model write , giving
For each , this is maximized at , and differentiating there gives . Hence
The normal variance estimate has denominator , not . A finite positive normal maximum requires ; if all observations coincide, the likelihood is unbounded as . The normal annealing calculation below assumes .
To maximize likelihood, choose the energy , so the annealing law is , not . This sign is important in the Poisson request: literally inserting a positive log-likelihood into a minimizing Boltzmann law would give a nonnormalizable probability density function growing like at infinity. We use likelihood-power annealing with the maximizing sign.
For the normal distribution model, take base measure . The joint annealing probability density function is
Completing the square gives the Gibbs sampling conditional for . Matching the conditional probability density function to the supplied inverse-gamma distribution convention gives
Integrating out contributes a factor , so the marginal law is
Thus this joint Boltzmann law is proper for . Its variance marginal and the normal conditional both concentrate at the stated estimates as . For example, for its marginal mean and variance are
Also .
There is a direct convergence proof for the actual changing-temperature Gibbs algorithm, beyond convergence of its equilibrium probability densities. Start at a finite deterministic , choose deterministic with , draw conditional on , then draw conditional on using the two boxed distributions. Conditional normal moments give
The inverse-gamma distribution first and second moments consequently yield
and
In the first recursion the coefficient of the preceding mean is at most ; in the second the coefficient of the preceding second moment is at most . The means therefore stay bounded, and then so do the second moments. Letting in these formulas gives and . Together with , this proves
This Gaussian likelihood Gibbs annealing convergence uses the explicit conditional structure and does not assume that one Gibbs sampling sweep is an exact equilibrium draw.
For the Poisson distribution model, the likelihood-power law with respect to has the normalized probability density function
so
The mean tends to and the variance to zero, proving convergence to a point mass at the Poisson maximum-likelihood estimate, including the zero-boundary case.
The Akaike information criterion is , counting one parameter for Poisson and two for normal. With and , the two forms are
and
Choose the smaller criterion. Likelihood constants that differ between models, including the factorial and terms, cannot be discarded.
For a fully specified trans-dimensional annealing for penalized likelihood, let or and define
Use the combined target, relative to counting measure on models and Lebesgue measure on their parameters,
where are fixed reference weights. For and both model components are integrable. The factor is a convenient temperature convention keeping the within-model laws exactly those already derived. The extra factor makes the zero-temperature objective AIC rather than unpenalized likelihood.
Use, for example, probability for a model-change attempt in either model and probability for a within-model update. Within the Poisson distribution model draw a fresh gamma value from its boxed law; within the normal distribution model perform the boxed Gibbs sampling sweep. For a Poisson-to-normal proposal, draw with a fixed , and set
The inverse normal-to-Poisson proposal is with reverse auxiliary . This dimension matching uses one auxiliary variable to augment the one-dimensional Poisson parameter space. The absolute Jacobian determinant is
If the forward model-jump selection probability is and the reverse probability is , the reversible-jump Markov chain Monte Carlo acceptance ratio is
Accept with probability . For the reverse move compute and use
For the suggested equal selection probabilities, and those factors cancel. The probability density function must still be included, despite symmetry of its normal distribution: it is an auxiliary draw in only the forward dimension-increasing proposal. Pairing a move with its inverse makes the accepted forward and reverse probability flows equal after the change of variables, proving detailed balance for each fixed-temperature model-switch kernel.
Cool through proper temperatures, allow enough model and parameter exploration at each stage, and retain the smallest observed . The minima of the two energies are exactly the two AIC values above. Low-temperature equilibrium mass concentrates on the better criterion and its fitted parameters; if they tie, the minimizing set contains both models. Unlike the direct normal Gibbs argument, this multimodel exploration claim needs an appropriate cooling/mixing justification and is not guaranteed by an arbitrary short run.
Finally, an exact statistical comparison must put both likelihoods on the same observable and reference-measure convention. A Poisson distribution model is a probability mass function on counts, whereas the displayed normal distribution model is a continuous probability density function. The formulas above are the question's formal Gaussian working-likelihood comparison. A coherent exact normal approximation for count observations can instead use probabilities of rounding intervals; that changes its likelihood and generally loses the simple analytic normal estimates. This distinction matters when interpreting the numerical AIC difference, while leaving the dimension-matching and annealing construction applicable to properly specified likelihoods.
Let , , and . Under the standard interpretation that the two stated priors are independent, multiply the normal likelihood by the normal coefficient prior and the inverse-gamma distribution density. The independent normal and inverse-gamma regression priors give
For a fixed model the determinant and Gaussian normalizing constants are independent of and may be omitted here; is assumed positive-definite. The coefficient prior is not scaled by , so replacing its precision by would describe a different prior.
For an additional check, completing the coefficient square and reading the variance kernel give the full conditional distributions
and
Marginal priors alone would not determine their joint prior without the independence assumption. The front-page inverse-gamma mean has a typographical error: this density has mean when , not a denominator . The density itself, used in the posterior calculation, fixes the correct parameterization.
Use the shape-scale convention for the inverse-gamma distribution: has density proportional to for . Write , , , and . The independent normal and inverse-gamma regression priors make the conditional log-likelihood plus log prior in equal, up to a constant, to
Define and . Completing the square turns this expression into plus a constant. Thus the conditional posterior distribution is the multivariate normal distribution
Positive definiteness of the prior covariance matrix makes this well defined even if the design matrix is rank deficient.
Holding fixed, the factors depending on are
Consequently
There is no additional in the shape: the normal distribution prior for is independent of . A prior scaled by would be a different model and would change that calculation.
A reversible-jump Markov chain Monte Carlo calculation compares normalized model-specific densities. The fixed-order posterior written only up to proportionality in part (a) omits normalizing factors that depend on and therefore cannot simply be canceled between dimensions. Moreover, the printed question does not specify the model-order prior distribution or the probabilities of selecting birth and death moves. Denote those by , and , respectively, retaining them in the answer.
Put , , , and . The centered polynomial birth move in reversible-jump sampling changes the fitted vector by , so its new residual is . Therefore
The map from to has ones down the diagonal and a single additional entry in the intercept row. Its Jacobian determinant is one.
Write and . The normalized multivariate normal distribution prior ratio is
Divide also by the forward proposal density . The common inverse-gamma distribution prior for and the common normal-likelihood normalization cancel, since stays fixed. The forward acceptance ratio is consequently
Here is the proposal standard deviation. If equal model priors and symmetric birth/death selection are explicitly adopted, their ratios become one; they are not determined by the PDF. The expression assumes the same hyperparameters for the variance prior in both models, as in part (a). A dimension-dependent variance prior would contribute its own ratio.
Put , which is positive definite because has full column rank. The log likelihood differs from by a term independent of . Differentiation gives the normal equations , hence
The distribution follows by writing and applying the linear transformation rule for a multivariate normal distribution.
Let and . Orthogonality of the residual to the columns of gives the exact decomposition
The prior is density proportional to with respect to . Thus the joint posterior distribution is
Completing the quadratic identifies
The sampling distribution treats the true as fixed and varies the data, whereas the conditional posterior fixes the observed data and distributes . Their covariance formulas agree, but their centres and their interpretations differ.
For a new row vector , let and suppose its new error is independent. With known , the maximum-likelihood plug-in prediction uses mean and noise variance , while a calibrated frequentist prediction must account for the estimator: . If variance is estimated, the maximum-likelihood estimator is , whereas the exact frequentist prediction pivot uses and has Student t-distribution with degrees of freedom.
In the Bayesian calculation, integrating first gives . Integrating out of the joint posterior gives . For , integration of the predictive normal against this inverse-gamma distribution has density proportional to
The substitution reduces the integral to a Gamma function. Normalizing identifies the Student t-distribution, giving
where the second parameter is a scale, not its variance. These Bayesian intervals coincide with the usual exact Gaussian-regression prediction intervals for this prior, although their probability interpretations differ. Posterior propriety requires and ; the former holds almost surely under the nondegenerate Gaussian model. Prediction of the noiseless conditional mean omits the added one in .
The bounded uniform prior distributions on the slopes and population intercept are proper and broad on their stated scale. They allow either sign but are not invariant to changes of units. The residual precision parameter has a proper shape-rate prior. Equivalently, has inverse-gamma distribution with the same shape and scale. This is a commonly used diffuse choice, but it is not literally noninformative and its tail behaviour merits sensitivity analysis.
The prior for is uniform on , so the induced density of is
It is integrable at zero, unlike the usual scale-invariant variance kernel . The latter kernel, often called the standard variance Jeffreys prior, would cause an improper posterior from a log-uniform random-effect scale prior here. After integrating the child intercepts, each child's response vector has a normal likelihood with covariance matrix . For fixed positive , this covariance remains nonsingular at , and the marginal likelihood has a positive finite limit there. On a compact set of the other parameters its positivity gives
Consequently the scale-invariant kernel cannot be normalized to a posterior distribution. A proper prior on the standard deviation avoids this problem. This familiar scale prior is not necessarily the actual Jeffreys prior for an additive variance component derived from the observed-data likelihood; the latter can be finite at zero.