 Hyperprior

A hyperprior is a prior distribution assigned to a hyperparameter that controls another prior distribution. In a hierarchical Bayesian model, this makes uncertainty about the prior's parameters part of the joint probability model, rather than replacing those parameters by fixed numbers.^[ref]

For data $y$, a model parameter $\theta$ and a hyperparameter $\lambda$, a common factorization isHere $p(y\mid \theta )$ supplies the likelihood function, $p(\theta \mid \lambda )$ is the conditional prior distribution, and $p(\lambda )$ is the hyperprior. After observing $y$, Bayesian inference gives a joint posterior distribution. The marginal posterior of $\theta$ averages over the updated hyperparameter distribution:For discrete hyperparameters, the integral is replaced by a sum.

An exact example makes the averaging visible. Let $\theta$ be a success probability and choose a two-point hyperprior,Both conditional beta distributions have expectation $1/2$, but they concentrate differently around it. Now observe two successes that are independent conditional on $\theta$, each with a Bernoulli distribution. Their conditional probability is ${\theta }^2$. Integrating against the two possible priors givesThus the data also update the hyperparameter:The conditional posterior distributions of $\theta$ are respectively $\mathrm{Beta}(3,1)$ and $\mathrm{Beta}(12,10)$. Their posterior mixture has meanThis is an original illustrative probability model, rather than data or a numerical example from the guide.

Fixing $\kappa$ at its prior mean $11$ would instead produce $\mathrm{Beta}(7.5,5.5)$ after the same observations, with posterior mean $7.5/13\approx 0.577$. Averaging the hyperparameter first has changed the model; it does not reproduce averaging the conditional posteriors with their updated probabilities. A hyperprior therefore adds a specified layer of uncertainty, while its own choice remains part of the assumptions of the hierarchical Bayesian model.

In Bayesian statistics, a hyperprior is a prior distribution placed on the hyperparameters of another distribution, which is itself the prior for the parameters of a model. To clarify, the Bayesian framework involves using prior distributions to quantify our beliefs about parameters before observing data. When these parameters have their own parameters, which we don't know and want to estimate, we refer to those as hyperparameters. The distribution assigned to these hyperparameters is what's known as a hyperprior.