Empirical Bayes method

An empirical Bayes method estimates prior hyperparameters from the marginal distribution of the observed data and then uses the resulting fitted prior in Bayesian inference.

A maximum marginal likelihood estimator chooses a hyperparameter by maximizing the data density obtained after integrating the model parameter against its prior distribution.

A hyperparameter controls a family of prior distributions or statistical models and is fixed, estimated, or assigned a further prior at a higher level of a hierarchical model.

A hyperprior is a prior distribution assigned to a hyperparameter that controls another prior distribution. In a hierarchical Bayesian model, this makes uncertainty about the prior's parameters part of the joint probability model, rather than replacing those parameters by fixed numbers.^[ref]

For data $y$, a model parameter $\theta$ and a hyperparameter $\lambda$, a common factorization isHere $p(y\mid \theta )$ supplies the likelihood function, $p(\theta \mid \lambda )$ is the conditional prior distribution, and $p(\lambda )$ is the hyperprior. After observing $y$, Bayesian inference gives a joint posterior distribution. The marginal posterior of $\theta$ averages over the updated hyperparameter distribution:For discrete hyperparameters, the integral is replaced by a sum.

An exact example makes the averaging visible. Let $\theta$ be a success probability and choose a two-point hyperprior,Both conditional beta distributions have expectation $1/2$, but they concentrate differently around it. Now observe two successes that are independent conditional on $\theta$, each with a Bernoulli distribution. Their conditional probability is ${\theta }^2$. Integrating against the two possible priors givesThus the data also update the hyperparameter:The conditional posterior distributions of $\theta$ are respectively $\mathrm{Beta}(3,1)$ and $\mathrm{Beta}(12,10)$. Their posterior mixture has meanThis is an original illustrative probability model, rather than data or a numerical example from the guide.

Fixing $\kappa$ at its prior mean $11$ would instead produce $\mathrm{Beta}(7.5,5.5)$ after the same observations, with posterior mean $7.5/13\approx 0.577$. Averaging the hyperparameter first has changed the model; it does not reproduce averaging the conditional posteriors with their updated probabilities. A hyperprior therefore adds a specified layer of uncertainty, while its own choice remains part of the assumptions of the hierarchical Bayesian model.