A hyperparameter controls a family of prior distributions or statistical models and is fixed, estimated, or assigned a further prior at a higher level of a hierarchical model.
A hyperprior is a prior distribution assigned to a hyperparameter that controls another prior distribution. In a hierarchical Bayesian model, this makes uncertainty about the prior's parameters part of the joint probability model, rather than replacing those parameters by fixed numbers.[ref]
Stan User's Guide by Stan Development Team. Version2.37, Regression Models, Hierarchical priors
For data , a model parameter and a hyperparameter , a common factorization isHere supplies the likelihood function, is the conditional prior distribution, and is the hyperprior. After observing , Bayesian inference gives a joint posterior distribution. The marginal posterior of averages over the updated hyperparameter distribution:For discrete hyperparameters, the integral is replaced by a sum.
An exact example makes the averaging visible. Let be a success probability and choose a two-point hyperprior,Both conditional beta distributions have expectation , but they concentrate differently around it. Now observe two successes that are independent conditional on , each with a Bernoulli distribution. Their conditional probability is . Integrating against the two possible priors givesThus the data also update the hyperparameter:The conditional posterior distributions of are respectively and . Their posterior mixture has meanThis is an original illustrative probability model, rather than data or a numerical example from the guide.
Fixing at its prior mean would instead produce after the same observations, with posterior mean . Averaging the hyperparameter first has changed the model; it does not reproduce averaging the conditional posteriors with their updated probabilities. A hyperprior therefore adds a specified layer of uncertainty, while its own choice remains part of the assumptions of the hierarchical Bayesian model.
Articles by others on the same topic
A hyperparameter is a configuration or parameter that is set before the training of a machine learning model begins and is not learned from the data during training. Essentially, these parameters influence the training process itself and can affect the model's performance. Hyperparameters differ from model parameters, which are the values adjusted by the learning algorithm during the training process, such as weights in a neural network.
In the case of machine learning in particular, it is not part of the training data set.
Hyperparameters can also be considered in domains outside of machine learning however, e.g. the step size in partial differential equation solver is entirely independent from the problem itself and could be considered a hyperparamter. One difference from machine learning however is that step size hyperparameters in numerical analysis are clearly better if smaller at a higher computational cost. In machine learning however, there is often an optimum somewhere, beyond which overfitting becomes excessive.