 Kullback–Leibler divergence

For probability densities $p$ and $q$, the Kullback-Leibler divergence from $q$ to $p$ is

Kullback-Leibler divergence, often abbreviated as KL divergence, is a measure from information theory that quantifies how one probability distribution diverges from a second, expected probability distribution. It is particularly useful in various fields such as statistics, machine learning, and information theory.