Let $p$ and $q$ be probability distributions with supports $\mathcal{X}$ and $\mathcal{Y}$ respectively, where $ \mathcal{X} \subset \mathcal{Y}$ . The relative entropy or Kullback-Leibler distance between two probability distributions $p$ and $q$ is defined as

\begin{equation} D(p||q) \defined \sum_{x \in \mathcal{X}} p(x) \log \frac{p(x)}{q(x)}. \end{equation} While $D(p||q)$ is often called a distance, it is not a true metric because it is not symmetric and does not satisfy the triangle inequality. However, we do have $D(p||q) \ge 0$ with equality iff $p = q$ .

		(1)
		(2)
		(3)
		(4)
		(5)
		(6)

Relative entropy also comes in a continuous version which looks just as one might expect. For continuous distributions $f$ and $g$ , $\mathcal{S}$ the support of $f$ , we have

\begin{equation} D(f||g) \defined \int_{\mathcal{S}} f \log \frac{f}{g}. \end{equation}