Let $X$ be a discrete random variable on a finite set $\mathcal{X}=\{x_{1},\ldots,x_{n}\}$, with probability distribution function $p(x)=\Pr(X=x)$. The entropy $H(X)$ of $X$ is defined as

The convention $0\log 0=0$ is adopted in the definition. The logarithm is usually taken to the base 2, in which case the entropy is measured in “bits,” or to the base e, in which case $H(X)$ is measured in “nats.”

If $X$ and $Y$ are random variables on $\mathcal{X}$ and $\mathcal{Y}$ respectively, the joint entropy of $X$ and $Y$ is

where $p(x,y)$ denote the joint distribution of $X$ and $Y$.

The Shannon entropy was first introduced by Shannon in 1948 in his landmark paper “A Mathematical Theory of Communication.” The entropy is a functional of the probability distribution function $p(x)$, and is sometime written as

It is noted that the entropy of $X$ does not depend on the actual values of $X$, it only depends on $p(x)$. The definition of Shannon’s entropy can be written as an expectation

The quantity $-\log_{b}p(x)$ is interpreted as the information content of the outcome $x\in\mathcal{X}$, and is also called the Hartley information of $x$. Hence the Shannon’s entropy is the average amount of information contained in random variable $X$, it is also the uncertainty removed after the actual outcome of $X$ is revealed.

We write $H(X)$ as $H_{n}(p_{1},\ldots,p_{n})$. The Shannon entropy satisfies the following properties.

1. 1.

   For any $n$, $H_{n}(p_{1},\ldots,p_{n})$ is a continuous and symmetric function on variables $p_{1}$, $p_{2},\ldots,p_{n}$.

2. 2.

   Event of probability zero does not contribute to the entropy, i.e. for any $n$,

   $$H_{{n+1}}(p_{1},\ldots,p_{n},0)=H_{n}(p_{1},\ldots,p_{n}).$$

3. 3.

   Entropy is maximized when the probability distribution is uniform. For all $n$,

   $$H_{n}(p_{1},\ldots,p_{n})\leq H_{n}\Big(\frac{1}{n},\ldots,\frac{1}{n}\Big).$$

   This follows from Jensen inequality,

   $$H(X)=E\Big[\log_{b}\Big(\frac{1}{p(X)}\Big)\Big]\leq\log_{b}\Big(E\Big[\frac{1}{p(X)}\Big]\Big)=\log_{b}(n).$$

4. 4.

   If $p_{{ij}}$, $1\leq i\leq m$, $1\leq j\leq n$ are non-negative real numbers summing up to one, and $q_{i}=\sum_{{j=1}}^{n}p_{{ij}}$, then

   $$H_{{mn}}(p_{{11}},\ldots,p_{{mn}})=H_{m}(q_{1},\ldots,q_{m})+\sum_{{i=1}}^{m}q_{i}H_{n}\Big(\frac{p_{{i1}}}{q_{i}},\ldots,\frac{p_{{in}}}{q_{i}}\Big).$$

   If we partition the $mn$ outcomes of the random experiment into $m$ groups, each group contains $n$ elements, we can do the experiment in two steps: first determine the group to which the actual outcome belongs to, and second find the outcome in this group. The probability that you will observe group $i$ is $q_{i}$. The conditional probability distribution function given group $i$ is $(p_{{i1}}/q_{i},\ldots,p_{{in}}/q_{i})$. The entropy

   $$H_{n}\Big(\frac{p_{{i1}}}{q_{i}},\ldots,\frac{p_{{in}}}{q_{i}}\Big)$$

   is the entropy of the probability distribution conditioned on group $i$. Property 4 says that the total information is the sum of the information you gain in the first step, $H_{m}(q_{1},\ldots,q_{m})$, and a weighted sum of the entropies conditioned on each group.

Khinchin in 1957 showed that the only function satisfying the above assumptions is of the form:

where $k$ is a positive constant, essentially a choice of unit of measure.

Entropy in the continuous case is called differential entropy.

Despite its seductively analogous form, continuous entropy cannot be obtained as a limiting case of discrete entropy.

We wish to obtain a generally finite measure as the “bin size” goes to zero. In the discrete case, the bin size is the (implicit) width of each of the $n$ (finite or infinite) bins/buckets/states whose probabilities are the $p_{n}$. As we generalize to the continuous domain, we must make this width explicit.

To do this, start with a continuous function $f$ discretized as shown in the figure:

As the figure indicates, by the mean-value theorem there exists a value $x_{i}$ in each bin such that

and thus the integral of the function $f$ can be approximated (in the Riemannian sense) by

where this limit and “bin size goes to zero” are equivalent.

We will denote

and expanding the $\log$ we have

As $\Delta\to 0$, we have

This leads us to our definition of the differential entropy (continuous entropy):