Kullback-Leibler divergence

A non-symmetric distance beetween probability distribution, defined as:

$$D_{KL}(p\mid \mid q):={\mathbb{E}}_p\left[\log \frac{p}{q}\right]={\mathbb{E}}_p\left[\log p-\log q\right]$$

Note that $D_{KL}(p\mid \mid q)=0$ iff $p=q$ a.s.