$\ell_1$ Minimization of Probability Distribution and KL-Divergence
Have you looked into Jensen-Shannon Divergence? It avoids the situation you described by adding a midpoint. Formally we have,
$$\mathrm{JSD}(P||Q) = \frac{1}{2} \mathrm{KL}(P||M) + \frac{1}{2} \mathrm{KL}(Q||M)$$ where $M = \frac{1}{2}(P + Q)$. Informally, JSD is nice when you want to compare distributions that seem to have the same shape, but dont completely overlap.