KL divergence asymmetry: mass-covering vs mode-seeking

What you are seeing: two probability distributions on the same axis. The thick blue curve is a fixed bimodal target $P$ (a mixture of two Gaussians at $\pm 2$). The orange curve is a unimodal approximation $Q$ (a single Gaussian) that you control.

KL divergence is not symmetric: $$D(P \| Q) = \int p(x)\,\log \frac{p(x)}{q(x)}\,dx \neq D(Q \| P).$$ Minimizing $D(P \| Q)$ over $Q$ ("forward" KL, also called M-step KL or "moment matching") gives a Q that covers all the mass of P, even if that means putting probability between the two modes where P has none. Minimizing $D(Q \| P)$ (reverse KL, used by standard variational inference) gives a Q that collapses onto a single mode: it ignores the other mode entirely. The two "optimal" Q's are very different.

Two reference markers in the inset boxes show, for the current P, where each KL is minimized over $(\mu_q, \sigma_q)$. Drag Q to compare.