EM on a 2D Gaussian mixture

What you are seeing: a synthetic 2D dataset drawn from three Gaussian clusters with known parameters (their true 2-sigma ellipses are drawn faint). The EM algorithm tries to recover those parameters using nothing but the data: it alternates between soft-assigning each point to a cluster (E-step) and re-fitting each cluster's mean, covariance, and weight from the soft assignments (M-step).

Each iteration is guaranteed to never decrease the log-likelihood $\log p(X | \theta) = \sum_n \log \sum_k \pi_k\,N(x_n; \mu_k, \Sigma_k)$. Watch the trace under the plot. The cluster ellipses are 2-sigma confidence regions of the currently estimated $\Sigma_k$. Each data point is colored by its argmax responsibility $\gamma_{nk}$.