Mutual Information of a Bivariate Gaussian

What you are seeing: the joint probability density of two correlated Gaussian random variables $X$ and $Y$ . The heatmap is $p(x, y)$ . The curves on the top and right are the marginals $p(x)$ and $p(y)$ . The number $I(X; Y)$ in the corner measures how much knowing $X$ reduces your uncertainty about $Y$ . It is zero when $X$ and $Y$ are independent and grows without bound as they become perfectly correlated.

For a 2D Gaussian with correlation $\rho$ there is a clean closed form: $I(X; Y) = -\tfrac{1}{2}\,\ln(1 - \rho^2).$ Slide $\rho$ to see how the heatmap stretches along the diagonal and how $I$ rises. We compute the same quantity numerically by trapezoidal integration on the same grid and report both numbers; they agree to a few percent at the default resolution.

Figure 1. Joint density

p(x, y)

of a bivariate Gaussian with correlation

\rho

. Top: marginal

p(x)

. Right: marginal

p(y)

. Method: analytic pdf on a 96 x 96 grid; differential entropies by trapezoidal sums.

rho0.600

sigma_x1.00

sigma_y1.00

WHAT TO TRY

Slide rho toward 1: the joint density collapses onto a line and the mutual information I(X;Y) grows without bound. Knowing X then pins down Y.
Set rho to 0: the heatmap becomes an axis-aligned blob, the joint factorizes into its marginals, and I(X;Y) drops to zero. Independence means no shared information.
Change sigma_x or sigma_y: the marginals widen but I(X;Y) depends only on the correlation, not the individual spreads. Mutual information is scale-free.