Back

Maximum-Entropy Distributions Zoo

What you are seeing: if all you know about a random variable is a couple of summary statistics (its support, mean, variance, etc.), what is the most "honest" probability density you can write down? The maximum-entropy principle picks the density p(x)p(x) that uses no extra structure beyond what those statistics force, by maximizing the differential entropy h(p)=−∫p(x) log⁡p(x) dxh(p) = -\int p(x)\,\log p(x)\,dx.

The answer depends entirely on the constraint:

The plot shows the chosen pdf with its analytic entropy and a numerical trapezoidal estimate. They agree to a few percent.

Figure 1. One-dimensional maximum-entropy distributions for four common constraint families. Method: closed-form pdfs, trapezoidal entropy on a 500-point grid.
family
mu0.00
scale (sigma/b/mean/width)1.00
added structure0.00

WHAT TO TRY

  • Switch the family: fix only the support and you get the uniform, fix mean and variance and you get the Gaussian, fix the mean on a positive half-line and you get the exponential. Each is the most honest density given its constraints.
  • Slide the scale (sigma, b, mean, width): the maximum-entropy density stretches to match while staying the flattest shape consistent with what you fixed.
  • Read the entropy: among all densities meeting the same constraints, the one shown has the highest differential entropy. Any other choice secretly assumes extra structure.