Back

Maximum-entropy distributions: a small zoo

What you are seeing: if all you know about a random variable is a couple of summary statistics (its support, mean, variance, etc.), what is the most "honest" probability density you can write down? The maximum-entropy principle picks the density p(x)p(x) that uses no extra structure beyond what those statistics force, by maximizing the differential entropy h(p)=−∫p(x) log⁡p(x) dxh(p) = -\int p(x)\,\log p(x)\,dx.

The answer depends entirely on the constraint:

The plot shows the chosen pdf with its analytic entropy and a numerical trapezoidal estimate. They agree to a few percent.

Figure 1. One-dimensional maximum-entropy distributions for four common constraint families. Method: closed-form pdfs, trapezoidal entropy on a 500-point grid.
family
mu0.00
scale (sigma/b/mean/width)1.00
added structure0.00

WHAT TO TRY

  • Vary each control and watch the rail readouts respond.
  • Compare the diagnostic plot against the live scene.