Structure or Noise?

We show how theory building can naturally distinguish between regularity and randomness. Starting from basic modeling principles, using rate distortion theory and computational mechanics we argue for a general information-theoretic objective function that embodies a trade-off between a model’s complexity and its predictive power. The family of solutions derived from this principle corresponds to a hierarchy of models. At each level of complexity, they achieve maximal predictive power, identifying a process’s exact causal organization in the limit of optimal prediction. Examples show how theory building can profit from analyzing a process’s causal compressibility, which is reflected in the optimal models’ rate-distortion curve.