The Invariance Principle

How statistical invariances will help us build AI systems exhibiting human-like performance by following human-like strategies.

Current machine learning systems crumble when the distributions of training and testing examples differ in spurious correlations. This is a major roadblock toward the development of advanced machine intelligence, which demands not only human-like performance but the deployment of human-like strategies. The prevalent approach in AI, fixated on recklessly minimizing average training error, falls short in producing AI systems capable of authentic out-of-distribution generalization. This book introduces the Invariance Principle, a new epistemological tool to unearth correlations invariant across diverse collections of empirical data.

The Invariance Principle, encapsulated in the axiom “frame your problem so its answer matches across circumstances," will not only find its practical incarnation in the family of Invariant Risk Minimization algorithms, but also illuminate our understanding of causation. It will permeate topics such as environment discovery, large-language models, self-supervised learning, mixing data augmentation, uncertainty estimation, and fairness. The author argues that the Invariance Principle is a central inductive bias fueling advances across fields of knowledge, such as physics, metaphysics, and cognitive science.

The final chapter includes personal examples of how invariance has shaped the author’s understanding of his own subjective experience, as well as how he has interpreted both Eastern and Western philosophical traditions.