Predictability Shapes Adaptation: An Evolutionary Perspective on Modes of Learning in Transformers

This paper investigates how the predictability of the training environment influences the balance between two distinct learning modes in Transformer models: in-weights learning (IWL), which is analogous to genetic encoding, and in-context learning (ICL), which is compared to phenotypic plasticity. Drawing parallels from evolutionary biology, the authors explore how environmental stability (consistency of tasks) and cue reliability (clarity of in-context examples) affect which learning strategy is favored. Experiments using regression and classification tasks demonstrate that stable environments promote IWL, while reliable cues enhance ICL, particularly in less stable settings. The study also reveals different learning trajectories, including shifts between ICL and IWL, suggesting that the relative computational cost of these strategies drives these dynamic preferences.