Google Research has proposed a continual-learning framework that gives language models an offline “sleep” phase for consolidating information acquired during use. The goal is to move useful facts from temporary context into model parameters while reducing catastrophic forgetting, where learning new material damages earlier capabilities.
During the active phase, the model processes requests and stores temporary information in fast memory components. In the sleep phase, it stops accepting new inputs and trains on self-generated practice examples. The system unlocks preallocated, low-rank mixture-of-experts memory blocks so incoming knowledge can be added without rewriting the model’s stable core. The neuroscience language is an analogy: the process still runs within a fixed architecture and human-defined training rules.
Researchers report that alternating the two phases helped models acquire new facts, improve reasoning and retain information from contexts reaching 10 million tokens. The work is a research framework rather than a feature available in current consumer assistants. It also does not eliminate the need to evaluate whether consolidated information is accurate. Its main contribution is a possible route beyond two familiar limits: static training cutoffs and memories that disappear when a context window closes.