A new arXiv paper introduces the Blockwise Causal Memory Transformer, or BCMT, as a possible way to make long-context language modeling less computationally expensive.
Standard transformer attention compares tokens densely, which becomes costly as sequences grow because the work scales quadratically with length. BCMT instead applies dense causal attention inside local blocks. Each block then creates an adaptive summary, aggregates those summaries through an exponential causal memory, and injects that memory back into token representations.
The goal is to move information across long contexts without explicit global attention between all distant tokens or learned recurrent memory states. That could matter for applications such as long-document analysis, coding, and agent memory, where context length keeps expanding faster than budgets. The paper remains architectural research, so the key question is whether the design can match standard transformers on real workloads while delivering the promised efficiency advantages.