A new arXiv paper proposes EntropyMoE, a mixture-of-experts architecture for tokenizer-free large language models. Tokenizer-free models process text at the byte level, avoiding fixed token vocabularies, but they still need efficient ways to decide where to spend computation.
The paper argues that current byte-patch architectures often apply the same dense feed-forward computation to every patch, even though some patches are more semantically complex than others. EntropyMoE instead routes patches through sparse experts based on entropy-aware signals, so the model can vary capacity across different pieces of input.
The result is research, not a product release, but it points to a practical problem in future language models: removing tokenizers is attractive for robustness and multilingual coverage, yet it can raise efficiency challenges. If approaches like this hold up under broader testing, tokenizer-free models could become more competitive without simply increasing compute everywhere.