FreeToken gives consumer computers a new way to run mixture-of-experts AI models whose full weights are too large for graphics memory. Instead of pausing the GPU whenever it needs an expert stored in system RAM, the open-source engine assigns work to the CPU and GPU according to current data-transfer speed.
Mixture-of-experts models activate only part of their parameters for each token, but they still need to move the selected weights through relatively slow PCIe links. FreeToken overlaps those transfers with computation, changes how graphics memory is divided between model weights and the attention cache, and saves intermediate states at logical boundaries in an agent workflow. That can avoid recomputing an entire prompt when a tool response changes part of the context.
The researchers report three to four times faster decoding and six to 30 times faster prompt processing than comparable runtimes. They ran Qwen3.6-35B at about 39 tokens per second on a laptop with an 8GB RTX 4060 and demonstrated much larger models on desktop workstations. Those are project benchmarks, and developers still need to test CPU dispatch, memory contention and concurrent jobs on their own hardware. The command-line and desktop clients support Nvidia RTX 30-, 40- and 50-series GPUs on Linux and Windows.