Google open-sources TPU Sync to move model caches and weights between systems

Google Cloud has released TPU Sync, an Apache 2.0 C++ library for moving model weights and key-value caches among TPU high-bandwidth memory, host memory, shared storage and the data-center network. Those transfers can leave accelerators waiting during long-context inference and reinforcement-learning updates.

For separated prompt processing and token generation, Google says median cache-transfer latency for Qwen3 235B fell from several seconds to about 230 milliseconds. It also reports average time-to-first-token reductions of 60% for that model and 90% for a 480B model at low concurrency, plus up to 90% on 64,000-token prefix-caching workloads. These are vendor measurements on specific configurations.

Python wheels connect TPU Sync to JAX and PyTorch. It can serve as a cache backend for TPU versions of vLLM and SGLang, and work with llm-d on Kubernetes for prefix-aware routing. The library has been validated on TPU v7x and v6e.

The repository warns that APIs are changing quickly and recommends pinning a known commit. Only development builds are on PyPI; the storage tier and reinforcement-learning weight synchronizer remain experimental, and the PyTorch package requires a matching build that is not on a public index. The release is therefore useful infrastructure for testing, not yet a stable drop-in dependency.