Hugging Face is adding direct support for GGUF model files to its Transformers library, allowing developers to run compact llama.cpp-style checkpoints with the same APIs they use for other models. The initial work focuses on Apple silicon and the Qwen3.5 architecture.
GGUF stores model weights, tokenizer data and optional chat templates in one file. Its quantized variants reduce memory use by representing most weights with fewer bits. In Hugging Face’s example, a Qwen3.5 4B checkpoint falls from 8.42 GB in BF16 format to 2.74 GB with Q4_K_M quantization. The trade-off is some loss of numerical precision, so results should be tested on the intended task.
Transformers reuses llama.cpp’s underlying ggml kernels and keeps compatible weights packed for Apple’s Metal framework. Developers select a Hub model and GGUF filename in `from_pretrained`; generation then uses the standard Transformers interface. If the optimized attention kernel is unavailable, the library falls back to PyTorch’s implementation with a warning.
Support currently requires an Apple silicon Mac, compatible recent PyTorch builds and the development version of Transformers until the next release. Broader architecture and hardware coverage will therefore take time.