AlloyDB proxy models replace some LLM calls with local inference

Google shipped AlloyDB AI functions with a proxy model architecture that can replace some external LLM calls with local inference inside the database. InfoQ reports that the system trains a lightweight model from LLM outputs and runs queries at database speed.

The approach targets cost and latency problems that appear when applications call large models repeatedly for structured database tasks. Local proxy models can keep common operations fast while reserving larger models for harder cases.

It reflects a broader pattern in enterprise AI: using smaller specialized models near the data instead of routing every task to a frontier API.