Researchers have developed a router that chooses among edge and cloud language models for each function-calling request based on capability, delay, power use and the current carbon intensity of electricity.
The system uses three tiers running different models on heterogeneous hardware. A lightweight k-nearest-neighbor predictor estimates whether each tier can answer a particular query accurately, how long it will take and how much power it will consume. Live grid data then helps select the lowest-emission tier expected to succeed.
In evaluations on function-calling benchmarks and several model families, the framework matched the accuracy of routing everything to the cloud while reducing operational carbon emissions by an average factor of four, according to the authors. Keeping suitable requests at the edge can avoid unnecessary use of larger remote models, while difficult requests can still move upward.
The result concerns operational emissions during inference, not the embodied carbon of manufacturing hardware or training models. It also depends on prediction quality, local device efficiency and the availability of timely grid-intensity data. The paper is a new arXiv preprint, so its fourfold average should not be assumed for every workload or region.