A global fintech has moved its internal GLM-5.2 coding assistant onto Together AI’s Dedicated Model Inference service, giving its own engineers control over endpoint capacity, configuration and testing. The change is meant to handle traffic that arrives in sharp bursts as teams use coding agents during working hours.
The workload has unusually long prompts and values concurrency more than raw token throughput. Together says the production setup runs at a 256,000-token context length across 56 Nvidia B200 GPUs. When requests later queued for up to three minutes, metrics showed a large pending prefill backlog rather than a compute bottleneck. The customer restored a tuned cache-aware routing policy and changed the number of in-flight requests without redeploying or taking the service offline.
The self-service platform also supports scaling, custom model weights and blue-green testing, where a new configuration runs beside the current one before taking traffic. The team considered a one-million-token context but stayed with 256,000 and 512,000 tokens because the larger window would reduce the concurrency headroom its usage pattern needs. The case study is supplied by the vendor, but it illustrates why production agent infrastructure often depends as much on observability and operational control as headline model speed.