GPU sharing cuts a clinical speech system from 16 servers to four

A speech-recognition deployment described by AWS, Nvidia and Heidi Health cut its GPU fleet from 16 instances to four by allowing multiple inference processes to share each accelerator. The setup maintained sub-second latency while handling 92.1 requests per second per GPU in the reported test.

A single request to the 600-million-parameter Nvidia Parakeet TDT model used only about 15 to 20 percent of a GPU. Default time slicing served processes sequentially, leaving much of the chip idle. CUDA Multi-Process Service allowed their kernels to run concurrently, while Nvidia Triton handled request scheduling. The team also used ONNX and TensorRT optimizations before deploying on Amazon EC2.

The 75 percent reduction is a result from this specific clinical-transcription workload, not a universal saving for every model. Gains depend on request shape, model size and latency targets, and sharing can introduce contention if demand is not controlled. The example nevertheless shows that utilization work can reduce serving costs without changing the underlying recognition model.