An AWS reference design reduces voice-agent latency by sending each completed sentence to a speech model while the language model continues generating the rest of its answer.
The sample hosts the 82-million-parameter Kokoro text-to-speech model on a SageMaker real-time endpoint. A persistent bidirectional session carries successive sentences and returns binary audio plus JSON control messages. In five isolated trials, sending the first sentence immediately reduced mean time to first audio from 329.3 milliseconds to 156.7 milliseconds, a 52.4 percent improvement.
With a live language model and deployed AgentCore runtime, the bidirectional approach produced first audio 3.5 times faster than waiting for the complete response; the local-container comparison was 2.4 times faster. Synthesis remains sequential, so the gain comes from starting early rather than generating several sentences’ audio at once.
The implementation includes tradeoffs. Existing browser pipelines can carry base64 audio through Server-Sent Events, but encoding expands payloads by about one third and consumers must buffer chunks that split audio frames. A fresh binary path avoids that overhead. AWS also warns that the sample pins an older PyTorch release with a known advisory and should be upgraded before production. The benchmark demonstrates lower startup latency, not overall conversation quality or operating cost.