How Together AI built the world’s fastest speech-to-text stack
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Together AI details the systems engineering behind their fastest-ranked speech-to-text stack on Artificial Analysis, serving NVIDIA Parakeet-TDT 0.6B v3 and OpenAI Whisper Large v3. Key optimizations include: multi-profile TensorRT engines tuned for real audio input shapes to avoid padding overhead; conditional CUDA graph nodes that move the decoder's branch logic onto the GPU, eliminating CPU round-trips and achieving 2-3x decoder speedup; collapsing preprocessing microservices to reduce kernel copies, using Unix domain sockets and shared memory for zero-copy audio transfer; switching from thread-per-connection to epoll-based evented I/O for streaming to reduce GIL contention and tail latency; and using gc.freeze() after startup preallocation to prevent Python's garbage collector from walking large long-lived object graphs, eliminating ~200ms p95 latency spikes.
