Reduce ASR inference costs by 75% with NVIDIA MPS on Amazon EC2 | Amazon Web Services
This post is a collaboration between AWS, NVIDIA and Heidi.
Reducing automatic speech recognition (ASR) inference costs on Amazon Elastic Compute Cloud (Amazon EC2) becomes critical when GPU utilization per request is low but latency requirements are strict. A single ASR inference request typically uses only 15–20 percent of a GPU’s compute capacity, yet the default time-slicing behavior in NVIDIA CUDA® forces sequential access, leaving 80 percent of the hardware idle. Heidi Health is an AI Care Partner that processes over 2.4 million clinical consultations per week across 190 countries. To sustain sub-second transcription latency at peak traffic, this inefficiency forces the company to run 16 GPU instances.
In a previous post, you learned how to fine-tune a Nemotron speech model, NVIDIA Parakeet TDT 0.6B V2 for clinical speech recognition. In this post, we focus on what comes after fine-tuning: serving that model efficiently. We demonstrate how NVIDIA...
Copyright of this story solely belongs to amazon.com. To see the full text click HERE
