The Voice AI latency problem nobody warns you about
The Voice AI latency problem nobody warns you about
Everyone talks about end-to-end latency - the time between a caller finishing their sentence and the AI starting its response. That number is real and it matters. But it is not the latency problem that kills Voice AI deployments in production. The problem nobody warns you about is cumulative latency - the way small delays stack invisibly across a conversation until the caller feels something is deeply wrong, without being able to explain why.
Every Voice AI vendor will tell you their platform achieves sub-500ms end-to-end latency. Some of them are even telling the truth. But here is what they will not tell you: sub-500ms latency on a single turn means almost nothing if your conversation has twelve turns and the latency creeps up by 50 milliseconds on every one of them.
By the time the caller reaches turn eight, they are waiting 900 milliseconds for a response. They cannot tell you the number. They have no idea what a millisecond feels like. But they can tell you - and they will - that something about the call feels off. The AI seems slow. It seems distracted. It does not feel like talking to something intelligent. It feels like talking to something that is buffering.
This post is about that problem. Why it happens, why it is so hard to catch before go-live, and what you can actually do about it.
Why end-to-end latency is the wrong metric to obsess over
End-to-end latency - the time from when a caller stops speaking to when the AI starts responding - is the metric that appears in every Voice AI vendor pitch deck. It is real, it matters, and sub-500ms is the right target for a single turn. But it is a point-in-time measurement. It tells you how fast your system is when it is fresh, warm, and processing a single utterance in isolation.
Production Voice AI does not work in isolation. It works across a conversation that might last four minutes, covering twelve or fifteen distinct turns. At each turn, the LLM receives a slightly longer conversation history. External API calls accumulate state. The STT engine processes audio from a caller who may be getting progressively more tired, more background-noisy, or more frustrated. The system is not doing the same job on turn twelve that it was doing on turn one.
If latency creeps by even 40 milliseconds per turn - a change that is invisible in any single-turn benchmark - a twelve-turn conversation ends with 480 additional milliseconds of latency on the final response. That is nearly a full extra second of silence on top of whatever the baseline latency already was. Callers do not analyse this. They just experience the call as increasingly unnatural, and they form an impression of the AI accordingly.
When a Voice AI platform publishes a latency benchmark, they are almost always measuring a single turn on a short prompt with a warm model, a pre-cached system prompt, and no function calls. That benchmark is not lying - it is accurately describing performance under ideal, isolated conditions. It is just not describing the conditions your callers will experience in a real conversation at peak traffic on a Monday morning.
The four hidden sources of latency accumulation
Cumulative latency does not come from one big failure. It comes from four small problems that each add a few milliseconds, and together add hundreds of milliseconds by the end of a conversation.
Every time the caller speaks, the transcript is appended to the conversation history sent to the LLM. By turn ten, the LLM is processing significantly more text than it was on turn one. Token processing is not free - a longer input means a longer time to first token. On a conversation with twelve turns and an average of 40 words per turn, you are adding roughly 480 words of context over the life of the call. For most LLMs, this adds 30 to 80 milliseconds to first-token latency by the end of the conversation. Invisible per turn. Significant in aggregate.
Many Voice AI deployments involve function calls - the LLM calling external APIs to check order status, retrieve account information, or look up booking availability. On a simple call, there might be one function call. On a complex call, there might be three or four, each adding the latency of the external API response. If your CRM API averages 200ms and your booking system API averages 350ms, a turn that requires both adds 550ms of pure API wait time on top of everything else in the pipeline. This does not show up in single-turn latency tests because single-turn tests rarely model realistic function call chains.
Network conditions are not static over the lifetime of a call. A caller who starts on a strong WiFi connection might walk into an area with weaker signal two minutes in. A caller on a mobile network experiences varying tower signal strength as they move. Each of these changes affects the RTP audio stream, which affects jitter buffer adjustments, which adds small amounts of processing delay. None of these are individually significant. Accumulated over a four-minute call, they can add 100 to 150 milliseconds of effective latency that was not present at the start of the conversation.
Speech-to-text engines perform differently on the first utterance of a call versus later utterances. On the first turn, the STT model has no acoustic context for this caller - their accent, their speaking pace, their ambient noise environment. By turn three or four, the model has calibrated. Paradoxically, this means early turns sometimes process faster than later turns - not because later turns are slower, but because the model is doing more acoustic adaptation work. Additionally, if a caller's audio quality degrades over the call, the STT engine may require more processing to reach the same confidence threshold, adding 20 to 50 milliseconds per affected turn.
Why this is so hard to catch before go-live
The standard approach to Voice AI testing is to write a set of test utterances, run them through the system one at a time, and measure latency on each response. This is a reasonable starting point. It is also almost completely useless for catching cumulative latency problems, because it tests turns in isolation rather than as a sequence.
What you need instead is end-to-end conversation simulation - automated tests that run a complete simulated conversation from turn one to turn fifteen, measure latency at each turn individually, and track the trend over the lifetime of the call. If your latency is flat across all fifteen turns, you are in good shape. If it is creeping - even slowly - you have a problem that will show up in production.
The second testing gap is load. Single-conversation latency testing tells you how the system performs with one concurrent call. Your production system may need to handle fifty, five hundred, or five thousand concurrent calls. LLM inference latency, STT API response times, and TTS generation all degrade under load - and they degrade non-linearly. The 50th concurrent call is not 50 times slower than the first. But it is meaningfully slower, and that additional latency arrives exactly when your system is under the most stress - during peak traffic hours when caller experience matters most.
"The demo always looks fast because demos are single-turn, warm-cache, low-load tests. Production is multi-turn, cold-start, peak-load reality. These are two completely different performance environments."
- What I now say at the start of every Voice AI UAT conversationWhat I have seen go wrong in production
On one deployment, our UAT testing showed consistent sub-400ms latency across all test cases. The client signed off. We went live. Within 48 hours, the client's customer satisfaction scores on AI-handled calls were significantly below the human-agent baseline - despite the AI giving accurate, relevant answers.
When we pulled call recordings and analysed turn-by-turn latency, the pattern was clear. Turns one through four were averaging 380ms - exactly what our UAT showed. Turn five through eight were averaging 520ms. Turn nine through twelve were averaging 780ms. The calls that prompted complaints were almost all longer conversations - exactly the ones where cumulative latency had pushed response time well past the threshold of naturalness.
The fix had three parts: We implemented context window pruning - trimming the earliest turns of the conversation history once the context exceeded a threshold, keeping the LLM input size bounded. We added a response cache for the three most common function calls, reducing API wait time by 60%. And we moved our load testing to simulate fifty concurrent conversations simultaneously rather than one at a time. The P95 latency on turn ten dropped from 780ms to 490ms. The CSAT scores recovered within two weeks.
How to measure latency correctly in Voice AI
The right way to measure Voice AI latency is not a single number. It is a distribution across the full conversation lifecycle, measured under realistic load. Here are the specific metrics that actually matter:
| Metric | Target | Why it matters |
|---|---|---|
| P50 latency (turn 1–3) | <400ms | First impression - callers form their view of AI quality in the first 30 seconds |
| P50 latency (turn 8–12) | <550ms | Late-conversation - where cumulative degradation becomes audible |
| P95 latency (all turns) | <800ms | Worst-case experience - the outliers that generate complaints |
| Latency drift per turn | <20ms | The cumulative degradation rate - should be near-zero in a healthy system |
| P95 under 50× load | <900ms | Peak traffic behaviour - what Monday morning actually looks like |
Five practical fixes for cumulative latency
Cap the conversation history sent to the LLM at a fixed token count - typically 2,000 to 4,000 tokens depending on your model. When the conversation exceeds this limit, drop the earliest turns rather than the most recent. The LLM retains recent context (which is what matters for the current response) while the input size stays bounded. This is the single most effective fix for context-driven latency growth and can reduce late-conversation LLM latency by 30 to 60 percent.
Identify the three to five external API calls your Voice AI makes most frequently and cache their responses with a short TTL - typically 30 to 120 seconds. A caller asking for their account balance twice in one call should not trigger two API calls to your banking system. Cache hits eliminate the entire external API latency for that turn, often saving 200 to 400 milliseconds on affected turns.
Generic STT models are optimised for general vocabulary. Domain-specific models - fine-tuned on your industry's terminology - are both faster and more accurate on your specific content. Deepgram's Nova models, for example, are consistently faster than Whisper on domain-specific content while matching or exceeding accuracy. Faster STT means less latency on every single turn, compounding positively across the entire conversation.
Some API calls cannot be cached and cannot be made faster. For these, build bridging phrases into your system prompt - responses the AI speaks immediately while the data loads. "Let me pull that up for you" or "Give me just a moment to check that" buys 1.5 to 2 seconds of perceived naturalness while the API responds. The caller hears an immediate acknowledgement rather than silence. This does not reduce actual latency but it transforms perceived latency dramatically - and perceived latency is what drives CSAT scores.
Rebuild your test suite around full conversation simulations - not individual utterances. Write ten representative conversation scripts covering your most common call types. Run each script end-to-end under load, measuring and logging latency on every turn. If turn ten is consistently slower than turn two, you have a cumulative latency problem to fix before go-live. This test approach would have caught the deployment failure I described above before a single real caller ever experienced it.
The question to ask before every go-live
Before any Voice AI system goes live, I now ask one question that did not exist in my UAT checklist for the first two years of my career: what is the P95 latency on turn ten of a conversation, under fifty concurrent calls?
If the answer is not available - because nobody has run that test - the system is not ready for production. Not because the technology is bad. Not because the vendor is wrong. But because the specific failure mode that affects real callers most severely is one that only appears when you stress-test the full conversation under realistic load. And that test takes about two hours to set up once you know what you are looking for.
End-to-end latency is the metric vendors measure. Cumulative latency across a full conversation under load is the metric that determines whether your callers come back. Make sure you are measuring the right one before you go live.
Want more honest writing on Voice AI in production?
I publish every week on Voice AI platforms, SIP telephony, and what it actually looks like to ship these systems for real clients - including the things that go wrong.
Follow with your Google account and get new posts in your Blogger reading list automatically.
Comments
Post a Comment