Why Do AI Voice Agents Pause, Interrupt and Mishear You?

You ring a business, an AI picks up, you ask something simple, and there is a gap. Not a long one. Just long enough that you wonder whether it heard you. Then you start talking again at the exact moment it starts talking, and you both stop. If you have used an AI voice agent for more than five minutes, you know the feeling.

I get asked a version of this all the time, usually framed as "why can't it just answer like a person?" When I talk to someone, I hear a word and whatever I know about it is there within a second. My thinking and my speaking run together. An AI voice agent does not work that way, and voice AI latency is not a bug that someone forgot to fix. It comes from how these systems are built, and once you see the parts, you can tell a vendor's real claims from the brochure.

Here's what matters most:

• People do not answer fast because they think fast. They start planning the reply while the other person is still talking. Conversation research puts the average gap between turns at roughly a fifth of a second.
• Most AI voice agents work as a relay: listen, write down the words, think, then speak. Each handoff adds time, and a well-tuned system still lands somewhere around two thirds of a second to well over a second.
• The hardest job is deciding when you have finished talking. Wait too long and the agent feels slow. Wait too little and it cuts you off mid-sentence.
• Phone lines make everything worse. A demo in a browser hears far cleaner audio than a real call, and names, emails and numbers are where agents fail most.
• Newer models that hear and speak directly are not automatically faster or better. They trade speed and natural tone for control and reliability.

Why a person answers in a fifth of a second

The interesting fact is not that people are fast. It is that people are faster than they should be.

In a study across ten languages, Stivers and colleagues found that speakers everywhere avoid overlapping and keep the silence between turns short, with averages that sit within a narrow band across cultures. The Moshi research paper cites that work for an average gap of about 230 milliseconds. Now compare that with how long it takes to produce speech. According to the review by Levinson and Torreira, even a single word takes a speaker more than 600 milliseconds to plan and say.

Those two numbers do not add up unless something else is going on, and the something else is prediction. While you are still talking, I am already guessing where your sentence is heading, drafting my answer, and listening for the cues in your grammar and tone that say you are about to stop. When you stop, I am ready. My memory is not a search either. What I know about the topic is already active in the background while I listen.

That is the bar an AI voice agent is measured against, whether or not the caller knows it. Callers do not compare your agent to other software. They compare it to every phone call they have ever had.

Where the second goes inside an AI voice agent

Most voice agents in production today are built as a chain of separate pieces. Picture a dental practice's agent, call her Ava, taking a call from a patient called Priya who says, "Hi, can I move my Thursday cleaning?"

1. Deciding Priya has finished. The system listens for silence. LiveKit, an open-source toolkit many voice agents are built on, waits 500 milliseconds by default before treating a pause as the end of a turn.
2. Turning speech into text. The speech recognition step finalises the words.
3. Thinking. The language model reads the conversation so far plus Ava's instructions, decides what to say, and produces the first word.
4. Turning text back into speech. A voice engine produces the first audio.
5. Getting it back to Priya. The phone network and the internet carry the sound back.

Hamming, a company that tests voice agents, publishes typical ranges for each step. In their guide, speech recognition takes about 100 to 400 milliseconds, the language model takes 300 to 1,000 milliseconds to its first word, speech generation takes 150 to 500, and the network adds 100 to 300. They put a typical total at 1 to 3.2 seconds and a well-tuned one at about 0.67 to 1.45 seconds. Their measured production figures are worse than the tuned budget: a median of 1.4 to 1.7 seconds, with the slowest 5 percent of turns running past 4 seconds. They sell testing, so read those as an industry view rather than an audit, but the shape matches what anyone who has built one of these sees.

The thinking step deserves one more sentence because it is where buyers can actually make a difference. Before the model says its first word, it has to read everything in front of it: the business instructions, the knowledge it was given, and the whole conversation. Engineers call this the prefill stage, and the length of what the model has to read is the main driver of how long you wait for that first word, as Redis explains. A voice agent stuffed with forty pages of policy will think more slowly than one given the two pages it needs for the call. When a vendor tells you to "just upload everything", that is a speed cost you will hear on every turn.

Add a lookup on top and the budget grows again. When Ava checks the calendar for Thursday, that is another trip to another system. One latency guide from Twig notes that a single 300 millisecond tool call can push the response past 1.1 seconds. People do this too. A good receptionist says "let me check that for you" and nobody minds. The trouble is that fillers have their own failure mode. In Full-Duplex-Bench-v3, a research benchmark for voice agents that use tools, one model said a filler like "let me check on that" before nearly every lookup, and those fillers overlapped the caller's speech 47.9 percent of the time.

The hardest part is knowing when you have finished

If I had to pick the single most underrated problem in voice agents, it is this one. Silence is a terrible signal. People pause in the middle of sentences all the time: to think, to breathe, to read something off a screen.

Take Priya again. Ava asks for her phone number, and Priya reads it the way most people do, in chunks: "07700... [pause]... 900... [pause]... 461." A system that treats every half-second of silence as the end of a turn jumps in after the first chunk. Now Priya is talking over Ava, Ava is talking over Priya, and the number is wrong. LiveKit names addresses, phone numbers and payment details as exactly the cases that make this hard. Their example of the underlying problem is a caller who says "I understand your point, but..." and trails off. A silence detector hears a finished turn. Any person knows there is more coming.

The fix the industry is moving to is to stop listening only for silence and start reading the words. LiveKit trained a small model that looks at what was said and judges whether the sentence sounds finished, then lengthens or shortens the wait accordingly. They report 85 percent fewer accidental interruptions compared with silence alone. Notice what that is. It is the machine doing a crude version of what your brain does: predicting the end of the turn from meaning, not waiting for the gap.

Then there are the interruptions that go the other way. People make small sounds while listening, "mm-hm", "right", "yeah", and those sounds mean "keep going". A voice agent that treats every sound as the caller taking the floor stops mid-sentence every time someone agrees with it. On speakerphone it can even hear its own voice coming back through the microphone and interrupt itself, which is why every guide on the subject, including Hamming's runbook, treats echo removal as a basic requirement rather than a refinement. The Moshi paper puts overlapping speech at around 10 to 20 percent of spoken time in normal conversation. A system designed for neat, alternating turns is designed for a conversation that does not exist.

Why the demo sounds better than the phone call

Most demos you will hear, including the one further down this page, run in a web browser. That is the best case. The browser carries rich, clean audio from a decent microphone.

A real phone call is a different instrument. Traditional phone audio is sampled at 8 kHz, which keeps only the lower slice of the sound range. Soniox, a speech recognition company, notes that this blurs the high-frequency detail that separates similar sounds, so "fifteen" and "sixteen" can collapse into each other. Add a caller on a mobile in a car with the window down and you can see why an agent that was flawless in the sales demo starts asking people to repeat themselves.

The research confirms where it breaks. Sierra's τ-Voice benchmark gave voice agents the same customer-service tasks as text agents. In the version of the paper I read, the best text agent with reasoning completed about 85 percent of tasks, while live voice agents managed roughly 26 to 38 percent once realistic noise, accents and interruptions were added. Their review of failures found transcription errors clustered in identity checks: agents often missed names and emails even when the caller spelled them letter by letter. Accents caused the largest single drop in one of their tests.

The practical lesson for a buyer is simple. Never judge a voice agent from a browser demo alone. Ring it from a mobile, in a noisy room, and spell an awkward email address at it.

What fixes it, and what does not

The good news is that most of the delay is engineering, and engineering improves. The fixes that work today:

• Run the steps at the same time instead of one after another. The voice can start speaking the first sentence while the model is still writing the second.
• Start thinking before the caller stops. Some systems pass the half-finished transcript to the model early so the answer is ready when the turn ends. LiveKit's guide, as summarised by Future AGI, puts the saving at 150 to 350 milliseconds per turn. The risk is that the early guess was wrong and has to be thrown away.
• Read meaning, not just silence, as described above.
• Keep the instructions short and the lookups fast. This one is on whoever designs the agent, not the model vendor.

The idea that gets the most hype is the "speech-to-speech" model, a single model that hears audio and speaks audio with no text in the middle. It keeps tone of voice, which the relay throws away, and it can sound more natural. But it is not automatically faster. One 2026 comparison published by Gradium, a vendor with its own interest, measured OpenAI's realtime model at 0.90 seconds, Google's at 1.14 seconds and a well-tuned relay at 0.755 seconds. The same source and Deepgram both point out the real trade: with no text checkpoint in the middle, it is harder to see why a call went wrong, harder to make the agent say exact approved wording, and harder to make it use your systems reliably.

Further out, research models such as Kyutai's Moshi listen and speak at the same moment, the way people do. Its paper reports a theoretical delay of 160 milliseconds and about 200 milliseconds in practice on one type of graphics card. That is human territory. I have not seen evidence that this kind of model can reliably book an appointment into a real calendar yet, so for a business it is a direction, not a product.

My own view on where this lands: simple turns will reach human-feeling speed within a year or two, and the pause will stop being the thing people notice. Turns that need a lookup will stay slower, because a calendar or a records system takes the time it takes. Bad phone audio, accents and people talking over each other will improve slowly, because those are not speed problems. They are problems of hearing and judgement.

If you want to feel the difference rather than read about it, the demo below is one of the voice agents we run, set up as a receptionist for a demo dental practice. Interrupt it. Say "mm-hm" while it talks. Read it a phone number in chunks.

When you are evaluating a vendor, ask for three numbers rather than one: the median response time, the time for the slowest 5 percent of turns, and the time for turns that include a calendar or records lookup. A vendor that only quotes an average is quoting the call you will never complain about. If you want the wider picture of where these systems earn their keep, read our guide to AI voice agents for business, and if the latency question is part of a bigger build, it is the first thing we measure when we build AI voice agents for clients.

FAQ

Why does my AI voice agent have a delay before it answers?

Because most voice agents are a chain of steps: detecting that you stopped, turning speech into text, the model reading the conversation and deciding what to say, and turning that back into speech. Each step adds time, and a waiting period to confirm you have finished is usually the biggest single piece. A well-tuned system lands around two thirds of a second to just over a second, and a lookup in a calendar or records system adds more.

Why does the AI cut me off when I pause?

Most systems decide you have finished speaking when they hear a short silence, often around half a second. People pause mid-sentence, especially when reading out a number or an address, so the agent jumps in too early. Better systems also read the words to judge whether the sentence sounds complete, which cuts these interruptions sharply, but no setting is right for every caller.

Is speech-to-speech AI better than a traditional voice agent?

It can sound more natural because it keeps tone of voice, and it removes some of the handoffs. It is not automatically faster, and it is harder to control: exact wording, reliable use of your business systems, and finding out why a call went wrong are all easier when there is a text record in the middle. For calls that must do something precise, the traditional relay is still the safer choice in my view.

What is a good response time for a voice AI agent?

Human turn gaps average roughly a fifth of a second, so anything an AI does will be slower. Industry guides aim for under about 800 milliseconds from the end of the caller's speech to the first sound back. Judge it on the slowest turns and on turns that include lookups, not on the average.

Why do AI voice agents struggle with names and email addresses?

Phone audio is narrow and blurs similar sounds, and names and emails have no context to help the system guess. In Sierra's τ-Voice research, transcription failures clustered in exactly this step, even when callers spelled the words out. Good designs avoid relying on it: they read details back, or text the caller a link to type them in.