Same LLM, twice the success rate

This is part of a series of findings from VAmoS Pro Bench, our voice-agent benchmark (here is the launch post). Six of its fourteen stacks run the same language model, OpenAI's gpt-4.1-mini, with the same prompt and the same tools. They complete between 17.3% and 36.4% of calls. Three of them are built from the exact same speech-to-text, LLM and text-to-speech models, and still range from 17.3% to 29.4%.

Task completion on VAmoS Pro Bench: the same 100 tasks × 3 runs per stack, 210 to 289 counted calls each after removing calls the benchmark itself spoiled. Lines are 95% Wilson intervals.

Six stacks, one model

Every stack is a thin transport around shared code: the same system prompt, the same sixteen tools, the same verification gate. What changes is who hears the caller, who decides when the caller has finished, and who speaks the reply.

Where they split: letting the caller finish

The clearest difference is how often the agent starts talking while the caller is still speaking. On the stacks at the top, it almost never does. On Pipecat, it happens on half of everything the caller says.

Talked over: share of the caller's utterances during which the agent started speaking. Verified on first try: of calls where the agent tried to verify the caller, the share where the first details it submitted were accepted. Cost is the mean per call at each vendor's published prices.

One place it matters is the start of every call, when the caller reads out an account number, a name and a postal code. Here is the start of one Pipecat call, as the agent heard it. The caller reads the account number in two breaths, and the agent answers each piece:

Pipecat and LiveKit hear the caller through the same Deepgram model, yet the first details Pipecat submits are accepted on 35% of calls that reach verification, against 77% for LiveKit. Pipecat also replies fastest of the six, and LiveKit slowest.

Talk-over explains only part of the gap

In our Jev turn-detection test, we cut Pipecat's talk-over from 52% of caller turns to 11%, and completion barely moved: 50 of 300 calls before, 53 after.

The stacks also differ in what they get done once the caller is verified. ElevenLabs makes the required account changes on 60.0% of graded calls, Pipecat on 32.0%.

Noise separates them further. With a television playing, ElevenLabs completes 15.9% and Gradium 10.1%; LiveKit, Vapi and Deepgram complete none. On a clean line, LiveKit leads the six at 43.2%.

What it took to build each one

The implementations are public on GitHub. Most needed a few defaults changed before they could hold a utility billing call:

• Pipecat: a caller turn may start only on a final transcript. With interim transcripts or background babble starting turns, some calls fell silent until they timed out.
• LiveKit: the tool-step limit raised from 3 to 10, since verifying and then looking up an account is already four steps.
• Vapi: the reply limit raised from 250 tokens, which cut off longer tool results, and its default noise reduction turned off.
• Gradium: a guard so a tool call re-issued while the first is still running returns the earlier result instead of running twice.
• ElevenLabs and Deepgram: neither exposes a silence threshold, so their turn-taking is the vendor's own.

What this does not show

These are our bridge configurations, not each vendor's best. We pinned end-of-turn silence near 0.8 s where a stack allowed it, turned off Vapi's noise reduction, and set the LLM temperature to 0.3 on ElevenLabs and Vapi while the other four use the default. gradbot wraps the shared prompt in its own scaffold. Different settings could move any of these numbers.

The intervals for Gradium, Vapi, Deepgram and Pipecat overlap, so those four are not cleanly separated. The gaps between ElevenLabs or LiveKit and Pipecat are large enough to rule out chance (p < 0.001). Vapi's figure rests on 210 calls, because 90 were removed for faults on our side, most often our simulated caller hanging up while the agent was looking something up. Costs leave out the servers that run the open-source frameworks.

The differences line up with turn-taking, but this run does not isolate any single cause.

Test your own stack

Picking a model is one decision out of many in a voice agent. Veris runs your whole stack against simulated callers and twins of your systems, so you can compare configurations on the calls you actually get.