NEW AI Audit and Voice AI now sit alongside the platform — one login, one system. Explore Voice AI →

Turn Taking in Conversational AI: How to Build and Tune Natural Voice Agent Conversations

Turn taking in conversational AI is the logic that lets a voice agent decide whose turn it is to talk. In practice that means three decisions, made over and over during a call: has the user finished, should the agent start speaking, and should the agent stop because the user has cut in. This guide shows you how to build a small turn-taking controller, connect it to the rest of a voice pipeline, and tune it with metrics instead of guesswork.

ElevenLabs has published an explainer on turn taking in conversational AI. This tutorial takes a vendor-neutral, implementation-first approach. The code is original and works with any speech-to-text and text-to-speech provider.

What you will build

You will build a turn-taking controller: a small state machine that sits between your audio input and your agent’s response logic. It does three things:

  • Endpointing. It decides when the user’s turn has ended, so the agent neither cuts in early nor leaves an awkward silence.
  • Barge-in handling. It stops the agent’s speech when the user genuinely interrupts.
  • Backchannel filtering. It ignores short acknowledgements such as “mm-hmm” or “right”, which mean the user is listening and not asking for the floor.

Who this guide is for

This guide is for developers and automation builders who already have, or plan to have, a basic voice agent: audio in, transcription, a language model and synthesised speech out. If you are still deciding whether a voice agent fits your business, start with an overview of AI automation for business processes and come back once the use case is clear.

Complexity: Intermediate. You should be comfortable reading Python and working with event callbacks.

Prerequisites

  • Python 3 installed locally.
  • A source of voice activity detection (VAD), meaning a component that labels each short audio frame as speech or non-speech. Many speech-to-text services expose this, and open-source VAD libraries exist as well.
  • A streaming speech-to-text (ASR, automatic speech recognition) service that returns partial transcripts while the user is still talking.
  • A text-to-speech service you can stop mid-sentence.
  • A handful of recorded test conversations, with consent from everyone recorded.

Key terms in plain language

Endpointing is the decision that a speaker has finished their turn. Silence alone is a weak signal, because people pause while they think.

Barge-in is when the user starts talking while the agent is still speaking. A good agent stops quickly when the interruption is real and keeps going when it is not.

Backchannels are short listener signals such as “yeah” or “uh-huh”. Treating them as interruptions makes an agent feel jumpy.

Echo cancellation removes the agent’s own voice from the microphone signal. Without it, the agent can “hear” itself and interrupt its own sentence. For more definitions, see the AI glossary.

Architecture overview

The controller is deliberately small and makes no network calls. It receives one event per audio frame and triggers two callbacks: one to start the agent’s turn and one to stop it.

  1. Microphone audio passes through echo cancellation.
  2. Each frame goes to VAD and to streaming ASR.
  3. The controller receives is_speech plus the latest partial transcript.
  4. When it detects the end of the user’s turn, it calls on_agent_turn, which runs your language model and starts text-to-speech.
  5. While the agent speaks, the controller watches for sustained user speech and calls on_barge_in to stop playback and cancel generation.

The controller moves between four states: listening, user speaking, endpoint pending (silence detected, decision not yet made) and agent speaking.

Security and cost considerations before you start

Voice recordings and transcripts are personal data. Decide upfront how long you keep audio, who can listen to it, and how you inform callers. If your agent handles regulated conversations, review the data flows with an AI compliance audit before going live.

Turn taking also affects cost. Each false start triggers a language-model call and speech synthesis that nobody hears. Each missed barge-in keeps generating audio the user is talking over. Good turn taking therefore cuts wasted spend as well as improving the experience.

Step 1: Create the configuration

Keep every threshold outside the code so you can tune it per use case without redeploying. Create turn_config.json:

{
  "silence_hold_ms": "<SILENCE_HOLD_MS>",
  "incomplete_extra_ms": "<INCOMPLETE_EXTRA_MS>",
  "barge_in_min_ms": "<BARGE_IN_MIN_MS>",
  "backchannel_words": ["yeah", "mm-hmm", "uh-huh", "right", "ok"]
}

Replace each placeholder with an integer number of milliseconds:

  • silence_hold_ms is how long the user must stay silent before the agent responds.
  • incomplete_extra_ms is the extra wait added when the transcript sounds unfinished.
  • barge_in_min_ms is how long the user must keep speaking over the agent before it stops.

Choose starting values by listening to your own recordings, not by copying someone else’s numbers. A support line for older customers and a fast internal assistant will need different settings.

Step 2: Write the turn-taking controller

Save the following as turn_taking.py. Each block is explained below the code.

# turn_taking.py - minimal, provider-neutral turn-taking controller
import json
import time
from enum import Enum

class State(Enum):
    LISTENING = "listening"                # waiting for the user
    USER_SPEAKING = "user_speaking"        # speech detected
    ENDPOINT_PENDING = "endpoint_pending"  # silence after speech, deciding
    AGENT_SPEAKING = "agent_speaking"      # text-to-speech is playing

INCOMPLETE_ENDINGS = {"and", "but", "so", "because", "um", "uh"}

class TurnController:
    def __init__(self, config, on_agent_turn, on_barge_in):
        # Thresholds come from config so they can be tuned without code changes.
        self.silence_hold = config["silence_hold_ms"] / 1000
        self.incomplete_extra = config["incomplete_extra_ms"] / 1000
        self.barge_in_min = config["barge_in_min_ms"] / 1000
        self.backchannels = set(config["backchannel_words"])
        self.on_agent_turn = on_agent_turn  # start LLM + text-to-speech
        self.on_barge_in = on_barge_in      # stop playback, cancel generation
        self.state = State.LISTENING
        self.silence_started = None
        self.overlap_started = None

    def looks_incomplete(self, text):
        # Cheap semantic hint: a trailing filler or conjunction suggests more is coming.
        words = text.lower().strip().rstrip(".,").split()
        return bool(words) and words[-1] in INCOMPLETE_ENDINGS

    def on_frame(self, is_speech, partial_text, now=None):
        now = now if now is not None else time.monotonic()
        if self.state == State.AGENT_SPEAKING:
            self._check_barge_in(is_speech, partial_text, now)
            return
        if is_speech:
            self.state = State.USER_SPEAKING
            self.silence_started = None
            return
        if self.state == State.USER_SPEAKING:
            self.state = State.ENDPOINT_PENDING
            self.silence_started = now
        if self.state == State.ENDPOINT_PENDING:
            hold = self.silence_hold
            if self.looks_incomplete(partial_text):
                hold += self.incomplete_extra
            if now - self.silence_started >= hold:
                self.state = State.AGENT_SPEAKING
                self.on_agent_turn(partial_text)

    def _check_barge_in(self, is_speech, partial_text, now):
        if not is_speech:
            self.overlap_started = None
            return
        if self.overlap_started is None:
            self.overlap_started = now
        words = set(partial_text.lower().split())
        only_backchannel = bool(words) and words <= self.backchannels
        if now - self.overlap_started >= self.barge_in_min and not only_backchannel:
            self.on_barge_in()
            self.state = State.USER_SPEAKING
            self.overlap_started = None

    def agent_finished(self):
        # Call this when playback ends without an interruption.
        self.state = State.LISTENING

def load_config(path="turn_config.json"):
    with open(path, encoding="utf-8") as f:
        return json.load(f)

How the controller works

  • The state enum makes every decision explicit. When something goes wrong in production, the logged state tells you which rule fired.
  • looks_incomplete adds a light semantic layer to endpointing. If the last word is a conjunction or a filler, the user is probably mid-thought, so the controller waits a little longer. You can later replace this with a classifier or a language-model check.
  • on_frame is called once per audio frame. Speech resets the silence timer. Silence after speech starts the endpoint timer, and the agent only takes the floor once the hold time has passed.
  • _check_barge_in runs only while the agent is talking. It requires sustained speech and skips utterances made up entirely of backchannel words. A cough or a quick “yeah” therefore does not stop the agent.
  • agent_finished returns control to the user when playback ends normally.

Step 3: Connect the callbacks

Your on_agent_turn callback should send the final transcript to your language model and stream the reply into text-to-speech. Your on_barge_in callback must do two things at once: stop audio playback immediately, and cancel any language-model request or synthesis still in progress. If you only stop playback, you keep paying for output nobody hears.

Also store what the agent had actually said before the interruption. Passing that partial reply into the next prompt helps the agent avoid repeating itself or acting as if the user heard the whole answer.

Step 4: Test with a simulated conversation

Before using live audio, feed the controller a scripted sequence of frames. That gives you repeatable tests:

# simulate.py - replay scripted frames through the controller
from turn_taking import TurnController, load_config

def agent_turn(text):
    print(f"AGENT TURN after: {text!r}")

def barge_in():
    print("BARGE-IN: stopping playback and cancelling generation")

ctrl = TurnController(load_config(), agent_turn, barge_in)

# Each tuple: (seconds since start, is_speech, partial transcript)
# Replace with frames exported from your own consented test recordings.
frames = []  # e.g. loaded from a CSV produced by your VAD + ASR

for t, speech, text in frames:
    ctrl.on_frame(speech, text, now=t)
    print(t, ctrl.state.value)

Expected output

For a recording where the user asks a question, pauses, and then interrupts the agent’s answer, the log should show this sequence: user_speaking, then endpoint_pending, then one AGENT TURN line, then agent_speaking, then a BARGE-IN line, and finally user_speaking again. A user saying “yeah” while the agent talks should leave the state at agent_speaking.

How to measure turn-taking quality

Tuning without metrics turns into an argument about how the agent “feels”. Log a timestamped event for every state change and compute these figures across your test set:

  • Response gap: the time between the end of user speech and the start of agent audio. Too long feels sluggish. Too short usually comes with false starts.
  • False interruption rate: how often the agent started talking while the user was still mid-thought. Review the transcripts to label these.
  • Missed barge-in rate: how often the user clearly tried to interrupt and the agent kept talking.
  • False barge-in rate: how often the agent stopped for a backchannel, a cough or background noise.
  • Overlap duration: how long both parties spoke at the same time before the agent yielded.
  • Wasted generation: language-model and speech output that was cancelled before being heard. This links turn-taking quality directly to cost.

Tuning the thresholds

Change one parameter at a time and re-run the same recordings:

  • Many false interruptions: raise silence_hold_ms or incomplete_extra_ms, or extend the list of incomplete endings.
  • Agent feels slow: lower silence_hold_ms and rely more on the semantic check.
  • Agent ignores real interruptions: lower barge_in_min_ms.
  • Agent stops for every “mm-hmm”: raise barge_in_min_ms or add words to backchannel_words.

Validation checklist

  • The agent never responds while the user is still speaking in your test recordings.
  • Backchannel words do not stop agent playback.
  • A sustained interruption stops audio and cancels generation.
  • The agent does not interrupt itself when the microphone picks up its own voice.
  • Every state change is logged with a timestamp.

Common errors and troubleshooting

  • The agent interrupts itself. Echo cancellation is missing or misconfigured, so VAD detects the agent’s own voice as user speech. Fix this at the audio layer, not in the controller.
  • Barge-in stops the audio but the agent keeps “thinking”. The callback stops playback but does not cancel the language-model stream. Cancel both.
  • Endpointing fires on noisy lines. Background noise is read as speech or breaks up silence. Tune or replace the VAD before changing turn thresholds.
  • The agent repeats itself after an interruption. It was not told which part of its reply had been spoken. Pass the delivered text into the next turn.

Cost controls

  • Cancel generation and synthesis on every barge-in.
  • Stream the language-model output into text-to-speech so you are not paying to generate long replies that may be interrupted.
  • Track wasted generation as a metric and alert when it rises.

Security checklist

  • Inform callers that they are speaking with an AI agent and whether the call is recorded.
  • Keep API keys in environment variables, for example <YOUR_API_KEY>, never in config files committed to version control.
  • Limit who can access raw audio and transcripts, and set a retention period.
  • Use synthetic or consented recordings for testing, with personal details removed.

Production-readiness checklist

  • Thresholds are versioned, and each change is tested against a fixed recording set.
  • Metrics are tracked per use case, since one global setting rarely fits every flow.
  • A human handover path exists for calls where the agent repeatedly loses the thread.
  • Sensitive actions triggered by the agent, such as payments or account changes, still require explicit confirmation.

Alternative implementation

Instead of a hand-built controller, you can use the turn-taking features built into a managed conversational AI platform. That shortens setup but gives you less visibility into why a decision was made. Whichever option you pick, the measurement approach above still applies. For a wider view of how voice agents fit into a business stack, see this guide to custom AI agents for growth teams.

How to extend it

  • Replace the keyword-based completeness check with a small classifier trained on your own labelled transcripts.
  • Use prosody, meaning pitch and rhythm, as an extra end-of-turn signal.
  • Adjust thresholds dynamically, for example waiting longer after the agent asks an open question.
  • Feed the logged metrics into a dashboard so product and engineering teams review the same numbers.

Put this AI capability into practice

Understanding the technology is only the first step. AiLunaPro can help your team turn this capability into a controlled operational workflow.

With AiLunaPro Custom AI Agents and Automation, you can:

  • Help design voice agents whose turn-taking rules match a specific customer journey
  • Set up measurement of response gaps, interruptions and wasted generation so tuning decisions rest on data
  • Review data handling for recorded calls and transcripts before a voice agent goes live
  • Integrate voice agents with existing automation workflows and human handover paths

Plan your AI automation with AiLunaPro

Sources

Verified on DANG! Fazier badge