Turn taking in conversational AI is the logic that lets a voice agent decide whose turn it is to talk. In practice that means three decisions, made over and over during a call: has the user finished, should the agent start speaking, and should the agent stop because the user has cut in. This guide shows you how to build a small turn-taking controller, connect it to the rest of a voice pipeline, and tune it with metrics instead of guesswork.
ElevenLabs has published an explainer on turn taking in conversational AI. This tutorial takes a vendor-neutral, implementation-first approach. The code is original and works with any speech-to-text and text-to-speech provider.
What you will build
You will build a turn-taking controller: a small state machine that sits between your audio input and your agent’s response logic. It does three things:
- Endpointing. It decides when the user’s turn has ended, so the agent neither cuts in early nor leaves an awkward silence.
- Barge-in handling. It stops the agent’s speech when the user genuinely interrupts.
- Backchannel filtering. It ignores short acknowledgements such as “mm-hmm” or “right”, which mean the user is listening and not asking for the floor.
Who this guide is for
This guide is for developers and automation builders who already have, or plan to have, a basic voice agent: audio in, transcription, a language model and synthesised speech out. If you are still deciding whether a voice agent fits your business, start with an overview of AI automation for business processes and come back once the use case is clear.
Complexity: Intermediate. You should be comfortable reading Python and working with event callbacks.
Prerequisites
- Python 3 installed locally.
- A source of voice activity detection (VAD), meaning a component that labels each short audio frame as speech or non-speech. Many speech-to-text services expose this, and open-source VAD libraries exist as well.
- A streaming speech-to-text (ASR, automatic speech recognition) service that returns partial transcripts while the user is still talking.
- A text-to-speech service you can stop mid-sentence.
- A handful of recorded test conversations, with consent from everyone recorded.
Key terms in plain language
Endpointing is the decision that a speaker has finished their turn. Silence alone is a weak signal, because people pause while they think.
Barge-in is when the user starts talking while the agent is still speaking. A good agent stops quickly when the interruption is real and keeps going when it is not.
Backchannels are short listener signals such as “yeah” or “uh-huh”. Treating them as interruptions makes an agent feel jumpy.
Echo cancellation removes the agent’s own voice from the microphone signal. Without it, the agent can “hear” itself and interrupt its own sentence. For more definitions, see the AI glossary.
Architecture overview
The controller is deliberately small and makes no network calls. It receives one event per audio frame and triggers two callbacks: one to start the agent’s turn and one to stop it.
- Microphone audio passes through echo cancellation.
- Each frame goes to VAD and to streaming ASR.
- The controller receives
is_speechplus the latest partial transcript. - When it detects the end of the user’s turn, it calls
on_agent_turn, which runs your language model and starts text-to-speech. - While the agent speaks, the controller watches for sustained user speech and calls
on_barge_into stop playback and cancel generation.
The controller moves between four states: listening, user speaking, endpoint pending (silence detected, decision not yet made) and agent speaking.
Security and cost considerations before you start
Voice recordings and transcripts are personal data. Decide upfront how long you keep audio, who can listen to it, and how you inform callers. If your agent handles regulated conversations, review the data flows with an AI compliance audit before going live.
Turn taking also affects cost. Each false start triggers a language-model call and speech synthesis that nobody hears. Each missed barge-in keeps generating audio the user is talking over. Good turn taking therefore cuts wasted spend as well as improving the experience.
Step 1: Create the configuration
Keep every threshold outside the code so you can tune it per use case without redeploying. Create turn_config.json:
{
"silence_hold_ms": "<SILENCE_HOLD_MS>",
"incomplete_extra_ms": "<INCOMPLETE_EXTRA_MS>",
"barge_in_min_ms": "<BARGE_IN_MIN_MS>",
"backchannel_words": ["yeah", "mm-hmm", "uh-huh", "right", "ok"]
}
Replace each placeholder with an integer number of milliseconds:
silence_hold_msis how long the user must stay silent before the agent responds.incomplete_extra_msis the extra wait added when the transcript sounds unfinished.barge_in_min_msis how long the user must keep speaking over the agent before it stops.
Choose starting values by listening to your own recordings, not by copying someone else’s numbers. A support line for older customers and a fast internal assistant will need different settings.
Step 2: Write the turn-taking controller
Save the following as turn_taking.py. Each block is explained below the code.
# turn_taking.py - minimal, provider-neutral turn-taking controller
import json
import time
from enum import Enum
class State(Enum):
LISTENING = "listening" # waiting for the user
USER_SPEAKING = "user_speaking" # speech detected
ENDPOINT_PENDING = "endpoint_pending" # silence after speech, deciding
AGENT_SPEAKING = "agent_speaking" # text-to-speech is playing
INCOMPLETE_ENDINGS = {"and", "but", "so", "because", "um", "uh"}
class TurnController:
def __init__(self, config, on_agent_turn, on_barge_in):
# Thresholds come from config so they can be tuned without code changes.
self.silence_hold = config["silence_hold_ms"] / 1000
self.incomplete_extra = config["incomplete_extra_ms"] / 1000
self.barge_in_min = config["barge_in_min_ms"] / 1000
self.backchannels = set(config["backchannel_words"])
self.on_agent_turn = on_agent_turn # start LLM + text-to-speech
self.on_barge_in = on_barge_in # stop playback, cancel generation
self.state = State.LISTENING
self.silence_started = None
self.overlap_started = None
def looks_incomplete(self, text):
# Cheap semantic hint: a trailing filler or conjunction suggests more is coming.
words = text.lower().strip().rstrip(".,").split()
return bool(words) and words[-1] in INCOMPLETE_ENDINGS
def on_frame(self, is_speech, partial_text, now=None):
now = now if now is not None else time.monotonic()
if self.state == State.AGENT_SPEAKING:
self._check_barge_in(is_speech, partial_text, now)
return
if is_speech:
self.state = State.USER_SPEAKING
self.silence_started = None
return
if self.state == State.USER_SPEAKING:
self.state = State.ENDPOINT_PENDING
self.silence_started = now
if self.state == State.ENDPOINT_PENDING:
hold = self.silence_hold
if self.looks_incomplete(partial_text):
hold += self.incomplete_extra
if now - self.silence_started >= hold:
self.state = State.AGENT_SPEAKING
self.on_agent_turn(partial_text)
def _check_barge_in(self, is_speech, partial_text, now):
if not is_speech:
self.overlap_started = None
return
if self.overlap_started is None:
self.overlap_started = now
words = set(partial_text.lower().split())
only_backchannel = bool(words) and words <= self.backchannels
if now - self.overlap_started >= self.barge_in_min and not only_backchannel:
self.on_barge_in()
self.state = State.USER_SPEAKING
self.overlap_started = None
def agent_finished(self):
# Call this when playback ends without an interruption.
self.state = State.LISTENING
def load_config(path="turn_config.json"):
with open(path, encoding="utf-8") as f:
return json.load(f)
How the controller works
- The state enum makes every decision explicit. When something goes wrong in production, the logged state tells you which rule fired.
looks_incompleteadds a light semantic layer to endpointing. If the last word is a conjunction or a filler, the user is probably mid-thought, so the controller waits a little longer. You can later replace this with a classifier or a language-model check.on_frameis called once per audio frame. Speech resets the silence timer. Silence after speech starts the endpoint timer, and the agent only takes the floor once the hold time has passed._check_barge_inruns only while the agent is talking. It requires sustained speech and skips utterances made up entirely of backchannel words. A cough or a quick “yeah” therefore does not stop the agent.agent_finishedreturns control to the user when playback ends normally.
Step 3: Connect the callbacks
Your on_agent_turn callback should send the final transcript to your language model and stream the reply into text-to-speech. Your on_barge_in callback must do two things at once: stop audio playback immediately, and cancel any language-model request or synthesis still in progress. If you only stop playback, you keep paying for output nobody hears.
Also store what the agent had actually said before the interruption. Passing that partial reply into the next prompt helps the agent avoid repeating itself or acting as if the user heard the whole answer.
Step 4: Test with a simulated conversation
Before using live audio, feed the controller a scripted sequence of frames. That gives you repeatable tests:
# simulate.py - replay scripted frames through the controller
from turn_taking import TurnController, load_config
def agent_turn(text):
print(f"AGENT TURN after: {text!r}")
def barge_in():
print("BARGE-IN: stopping playback and cancelling generation")
ctrl = TurnController(load_config(), agent_turn, barge_in)
# Each tuple: (seconds since start, is_speech, partial transcript)
# Replace with frames exported from your own consented test recordings.
frames = [] # e.g. loaded from a CSV produced by your VAD + ASR
for t, speech, text in frames:
ctrl.on_frame(speech, text, now=t)
print(t, ctrl.state.value)
Expected output
For a recording where the user asks a question, pauses, and then interrupts the agent’s answer, the log should show this sequence: user_speaking, then endpoint_pending, then one AGENT TURN line, then agent_speaking, then a BARGE-IN line, and finally user_speaking again. A user saying “yeah” while the agent talks should leave the state at agent_speaking.
How to measure turn-taking quality
Tuning without metrics turns into an argument about how the agent “feels”. Log a timestamped event for every state change and compute these figures across your test set:
- Response gap: the time between the end of user speech and the start of agent audio. Too long feels sluggish. Too short usually comes with false starts.
- False interruption rate: how often the agent started talking while the user was still mid-thought. Review the transcripts to label these.
- Missed barge-in rate: how often the user clearly tried to interrupt and the agent kept talking.
- False barge-in rate: how often the agent stopped for a backchannel, a cough or background noise.
- Overlap duration: how long both parties spoke at the same time before the agent yielded.
- Wasted generation: language-model and speech output that was cancelled before being heard. This links turn-taking quality directly to cost.
Tuning the thresholds
Change one parameter at a time and re-run the same recordings:
- Many false interruptions: raise
silence_hold_msorincomplete_extra_ms, or extend the list of incomplete endings. - Agent feels slow: lower
silence_hold_msand rely more on the semantic check. - Agent ignores real interruptions: lower
barge_in_min_ms. - Agent stops for every “mm-hmm”: raise
barge_in_min_msor add words tobackchannel_words.
Validation checklist
- The agent never responds while the user is still speaking in your test recordings.
- Backchannel words do not stop agent playback.
- A sustained interruption stops audio and cancels generation.
- The agent does not interrupt itself when the microphone picks up its own voice.
- Every state change is logged with a timestamp.
Common errors and troubleshooting
- The agent interrupts itself. Echo cancellation is missing or misconfigured, so VAD detects the agent’s own voice as user speech. Fix this at the audio layer, not in the controller.
- Barge-in stops the audio but the agent keeps “thinking”. The callback stops playback but does not cancel the language-model stream. Cancel both.
- Endpointing fires on noisy lines. Background noise is read as speech or breaks up silence. Tune or replace the VAD before changing turn thresholds.
- The agent repeats itself after an interruption. It was not told which part of its reply had been spoken. Pass the delivered text into the next turn.
Cost controls
- Cancel generation and synthesis on every barge-in.
- Stream the language-model output into text-to-speech so you are not paying to generate long replies that may be interrupted.
- Track wasted generation as a metric and alert when it rises.
Security checklist
- Inform callers that they are speaking with an AI agent and whether the call is recorded.
- Keep API keys in environment variables, for example
<YOUR_API_KEY>, never in config files committed to version control. - Limit who can access raw audio and transcripts, and set a retention period.
- Use synthetic or consented recordings for testing, with personal details removed.
Production-readiness checklist
- Thresholds are versioned, and each change is tested against a fixed recording set.
- Metrics are tracked per use case, since one global setting rarely fits every flow.
- A human handover path exists for calls where the agent repeatedly loses the thread.
- Sensitive actions triggered by the agent, such as payments or account changes, still require explicit confirmation.
Alternative implementation
Instead of a hand-built controller, you can use the turn-taking features built into a managed conversational AI platform. That shortens setup but gives you less visibility into why a decision was made. Whichever option you pick, the measurement approach above still applies. For a wider view of how voice agents fit into a business stack, see this guide to custom AI agents for growth teams.
How to extend it
- Replace the keyword-based completeness check with a small classifier trained on your own labelled transcripts.
- Use prosody, meaning pitch and rhythm, as an extra end-of-turn signal.
- Adjust thresholds dynamically, for example waiting longer after the agent asks an open question.
- Feed the logged metrics into a dashboard so product and engineering teams review the same numbers.
Put this AI capability into practice
Understanding the technology is only the first step. AiLunaPro can help your team turn this capability into a controlled operational workflow.
With AiLunaPro Custom AI Agents and Automation, you can:
- Help design voice agents whose turn-taking rules match a specific customer journey
- Set up measurement of response gaps, interruptions and wasted generation so tuning decisions rest on data
- Review data handling for recorded calls and transcripts before a voice agent goes live
- Integrate voice agents with existing automation workflows and human handover paths
