How to Build a Claude-Powered Voice Assistant

Building a Claude-powered voice assistant means wiring together three distinct systems — speech-to-text (STT), a language model that reasons and responds (Claude), and text-to-speech (TTS) — plus the glue code that makes the round trip feel conversational instead of laggy. None of these pieces is exotic on its own; the real engineering work is in latency, interruption handling, and keeping Claude's context coherent across turns.

This guide walks through the architecture end to end: capturing audio, transcribing it, sending it to Claude with streaming enabled, converting the response back to speech, and adding tool use so the assistant can actually do things (check a calendar, look up order status, control a device) rather than just talk.

Architecture overview

A voice assistant pipeline has four stages that repeat every turn:

1. Capture — record audio from a mic (browser MediaRecorder, a mobile SDK, or a phone call via Twilio).
2. Transcribe — convert audio to text with an STT engine (Whisper, Deepgram, Google STT, or a provider's realtime transcription API).
3. Reason and respond — send the transcript to Claude, optionally with tool definitions, and get back text (streamed, ideally).
4. Speak — convert Claude's text back to audio with a TTS engine (ElevenLabs, Amazon Polly, Google TTS) and play it.

The STT and TTS legs are provider-specific and well documented elsewhere, so this article focuses on the middle step — getting Claude to behave well in a voice loop — and on how to structure the request/response cycle so the whole thing feels fast.

Step 1: Transcribe speech to text

Whatever STT engine you pick, you want partial/interim transcripts if the provider supports them, because waiting for a full utterance before doing anything adds dead air. A common pattern:

// Pseudocode: STT engine emits partial and final transcripts
sttStream.on("transcript", (event) => {
  if (event.isFinal) {
    sendToClaude(event.text);
  }
});

Only send the final transcript to Claude — sending partials wastes tokens and produces half-formed responses.

Step 2: Send the transcript to Claude

This is where a lot of the pipeline's perceived latency lives, so streaming matters. Instead of waiting for Claude to finish composing a full reply, stream tokens as they arrive and start TTS on sentence boundaries.

Using SubToAPI's streaming endpoint (your existing Claude access exposed as an HTTPS API with a sub_live_... key, see /docs/quickstart):

curl -N https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-20250514",
    "max_tokens": 400,
    "stream": true,
    "system": "You are a concise voice assistant. Keep replies under 3 sentences unless asked for detail.",
    "messages": [
      {"role": "user", "content": "What is on my calendar tomorrow?"}
    ]
  }'

The system prompt matters more for voice than for chat: tell Claude explicitly to avoid markdown, bullet lists, and long paragraphs, since none of that reads naturally out loud. Also cap max_tokens lower than you would for a text UI — a 600-word answer is fine on screen, terrible spoken aloud.

Full streaming details, including how to parse the SSE chunks on the client, are in /docs/streaming.

Step 3: Feed streamed text into TTS incrementally

Buffer tokens until you hit a sentence-ending punctuation mark, then fire that chunk to the TTS engine while Claude keeps generating the next sentence. This overlaps generation and speech synthesis instead of running them sequentially:

let buffer = "";

for await (const chunk of claudeStream) {
  buffer += chunk.text;
  const match = buffer.match(/^(.*?[.!?])\s(.*)$/s);
  if (match) {
    speak(match[1]);       // send sentence to TTS now
    buffer = match[2];      // keep the remainder
  }
}
if (buffer.trim()) speak(buffer);

This one change — speaking sentence-by-sentence instead of waiting for the full reply — is usually the single biggest latency win in a voice assistant.

Step 4: Give the assistant things to do with tool use

A voice assistant that can only talk is a novelty. The useful version can check a calendar, create a reminder, look up an order, or query an internal API. Claude's tool use (function calling) lets you define these actions as JSON schemas; Claude decides when to call them and with what arguments.

{
  "name": "get_calendar_events",
  "description": "Fetch calendar events for a given date",
  "input_schema": {
    "type": "object",
    "properties": {
      "date": { "type": "string", "description": "ISO 8601 date" }
    },
    "required": ["date"]
  }
}

When Claude returns a tool_use block instead of plain text, your server executes the actual function (hit the calendar API), sends the result back in a follow-up message, and Claude produces the final spoken answer. Because this adds a round trip, keep tool handlers fast — under 500ms if you can — or the assistant will feel like it's thinking too hard. Full examples of defining and handling tools are in /docs/tools.

Step 5: Handle interruptions and barge-in

Real conversations involve people talking over each other. If your assistant is mid-sentence and the user starts speaking, you need to:

• Stop audio playback immediately
• Cancel the in-flight Claude stream (close the connection)
• Treat the new audio as a fresh turn, not a continuation

Skipping this makes the assistant feel robotic and unresponsive compared to a human conversation partner — it's a small amount of extra state management that has an outsized effect on perceived quality.

Managing keys and cost across a voice product

If you're shipping this as a product rather than a demo, you'll want per-feature or per-customer API keys so you can see which part of the pipeline is burning tokens — voice sessions tend to generate a lot more round trips than a typical chat UI because every turn is short and frequent. SubToAPI issues scoped sub_live_... keys per environment and reports usage metadata per request, which makes it straightforward to track spend without building that instrumentation yourself. Plans start at €9/month for solo projects, with team seats at €19 and €49 for larger usage — see /pricing. You can start building against the quickstart with a free trial at /signup.

Testing before you ship

Before putting this in front of real users, test with:

• Background noise and overlapping speech
• Non-English accents if your STT doesn't lock to one locale
• Long silences (does the pipeline time out gracefully?)
• Rapid-fire follow-up questions (does context stay coherent across turns?)

A voice assistant that works in a quiet demo often breaks in a noisy kitchen or car — budget real testing time for that gap.

questions

Do I need a special Claude model for voice, or does the standard API work? The standard Claude messages API works fine — there's no separate "voice" model. The adaptation happens in your system prompt (short, non-markdown replies) and in how you stream and chunk the output, not in the model itself.

How do I keep latency low enough for a natural conversation? Stream Claude's response and start text-to-speech on sentence boundaries instead of waiting for the full reply, use a fast STT engine with interim results, and keep max_tokens low so Claude doesn't over-generate.

Can the assistant take actions, like booking a meeting or checking an order? Yes, via tool use (function calling). You define the available actions as JSON schemas, Claude requests a tool call with structured arguments, your server executes it, and the result feeds back into the conversation — see /docs/tools for implementation details.