How to Build a Voice Assistant with Claude

Building a voice assistant with Claude means wiring together three separate systems — speech-to-text (STT), Claude for language understanding and reasoning, and text-to-speech (TTS) — into a pipeline that feels fast and natural to talk to. Claude itself doesn't hear or speak; it only takes text in and returns text out. Your job is to get audio into text quickly, give Claude the right context and tools, and stream its response back out as speech with minimal delay.

This guide walks through the actual architecture: capturing audio, transcribing it, structuring the conversation for Claude, streaming the reply, handling actions through tool use, and converting the response to audio — plus the latency tricks that separate a usable voice assistant from a frustrating one.

The core pipeline

A voice assistant is a loop with four stages:

1. Capture — record audio from a mic or phone call until the user stops talking (voice activity detection).
2. Transcribe — send the audio to an STT engine (Whisper, Deepgram, AssemblyAI, or a platform's native STT) and get text back.
3. Reason — send that text to Claude along with conversation history and any tools it might need, and get a text response, ideally streamed.
4. Speak — pipe Claude's response into a TTS engine and play the audio, ideally starting playback before the full response has finished generating.

Each stage adds latency, so the real engineering work is overlapping these stages rather than running them strictly in sequence.

Setting up the Claude layer

Whether you call Claude directly or through a proxy, the integration pattern is the same: maintain a running message history and send it with each new user turn.

const messages = [
  { role: "system", content: "You are a concise voice assistant. Keep answers short and conversational — this will be read aloud." }
];

async function handleUserTurn(transcript) {
  messages.push({ role: "user", content: transcript });

  const response = await fetch("https://api.subtoapi.app/v1/messages", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      model: "claude-sonnet-4",
      max_tokens: 300,
      messages
    })
  });

  const data = await response.json();
  const reply = data.content[0].text;
  messages.push({ role: "assistant", content: reply });
  return reply;
}

A few things that matter specifically for voice:

• Keep the system prompt explicit about brevity. Claude's default writing style is good for reading, not listening. Tell it to avoid markdown, bullet points, and long lists, and to answer in one or two sentences unless asked for detail.
• Cap max_tokens aggressively. Long responses mean long waits before TTS can start, and nobody wants a two-minute monologue from their smart speaker.
• Trim history. Voice conversations drift; keep only the last several turns plus a running summary if the session gets long, to control both latency and cost.

If you're using SubToAPI as your API layer, this is standard chat completion behavior — see /docs/messages for the full request/response shape, and /docs/quickstart to get an application key set up in minutes.

Streaming for low latency

The single biggest lever for making a voice assistant feel responsive is streaming the model output and feeding it into TTS sentence-by-sentence, rather than waiting for the full response.

const response = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    model: "claude-sonnet-4",
    max_tokens: 300,
    stream: true,
    messages
  })
});

let buffer = "";
for await (const chunk of parseSSE(response.body)) {
  buffer += chunk.text;
  const sentenceEnd = buffer.match(/[.!?]\s/);
  if (sentenceEnd) {
    const sentence = buffer.slice(0, sentenceEnd.index + 1);
    buffer = buffer.slice(sentenceEnd.index + 1);
    speak(sentence); // send this chunk to TTS immediately
  }
}
if (buffer.trim()) speak(buffer);

This cuts perceived latency dramatically: instead of waiting for the entire answer to generate before any audio plays, the user hears the first sentence while Claude is still writing the rest. See /docs/streaming for the full event format.

Giving the assistant actions, not just answers

A voice assistant that can only talk is a chatbot with a microphone. Real assistants need to check a calendar, set a reminder, look up an order, or control a device. This is where tool use comes in — you define functions Claude can call, and Claude decides when to call them based on the conversation.

{
  "tools": [
    {
      "name": "set_reminder",
      "description": "Schedule a reminder for the user",
      "input_schema": {
        "type": "object",
        "properties": {
          "text": { "type": "string" },
          "when": { "type": "string", "description": "ISO 8601 datetime" }
        },
        "required": ["text", "when"]
      }
    }
  ]
}

When Claude returns a tool_use block instead of plain text, your backend executes the actual function (write to a database, call a calendar API, trigger a smart home command) and sends the result back in the next message so Claude can confirm it verbally — "Done, I'll remind you at 3pm." Full details and examples are in /docs/tools.

Handling interruptions and turn-taking

Voice conversations aren't strictly turn-based like text chat. Users interrupt, pause mid-sentence, and say "wait" or "stop" mid-response. Practical handling:

• Run voice activity detection on the mic continuously, even while TTS is playing, so you can detect barge-in and cut playback.
• When interrupted, discard the in-flight Claude stream (or let it finish server-side but stop playing audio) and treat the new utterance as a fresh turn.
• Add a short silence threshold (300–600ms) before treating a pause as "end of turn" so Claude isn't triggered mid-sentence.

Production considerations

Once the prototype works, the operational concerns are the same as any API-backed product: key management, rate limits, and cost visibility, especially if you're running this across many concurrent calls or devices.

If you're shipping to multiple devices, kiosks, or a team of developers building different assistant features, route Claude calls through a proxy that gives each integration its own API key rather than sharing one Anthropic account key across every service. SubToAPI issues separate sub_live_... keys per application with usage breakdown by key, which makes it easy to see which voice feature is driving token spend and to revoke a single key without taking down everything else. Plans start at Solo (€9) for a single integration, with Team and Scale tiers for multi-seat setups — see /pricing, and you can try it with a free trial at /signup.

FAQ

Does Claude have built-in speech recognition or voice output? No. Claude is text-in, text-out. You need a separate STT engine (Whisper, Deepgram, etc.) to transcribe audio into text before calling Claude, and a TTS engine (ElevenLabs, Google TTS, Amazon Polly) to convert Claude's text reply back into audio.

How do I reduce latency in a Claude-powered voice assistant? Stream the response and start TTS as soon as the first complete sentence arrives instead of waiting for the full reply. Also cap max_tokens, instruct Claude to answer briefly, and trim conversation history to keep each request fast.

Can a Claude voice assistant take actions, like setting reminders or checking data? Yes, through tool use. You define functions with input schemas, Claude emits a tool_use block when it decides an action is needed, your backend executes it, and the result is fed back so Claude can confirm the outcome in natural language.