How to Build a Voice Assistant with Claude API

Building a voice assistant with the Claude API means chaining three components: speech-to-text (STT) to capture what the user says, Claude to understand intent and generate a response, and text-to-speech (TTS) to speak that response back. Claude itself doesn't process audio directly — it's a text model — so the voice layer is your responsibility, but the reasoning, memory, and tool-calling logic live entirely in the Claude API calls in between.

This guide walks through the architecture, the code for wiring the pieces together, and the specific things that make voice assistants feel fast or feel sluggish, since latency matters far more here than in a typical chat UI.

The core pipeline

A voice assistant built on Claude follows this loop:

1. Capture audio from the microphone (browser MediaRecorder, a phone SDK, or a telephony provider).
2. Transcribe the audio to text with an STT service (Whisper, Deepgram, AssemblyAI, or a provider's streaming ASR).
3. Send the transcript to Claude along with conversation history and any tools the assistant can use (checking a calendar, looking up an order, controlling a device).
4. Stream Claude's response as it's generated rather than waiting for the full reply.
5. Convert the response to speech incrementally, so audio starts playing before the text finishes generating.
6. Play the audio and loop back to listening.

Steps 4 and 5 are where most of the perceived latency lives, and where streaming makes the biggest practical difference.

Setting up the Claude calls

Each turn of the conversation is a standard messages request. You maintain the conversation array yourself — Claude has no built-in memory between calls.

const messages = [
  { role: "user", content: transcript }
];

const response = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    model: "claude-sonnet-4-5",
    max_tokens: 512,
    system: "You are a concise voice assistant. Keep replies short and conversational, since they will be spoken aloud.",
    messages
  })
});

Two details matter specifically for voice:

• Keep max_tokens low and the system prompt explicit about brevity. A response written for a chat window reads awkwardly out loud — long lists, markdown formatting, and multi-paragraph explanations don't translate to speech. Tell Claude directly to answer in short, spoken-style sentences.
• Strip markdown before sending to TTS. Even with a good system prompt, Claude may occasionally output bullet points or bold text. Run a quick regex pass to remove *, #, and code fences before handing text to the speech engine.

If you're calling Claude through SubToAPI, this is a normal /v1/messages call using your sub_live_ key — the voice layer sits entirely outside the API call itself, so no special setup is needed beyond the usual authentication.

Streaming for lower perceived latency

Waiting for Claude to finish a full response before starting TTS adds a second or more of dead air, which feels broken in a voice interface. Instead, stream the response and start speaking as soon as you have a complete sentence.

const stream = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    model: "claude-sonnet-4-5",
    max_tokens: 512,
    stream: true,
    messages
  })
});

let buffer = "";
for await (const chunk of stream.body) {
  buffer += decodeChunk(chunk);
  const sentenceEnd = buffer.match(/[.!?]\s/);
  if (sentenceEnd) {
    const sentence = buffer.slice(0, sentenceEnd.index + 1);
    buffer = buffer.slice(sentenceEnd.index + 1);
    sendToTTS(sentence.trim());
  }
}
if (buffer.trim()) sendToTTS(buffer.trim());

This sentence-chunking approach means the assistant starts speaking after the first clause instead of after the entire reply. Details on the event format are in the streaming docs — the pattern is standard Claude streaming, just consumed by a sentence buffer instead of a UI.

Adding actions with tool use

A voice assistant that can only talk is limited. Most useful assistants need to check a calendar, place an order, or query a database mid-conversation. Claude's tool use lets you define functions it can call, and it will pause generation to request one when needed.

const tools = [
  {
    name: "get_weather",
    description: "Get current weather for a city",
    input_schema: {
      type: "object",
      properties: { city: { type: "string" } },
      required: ["city"]
    }
  }
];

When Claude returns a tool_use block instead of plain text, you run the function, send the result back as a tool_result, and Claude produces the final spoken answer incorporating it. In a voice flow, this typically means holding TTS until the tool round-trip completes, or playing a short filler ("let me check that") while it runs. See the tool use docs for the full request/response shape.

Handling interruptions and turn-taking

Voice conversations aren't strictly turn-based — users interrupt. A production assistant needs:

• Voice activity detection (VAD) to know when the user has stopped speaking, not just a fixed silence timeout.
• Barge-in handling: if the user starts talking while TTS is playing, stop playback immediately and start a new STT pass rather than queuing.
• A short-term buffer of the last few turns (not the entire history) sent to Claude, both to control token usage and to keep response latency predictable as conversations get long.

Choosing where to run this

For a browser-based prototype, you can run STT and TTS client-side (Web Speech API) and call Claude from a lightweight backend that holds your API key. For telephony (Twilio, etc.) or always-on devices, you'll want a server process managing the audio streams and making the Claude calls, since API keys should never sit in client-side or embedded device code.

If you're managing this across a team — multiple developers building against the same Claude access, with separate keys for a prototype, a staging bot, and a production line — a layer like SubToAPI gives each environment its own sub_live_ key with independent usage tracking, so you can see exactly which part of the voice pipeline is consuming tokens without digging through a shared account.

Getting started

The fastest path to a working prototype: pick one STT provider with streaming support, wire it to a Claude /v1/messages call with a brief system prompt, and pipe the response to any TTS API. Get that loop working end-to-end before adding tools, interruption handling, or telephony — voice UX bugs are much easier to diagnose in a simple loop than a fully-featured pipeline. Sign up at /signup if you need a Claude API key to start building.

Questions

Does Claude have native audio input or output? No. Claude processes text only. You need a separate STT service to transcribe audio into text before sending it to Claude, and a separate TTS service to convert Claude's text response into speech.

Why does my voice assistant feel slow even though Claude responds quickly? Latency usually comes from waiting for full responses before starting TTS, or from STT with high silence-detection thresholds. Stream Claude's output sentence-by-sentence into TTS instead of waiting for the complete reply.

Can I add real actions, like checking an order status, to a voice assistant? Yes, using Claude's tool use feature. Define functions with input schemas, and Claude will request a tool call when it needs external data, pausing generation until you return the result.