How to Build a Claude-Powered Voice Assistant

Building a Claude-powered voice assistant means chaining three systems together: speech-to-text (STT) to turn audio into words, Claude to understand the request and generate a response, and text-to-speech (TTS) to speak the answer back. The hard part isn't any single piece — it's making the round trip fast enough that a conversation feels natural instead of laggy.

This guide walks through the architecture, the code for each stage, and the specific tricks (streaming, partial playback, interruption handling) that separate a demo from something people actually want to talk to.

The basic pipeline

A voice assistant is a loop with four stages:

1. Capture audio from the microphone until the user stops talking (voice activity detection).
2. Transcribe the audio to text with an STT service.
3. Send the text to Claude and get a response.
4. Convert the response to speech and play it back.

Mic → VAD → STT → Claude API → TTS → Speaker

Each hop adds latency. A naive implementation that waits for the full Claude response before starting TTS can easily take 3-5 seconds before the user hears anything — that feels broken in a voice context, even though it's perfectly acceptable for a chat UI.

Step 1: capture and transcribe audio

You don't need to build STT yourself. Use a service like Whisper (self-hosted or via API) or a cloud STT provider that supports streaming transcription. Streaming STT is important: it lets you start processing the user's speech before they've finished the sentence, and it gives you a natural endpoint-of-speech signal.

A minimal Node.js capture loop looks like this (pseudocode for the STT call, since the actual provider API varies):

const audioStream = startMicCapture(); // returns a readable stream of PCM audio

const transcript = await transcribeStream(audioStream, {
  onPartial: (text) => console.log("partial:", text),
  onFinal: (text) => handleUserTurn(text),
});

The onFinal callback fires once the STT engine detects silence or an endpoint. That final text is what you send to Claude.

Step 2: send the transcript to Claude

Once you have clean text, the request to Claude looks the same as any chat completion. If you're already using SubToAPI to turn your Claude access into an API, the call is a standard HTTPS request:

const response = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "claude-sonnet-4",
    max_tokens: 400,
    system: "You are a voice assistant. Keep answers short, conversational, and free of markdown or bullet points — your output will be read aloud.",
    messages: [{ role: "user", content: userTranscript }],
  }),
});

The system prompt matters more for voice than for chat. Claude's default style uses headings, bullet points, and code blocks, which sound awkward or nonsensical when spoken. Explicitly instruct it to write in plain, short, spoken sentences. See /docs/messages for the full request schema.

Step 3: stream the response into TTS

This is where latency gets won or lost. Instead of waiting for the full response, stream it and feed completed sentences into TTS as they arrive. Most TTS engines accept text chunk by chunk, so you can start speaking the first sentence while Claude is still generating the third.

const stream = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "claude-sonnet-4",
    max_tokens: 400,
    stream: true,
    messages: [{ role: "user", content: userTranscript }],
  }),
});

let buffer = "";
for await (const event of parseSSE(stream.body)) {
  if (event.type === "content_block_delta") {
    buffer += event.delta.text;
    const sentenceEnd = buffer.match(/[.!?]\s/);
    if (sentenceEnd) {
      const sentence = buffer.slice(0, sentenceEnd.index + 1);
      buffer = buffer.slice(sentenceEnd.index + 1);
      speak(sentence); // send to TTS and queue for playback
    }
  }
}
if (buffer.trim()) speak(buffer);

Splitting on sentence boundaries rather than waiting for the full response typically cuts time-to-first-audio from a few seconds to a few hundred milliseconds. Full streaming details are in /docs/streaming.

Step 4: handle interruptions and barge-in

A real conversation includes the user interrupting mid-response. Handle this by:

• Running VAD continuously, even while TTS is playing.
• When new speech is detected during playback, stop the audio immediately and cancel the in-flight Claude stream.
• Treat the new utterance as a fresh turn, appending it to conversation history so Claude has context for the interruption.

Without this, your assistant will talk over the user and feel unusable in practice, no matter how good the responses are.

Step 5: add tools for real actions

A voice assistant that can only talk is limited. If you want it to check a calendar, look up an order, or control a device, give Claude tool definitions and let it call them mid-conversation. The flow is: Claude returns a tool_use block instead of plain text, your code executes the function, and you send the result back as a tool_result so Claude can finish the response in natural language. The mechanics are covered in /docs/tools — the voice-specific part is just making sure the tool round trip doesn't block audio for too long, since function calls add another network hop before the user hears anything.

Managing keys and cost across a team

If more than one person on your team is building against Claude — a mobile app developer, a backend engineer, a QA tester — you don't want to share one raw API key or manage billing manually. SubToAPI issues separate sub_live_... keys per person or environment under one subscription, so you can see usage per key, rotate a key if a device is compromised, and keep billing on a single invoice. Plans start at Solo for a single builder and go up to Team and Scale for multiple seats — see /pricing. Setup takes about five minutes; see /docs/quickstart.

Putting it together

A production-ready Claude voice assistant is really a latency-optimization problem wrapped around three well-understood components: STT, an LLM, and TTS. Get the streaming and sentence-chunking right, handle interruptions explicitly, and keep your system prompt tuned for spoken output, and the result will feel far more responsive than a simple "wait, then speak" implementation.

Questions

Does Claude have native audio input/output? No — Claude works with text. You need a separate STT engine to transcribe audio into text before sending it to Claude, and a TTS engine to convert its text response back into audio.

How do I reduce the delay before the assistant starts speaking? Stream the Claude response and feed it into TTS sentence by sentence instead of waiting for the full reply. This is the single biggest latency improvement you can make; see /docs/streaming.

Can the assistant take actions, like booking a meeting? Yes, using Claude's tool use feature. Define functions for the actions you want, let Claude call them when relevant, and return the results so it can speak a natural confirmation. Details are in /docs/tools.