How to Build a Claude-Powered Voice Assistant App

Building a Claude-powered voice assistant app means wiring together three pieces: speech recognition to turn audio into text, Claude to understand and respond, and speech synthesis to turn the response back into audio. The architecture is simpler than it sounds — the hard parts are latency, streaming, and keeping conversation context across turns.

This guide walks through the full pipeline, from capturing a user's voice to playing back Claude's answer, with working code you can adapt to a web app, mobile app, or desktop assistant.

The core architecture

A voice assistant built on Claude has four stages that run in sequence for every turn of conversation:

1. Capture audio from the microphone (browser, mobile SDK, or desktop app)
2. Speech-to-text (STT) converts audio to a text transcript
3. Claude processes the transcript plus conversation history and generates a response
4. Text-to-speech (TTS) converts the response text back into audio and plays it

The STT and TTS steps are not Claude's job — you'll use a dedicated service for those (browser Web Speech API, Whisper, Deepgram, ElevenLabs, or your OS's native speech APIs). Claude handles the reasoning in the middle: understanding intent, maintaining context, calling tools, and producing the reply text.

Step 1: Capture and transcribe speech

For a web app, the simplest starting point is the browser's built-in SpeechRecognition API, which gives you a live transcript with no extra infrastructure:

const recognition = new webkitSpeechRecognition();
recognition.continuous = false;
recognition.interimResults = false;
recognition.lang = "en-US";

recognition.onresult = (event) => {
  const transcript = event.results[0][0].transcript;
  sendToClaude(transcript);
};

recognition.start();

For production apps, especially mobile, you'll likely want a dedicated STT provider (Whisper API, Deepgram, Google Speech-to-Text) for better accuracy across accents and noisy environments. The output you need is the same either way: a clean text string to send to Claude.

Step 2: Send the transcript to Claude

Once you have text, send it to Claude along with the running conversation history. This is where most of the voice assistant's "intelligence" lives — Claude decides what to say back, whether to call a tool (check a calendar, look up an order, control a smart device), and how to phrase the answer naturally for speech rather than reading text.

If you're calling the Claude API directly, you manage API keys, rate limits, and usage tracking yourself. If you want that handled for you, SubToAPI turns your existing Claude access into a standard HTTPS API with application keys, so your voice assistant backend just makes a normal REST call:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "max_tokens": 300,
    "messages": [
      {"role": "user", "content": "Set a reminder for 3pm to call the dentist"}
    ]
  }'
}'

See the quickstart and messages docs for the full request format. Keep system prompts short for voice assistants — instruct Claude to answer conversationally, avoid markdown or bullet lists (they sound awful read aloud), and keep responses under a sentence or two unless the user asks for detail.

Step 3: Stream the response for low latency

Voice assistants feel broken if there's a long silent pause before the reply starts playing. The fix is streaming: start converting and playing audio as soon as the first chunk of Claude's response arrives, instead of waiting for the full text.

const response = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    model: "claude-sonnet-4-5",
    max_tokens: 300,
    stream: true,
    messages: [{ role: "user", content: transcript }]
  })
});

const reader = response.body.getReader();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += new TextDecoder().decode(value);

  // flush complete sentences to TTS as they arrive
  const sentences = buffer.match(/[^.!?]+[.!?]/g);
  if (sentences) {
    sentences.forEach(speakWithTTS);
    buffer = buffer.replace(sentences.join(""), "");
  }
}

Splitting on sentence boundaries and sending each finished sentence to your TTS engine independently cuts perceived latency dramatically — the user hears the first words while Claude is still generating the rest. Full details on handling SSE chunks are in the streaming docs.

Step 4: Convert text back to speech

For TTS, options range from free (browser SpeechSynthesis API) to high-quality paid services (ElevenLabs, Google Cloud TTS, Amazon Polly) that sound much more natural and support emotional tone. For a prototype:

function speakWithTTS(text) {
  const utterance = new SpeechSynthesisUtterance(text);
  utterance.rate = 1.0;
  speechSynthesis.speak(utterance);
}

For a shipped product, a neural TTS provider is worth the cost — robotic voices undermine trust in an otherwise smart assistant.

Handling conversation memory

Voice assistants need context across turns ("set a reminder" → "actually make it 4pm" should work without re-explaining). Store the running message history per session and send it with each Claude request, trimming older turns as the conversation grows to control token usage and latency. Keep the history short and summarize old context rather than sending every raw turn — this matters more for voice than chat because latency compounds with context length.

Giving the assistant real actions

A voice assistant that can only talk, not act, is a chatbot with extra steps. If the assistant needs to check a calendar, send a message, or query a database, use Claude's tool-calling support so it can request structured actions instead of hallucinating results. The tools docs cover defining tool schemas and handling Claude's tool-use responses in your backend.

Putting it together for production

Once the pipeline works, the remaining work is operational: rate limiting per user, tracking usage so costs stay predictable, and giving team members their own access without sharing one raw key. SubToAPI handles the API-key and usage side of this with per-key usage metadata and team seats, so you can focus on the voice pipeline instead of building billing and access control from scratch. Compare plans or start with a free trial at signup.

Questions

Do I need a separate speech-to-text and text-to-speech service, or does Claude handle audio directly? Claude processes text, not raw audio, so you need a dedicated STT service to transcribe speech and a TTS service to synthesize the reply. Claude sits in the middle handling understanding and response generation.

How do I reduce the delay between speaking and hearing a reply? Stream Claude's response and send completed sentences to your TTS engine as they arrive, rather than waiting for the full reply. This alone cuts perceived latency more than any other optimization.

Can a Claude-powered voice assistant take real actions, like setting reminders or checking data? Yes, using Claude's tool-calling support. Define tools for the actions you need (calendar, database lookups, device control) and let Claude request them with structured input instead of just generating text.