How to Build a Voice Assistant With Claude API

Building a voice assistant with the Claude API means wiring together three pieces: speech-to-text to capture what the user says, Claude to understand and respond, and text-to-speech to speak the answer back. Claude itself doesn't process raw audio directly through a conversational voice interface — you need a pipeline that converts speech to text, sends that text to Claude for reasoning, and converts the response back into audio.

This article walks through that pipeline end to end: capturing audio, transcribing it, structuring the conversation for Claude, streaming the response so the assistant feels responsive instead of laggy, optionally calling tools (to check a calendar, look up an order, control a device), and speaking the result. We'll use plain curl/JavaScript examples throughout, and show where a hosted API layer like SubToAPI simplifies the Claude side of the stack.

The basic architecture

A voice assistant is a loop with four stages:

1. Capture audio — microphone input, either push-to-talk or continuous listening with voice activity detection (VAD).
2. Transcribe — speech-to-text (STT) converts audio to a text string. Options include Whisper, Deepgram, AssemblyAI, or browser-native Web Speech API for prototypes.
3. Reason and respond — the transcribed text goes to Claude as a chat message, along with conversation history and any tools you want it to be able to call.
4. Speak — text-to-speech (TTS) converts Claude's response into audio and plays it back.

Latency budget matters here. Users tolerate maybe 1–2 seconds of silence before a voice interaction feels broken. That means you want streaming at every stage where possible: streaming STT (partial transcripts as the user talks), streaming Claude responses (tokens as they're generated), and streaming TTS (audio chunks as text arrives).

Step 1: Transcribe the audio

For a prototype, the Web Speech API in the browser gets you a working transcript with zero backend code:

const recognition = new webkitSpeechRecognition();
recognition.continuous = false;
recognition.interimResults = true;

recognition.onresult = (event) => {
  const transcript = event.results[0][0].transcript;
  if (event.results[0].isFinal) {
    sendToClaude(transcript);
  }
};

recognition.start();

For production, a dedicated STT service gives you better accuracy, punctuation, and noise handling, and it runs server-side so you're not dependent on browser support.

Step 2: Send the transcript to Claude

Once you have a text transcript, treat it like any other chat message. Keep a running conversation array so Claude has context from earlier turns — this is what makes a voice assistant feel like it's actually listening, not resetting every sentence.

Using SubToAPI's Messages endpoint, a turn looks like this:

async function sendToClaude(transcript, history) {
  const response = await fetch('https://api.subtoapi.app/v1/messages', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${process.env.SUBTOAPI_KEY}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      model: 'claude-sonnet-4-5',
      max_tokens: 400,
      system: 'You are a concise voice assistant. Keep answers short and speakable — no bullet points, no markdown.',
      messages: [...history, { role: 'user', content: transcript }]
    })
  });

  return response.json();
}

That system prompt line matters more for voice than for a chat UI: Claude's default writing style includes formatting, headers, and lists that make no sense read aloud. Tell it explicitly to write in short, spoken sentences.

Full request and response shapes are in the Messages API docs.

Step 3: Stream the response for lower latency

Waiting for Claude to finish a full response before starting TTS adds unnecessary delay. Instead, stream tokens and start feeding completed sentences to your TTS engine as soon as they're available, rather than waiting for the entire reply.

const stream = await fetch('https://api.subtoapi.app/v1/messages', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SUBTOAPI_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    model: 'claude-sonnet-4-5',
    max_tokens: 400,
    stream: true,
    messages: history
  })
});

let buffer = '';
for await (const chunk of streamToTextChunks(stream.body)) {
  buffer += chunk;
  if (/[.!?]\s*$/.test(buffer)) {
    speak(buffer); // send this sentence to TTS now
    buffer = '';
  }
}

This sentence-chunking approach is the single biggest lever for making a voice assistant feel instant instead of sluggish. Details on event formats and parsing are in the streaming docs.

Step 4: Give it tools, not just talk

A voice assistant that can only chat is a novelty. One that can check your calendar, place an order, or query a database is useful. Claude's tool use lets you define functions (get_weather, check_order_status, create_calendar_event) and Claude will decide when to call them and with what arguments, based on the conversation.

{
  "tools": [
    {
      "name": "get_calendar_events",
      "description": "Fetch events for a given date",
      "input_schema": {
        "type": "object",
        "properties": { "date": { "type": "string" } },
        "required": ["date"]
      }
    }
  ]
}

When Claude returns a tool call instead of plain text, your backend executes it, feeds the result back as a tool result message, and Claude incorporates it into a spoken response — "You have two meetings today, one at 10am and one at 3pm." Tool definitions and the full request/response cycle are covered in the tools documentation.

Step 5: Speak the response

Any TTS provider (ElevenLabs, Google Cloud TTS, Amazon Polly, or the browser's SpeechSynthesis API) can take the text chunks from step 3 and play audio. Keep sentences short and avoid markdown in your system prompt — numbered lists and bold text don't translate to speech and will make the TTS output sound broken.

Handling interruptions and turn-taking

A natural voice assistant needs to handle the user talking over it. Practically, this means: stop audio playback immediately when new speech is detected, cancel the in-flight Claude stream if the user interrupts mid-response, and discard any buffered but unspoken sentences. This logic lives in your client, not in Claude — the API just needs a fresh request with updated history once the interruption is resolved.

Why use a hosted API layer

Handling raw Claude API keys, retries, rate limits, and usage tracking across a product with many users adds real operational overhead on top of the audio pipeline you're already building. SubToAPI gives you application-level API keys (sub_live_...), streaming support, and usage metadata per key out of the box, so your voice assistant's reasoning layer is one less thing to manage. Start with the quickstart guide, check pricing for Solo, Team, and Scale plans, or sign up for a free trial.

questions

Does Claude process audio directly? No. Claude works with text. You need a speech-to-text step before sending input to Claude and a text-to-speech step after receiving its response — Claude handles the reasoning in the middle.

How do I reduce latency in a voice assistant? Stream everything: partial transcripts from STT, token-by-token responses from Claude (see /docs/streaming), and sentence-level chunks to your TTS engine instead of waiting for full responses at each stage.

Can a Claude-powered voice assistant take actions, not just answer questions? Yes, using tool use. Define functions for actions like checking orders or creating calendar events, and Claude will call them with structured arguments when the conversation requires it. See /docs/tools.