Claude API Voice Assistant Integration Example

What a Claude-powered voice assistant actually needs

A voice assistant built on the Claude API is not one integration — it's three pieces chained together: speech-to-text (STT) to turn audio into a transcript, Claude to generate the response, and text-to-speech (TTS) to turn that response back into audio. Claude itself doesn't accept or produce audio directly, so the "integration" work is really about wiring these three stages so the latency feels conversational instead of robotic.

This article walks through a working example: a browser or server app that captures a question, sends the transcript to Claude, streams the reply token by token, and speaks it back as soon as sentences are ready — which is the part most tutorials skip and the part that actually determines whether your assistant feels responsive.

The three-stage pipeline

Microphone audio
   → STT (Whisper, Deepgram, browser SpeechRecognition)
   → Claude API (text in, text out)
   → TTS (ElevenLabs, Azure, browser speechSynthesis)
   → Speaker output

Each stage adds latency. A naive implementation waits for the full Claude response before sending anything to TTS, which means users hear nothing for 2-4 seconds after Claude starts "thinking." The fix is streaming: start speaking the first sentence while Claude is still generating the rest.

Step 1: capture and transcribe speech

For a browser-based assistant, the Web Speech API is the fastest way to prototype:

const recognition = new webkitSpeechRecognition();
recognition.continuous = false;
recognition.interimResults = false;

recognition.onresult = async (event) => {
  const transcript = event.results[0][0].transcript;
  await sendToClaude(transcript);
};

recognition.start();

For production, most teams use Deepgram or Whisper for better accuracy with background noise and accents, especially on phone-call style integrations (Twilio, telephony SDKs).

Step 2: stream Claude's response sentence by sentence

This is the core of a good voice integration. You want Claude to stream tokens so you can detect sentence boundaries and hand each finished sentence to TTS immediately, rather than waiting for the whole reply.

Here's the pattern using SubToAPI's streaming endpoint, which exposes Claude through a standard API key (sub_live_...) so you don't need separate Anthropic billing setup for a prototype:

const response = await fetch('https://api.subtoapi.app/v1/messages', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SUBTOAPI_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    model: 'claude-sonnet-4-5',
    max_tokens: 400,
    stream: true,
    system: 'You are a concise voice assistant. Keep responses short, conversational, and avoid lists or markdown since this will be spoken aloud.',
    messages: [{ role: 'user', content: transcript }]
  })
});

let buffer = '';
const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  const chunk = decoder.decode(value);
  buffer += parseTextDelta(chunk); // extract text from SSE events

  // Flush complete sentences to TTS as soon as they appear
  const sentences = buffer.match(/[^.!?]+[.!?]+/g);
  if (sentences) {
    sentences.forEach(speak);
    buffer = buffer.slice(sentences.join('').length);
  }
}
if (buffer.trim()) speak(buffer);

The system prompt matters a lot here. Voice output sounds wrong if Claude returns bullet points, headers, or code blocks, so explicitly instruct it to respond in plain spoken sentences. Details on message formatting and streaming event shapes are in the docs/messages and docs/streaming references.

Step 3: speak the response

For quick prototypes, the browser's built-in synthesis is enough:

function speak(sentence) {
  const utterance = new SpeechSynthesisUtterance(sentence.trim());
  utterance.rate = 1.05;
  speechSynthesis.speak(utterance);
}

For production voice products, swap this for ElevenLabs or Azure Neural TTS, which support streaming audio generation too — meaning you can pipe Claude's sentence stream straight into a TTS stream without buffering a full response on either end.

Handling interruptions and tool use

Real voice assistants need to handle a user talking over the assistant ("barge-in") and need access to real data — calendar, weather, account info — not just conversation. For the data problem, Claude's tool use lets the model call functions mid-response (checking a calendar, looking up an order) before finishing its spoken answer; see docs/tools for the schema. For barge-in, cancel the in-flight fetch request and stop TTS playback immediately when new microphone input is detected — don't wait for Claude's stream to finish.

Why route through an API layer instead of calling Anthropic directly

If you're building a voice assistant as a side project or internal tool, you likely already have a Claude subscription rather than a pay-per-token Anthropic API contract. SubToAPI turns that subscription into a standard HTTPS API: you get a sub_live_... key, the same streaming behavior shown above, and usage metadata per request so you can see token cost per voice interaction without setting up separate API billing. For a team building a voice product with multiple developers, team seats on the Team (€19/seat) or Scale (€49/seat) plans keep everyone on one dashboard instead of sharing one account. There's a free trial at /signup and a full walkthrough at /docs/quickstart if you want to get the pipeline above running in the next ten minutes.

Latency checklist

• Stream everything. Both Claude's output and your TTS input/output should be streamed, not buffered.
• Keep system prompts short and explicit about spoken format. No markdown, no lists, short sentences.
• Cap max_tokens conservatively. Voice answers should be brief; a 1000-token reply is a bad voice experience even if it's a good chat answer.
• Cancel in-flight requests on interruption. A voice assistant that keeps talking after the user interrupts feels broken immediately.
• Test on real network conditions. Mobile connections add latency that doesn't show up on localhost testing.

Questions

Does the Claude API support audio input or output directly? No. Claude processes text only. You need a separate STT service to convert speech to text before calling Claude, and a separate TTS service to convert its text response back to speech.

How do I reduce the delay between asking a question and hearing an answer? Stream Claude's response and send completed sentences to TTS as they arrive instead of waiting for the full reply — this is the single biggest latency improvement available.

Can I build a phone-based voice assistant with this same pattern? Yes. Replace the browser's STT/TTS with a telephony-integrated provider (Twilio Voice, Deepgram for transcription) and keep the same Claude streaming logic in the middle — the architecture doesn't change, only the audio I/O layer.