Claude API Voice Assistant Integration Tutorial

How the Claude API Fits Into a Voice Assistant

A voice assistant is really three systems chained together: speech-to-text (STT) to turn audio into a transcript, a language model to understand the transcript and decide what to say or do, and text-to-speech (TTS) to turn the reply back into audio. Claude handles the middle piece — reasoning, conversation memory, and tool calls — but it does not transcribe or synthesize audio itself. This tutorial shows the full pipeline, where Claude sits in it, and how to keep latency low enough that the assistant feels responsive rather than laggy.

If you're integrating Claude into a voice product, the two things that matter most are streaming (so the assistant starts talking before the full response is generated) and tool use (so it can check a calendar, look up an order, or run a search instead of just talking). We'll cover both.

Architecture Overview

A typical voice assistant pipeline looks like this:

Microphone → STT engine → transcript → Claude API → text chunks → TTS engine → speaker

For real-time assistants (phone calls, smart speakers, in-app voice), you want streaming at every stage: streaming STT that emits partial transcripts, streaming Claude responses that emit tokens as they're generated, and streaming TTS that starts speaking before the whole sentence is synthesized. Chaining non-streaming components adds seconds of dead air, which users notice immediately in voice interfaces more than in chat.

Step 1: Capture and Transcribe Audio

Use any STT provider that supports streaming (Deepgram, AssemblyAI, Whisper-based services, or your platform's native speech recognition). The output you need is a finalized transcript segment — most STT APIs emit "interim" results followed by a "final" result once the speaker pauses. Only send final segments to Claude; sending every interim update wastes requests and confuses turn-taking.

sttStream.on("transcript", (event) => {
  if (event.isFinal) {
    handleUserUtterance(event.text);
  }
});

Step 2: Send the Transcript to Claude

Once you have a finalized utterance, send it to Claude along with the running conversation history. Keep a lightweight in-memory array of role/content pairs per session so Claude has context across turns — voice conversations are stateless on the wire, so you own the memory.

async function handleUserUtterance(text) {
  conversation.push({ role: "user", content: text });

  const response = await fetch("https://api.subtoapi.app/v1/messages", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      model: "claude-sonnet-4",
      max_tokens: 400,
      messages: conversation
    })
  });

  const data = await response.json();
  conversation.push({ role: "assistant", content: data.content });
  return data.content;
}

Keep max_tokens modest for voice. Long replies sound unnatural when read aloud and take longer to synthesize — most voice assistants cap responses at a few sentences and rely on follow-up turns for detail.

Step 3: Stream Responses to Cut Latency

Waiting for a full response before speaking is the single biggest source of perceived lag in voice assistants. Stream Claude's output and start sending text to your TTS engine sentence-by-sentence, not after the full reply finishes.

const stream = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    model: "claude-sonnet-4",
    max_tokens: 400,
    stream: true,
    messages: conversation
  })
});

let buffer = "";
for await (const chunk of stream.body) {
  buffer += chunk.toString();
  const sentences = extractCompleteSentences(buffer);
  for (const sentence of sentences.complete) {
    ttsQueue.push(sentence); // send to TTS as soon as a sentence is ready
  }
  buffer = sentences.remainder;
}

The pattern is: buffer tokens until you hit a sentence boundary (period, question mark, exclamation point), then hand that chunk to TTS immediately. Full details on handling streamed chunks are in the streaming docs.

Step 4: Convert Claude's Reply to Speech

Pipe each sentence to your TTS provider as it arrives. Most TTS APIs (ElevenLabs, Azure, Google, Amazon Polly) accept short text segments and return audio quickly enough to feel continuous if you're feeding them sentence-by-sentence rather than waiting for the whole response.

async function speak(sentence) {
  const audio = await ttsClient.synthesize(sentence, { voice: "assistant-1" });
  audioPlayer.enqueue(audio);
}

Queue the audio clips in order and play them back-to-back. This is where most of the "does it feel like a real conversation" quality comes from — get this part right and users forgive a slightly slower first word.

Step 5: Handle Tool Use for Real Actions

A voice assistant that can only talk is limited. Most useful assistants need to check a calendar, pull order status, or search a knowledge base mid-conversation. Claude's tool use lets you define functions it can call, and it will pause generation to request a tool call instead of guessing an answer.

{
  "model": "claude-sonnet-4",
  "messages": [{ "role": "user", "content": "What's my delivery status for order 4821?" }],
  "tools": [
    {
      "name": "get_order_status",
      "description": "Look up the current status of an order by ID",
      "input_schema": {
        "type": "object",
        "properties": { "order_id": { "type": "string" } },
        "required": ["order_id"]
      }
    }
  ]
}

When Claude returns a tool call, execute it on your side (a database lookup, an API call), send the result back in the conversation, and let Claude continue the response with the real data. This is the difference between a voice assistant that sounds smart and one that actually is. Full tool-calling examples are in the tools documentation.

Choosing Between Direct Claude Access and an API Gateway

For a single prototype, calling Claude directly is fine. For a production voice assistant — especially one with a team shipping features, multiple environments, or usage you need to track per feature — an API layer like SubToAPI turns your existing Claude access into a standard HTTPS API with its own sub_live_ keys, streaming support, and usage metadata per key. That matters for voice products specifically because you often run separate keys for staging call flows, production call flows, and internal testing, and want to see exactly where latency or token spend is coming from without digging through raw Claude logs. Setup takes a few minutes — see the quickstart and pricing if you want to compare plans before wiring it into your call pipeline.

FAQ

Does Claude's API support audio input directly?

No. Claude's API works with text (and in some setups, images). You need a separate STT service to convert audio to text before sending it to Claude, and a TTS service to convert its text reply back into audio.

How do I keep response latency low enough for a phone call?

Stream the Claude response and start TTS synthesis on each completed sentence rather than waiting for the full reply. Combined with streaming STT, this keeps the perceived gap between the user finishing speaking and the assistant starting to talk under a second in most setups.

Can the voice assistant take real actions, like booking appointments?

Yes, through tool use. Define the action as a tool with a schema, let Claude request the call when relevant, execute it on your backend, and feed the result back so Claude can confirm the action in its spoken reply. See the tools guide for the full pattern.