Build a Voice Assistant Using Claude API: Full Guide

Building a voice assistant with Claude API means wiring together three pieces: speech-to-text (STT) to capture what the user says, Claude to understand the request and generate a response, and text-to-speech (TTS) to speak the answer back. Claude itself doesn't process audio directly — it's a text-in, text-out model — so the "voice" part of a voice assistant always lives in the STT/TTS layer around it, while Claude handles reasoning, memory, and tool use.

This guide walks through the actual architecture, shows working code for the core loop, and covers the details that matter for a voice product specifically: latency, streaming, and interruption handling. If you just need to know how the pieces fit together, read the next two sections; if you're implementing, jump to the code.

The basic architecture

A voice assistant pipeline looks like this:

1. Microphone input → audio stream
2. Speech-to-text → transcribed text (e.g. Whisper, Deepgram, AssemblyAI, or a browser's Web Speech API for prototyping)
3. Claude API → generates a text response, optionally calling tools
4. Text-to-speech → converts the response to audio (e.g. ElevenLabs, Amazon Polly, OpenAI TTS, or platform-native TTS)
5. Speaker output → audio played back to the user

Claude sits in the middle as the "brain." It doesn't need to know audio exists — you just feed it transcribed text and get text back. This separation is actually an advantage: you can swap STT or TTS providers without touching your Claude integration, and you can test the conversational logic entirely with text before wiring up audio.

Why latency design matters more than model choice

For a chat app, a 2-second response feels fine. For a voice assistant, 2 seconds of silence after someone finishes speaking feels broken. Voice UX has a much lower tolerance for latency, so your architecture decisions should optimize for perceived responsiveness:

• Stream the Claude response token by token and start TTS on partial sentences instead of waiting for the full reply.
• Stream STT so you know the user stopped talking as early as possible (most STT providers support this via WebSocket).
• Keep system prompts and tool schemas lean. Every extra instruction adds tokens Claude has to process before the first output token appears.
• Use short, conversational system prompts. Voice responses should be shorter than chat responses — nobody wants a five-paragraph answer read aloud.

Building the core loop

Here's a minimal Node.js example of the STT → Claude → TTS loop, using SubToAPI as the Claude endpoint. SubToAPI turns your Claude access into a standard HTTPS API with an application key (sub_live_...), which is convenient if you want a clean single-key setup for a voice product without managing Claude account credentials directly in your backend.

import fetch from "node-fetch";

async function getClaudeReply(transcript, conversationHistory) {
  const response = await fetch("https://api.subtoapi.app/v1/messages", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      model: "claude-3-5-sonnet",
      max_tokens: 300,
      system: "You are a voice assistant. Keep replies short, spoken-language style, under 3 sentences unless asked for detail.",
      messages: [
        ...conversationHistory,
        { role: "user", content: transcript }
      ]
    })
  });

  const data = await response.json();
  return data.content[0].text;
}

This is the synchronous version — fine for prototyping. For production, switch to streaming so TTS can start speaking before Claude finishes generating:

async function streamClaudeReply(transcript, conversationHistory, onChunk) {
  const response = await fetch("https://api.subtoapi.app/v1/messages", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      model: "claude-3-5-sonnet",
      max_tokens: 300,
      stream: true,
      messages: [...conversationHistory, { role: "user", content: transcript }]
    })
  });

  for await (const chunk of response.body) {
    onChunk(chunk.toString());
  }
}

Feed each decoded text chunk into your TTS engine as soon as you've buffered a full sentence, rather than waiting for the whole response. Most TTS APIs accept short text segments, so batching by sentence boundary (splitting on ., ?, !) gives a good balance between latency and speech naturalness. See /docs/streaming for details on handling streamed responses.

Adding tools for real assistant behavior

A voice assistant that can only chat isn't very useful — it needs to check calendars, look up orders, control smart devices, or query a database. Claude's tool use (function calling) handles this: you define tools with a JSON schema, Claude decides when to call them, and you execute the actual logic in your backend.

{
  "name": "get_weather",
  "description": "Get current weather for a city",
  "input_schema": {
    "type": "object",
    "properties": {
      "city": { "type": "string" }
    },
    "required": ["city"]
  }
}

When Claude decides it needs weather data to answer, it returns a tool call instead of a text reply. Your code runs the actual lookup, sends the result back, and Claude incorporates it into the final spoken response. This is what turns "I don't have real-time data" into "It's 18°C and cloudy in Berlin right now." See /docs/tools for the full request/response shape.

Handling interruptions and turn-taking

Real voice conversations aren't strictly turn-based — users interrupt. A production assistant needs to detect when the user starts talking again mid-response and stop TTS playback immediately, then treat the new input as a fresh (or continuing) turn. This is handled entirely in your STT/audio layer, not in Claude — but it affects how you manage conversation history: you should log what was actually spoken before the interruption, not the full generated text, so Claude's next reply has accurate context.

Keep conversation history capped to the last several turns rather than sending the entire session — voice conversations can run long, and unbounded history increases both latency and cost per turn.

Getting started

If you're prototyping, start with the text loop first: get transcripts in, Claude replies out, verify the conversational quality and tool behavior before adding audio at all. Once that's solid, layer in streaming STT and TTS for real-time feel. SubToAPI gives you a single API key, streaming support, and usage metadata across your team, which simplifies the backend side while you focus on the audio pipeline. Check /docs/quickstart to get an API key running in minutes, or /pricing for plan details.

Questions

Can Claude process audio directly, without a separate STT step? No — Claude's API takes text input and returns text output. You need a speech-to-text service to transcribe audio before sending it to Claude, and a text-to-speech service to convert the reply back to audio.

What's the biggest source of latency in a Claude-based voice assistant? Usually waiting for the full response before starting TTS. Streaming the Claude response and starting speech synthesis on completed sentences, rather than the whole reply, cuts perceived latency significantly.

Do I need function calling for a basic voice assistant? Not for a simple Q&A or chat assistant, but any assistant that needs live data (calendar, orders, device state) requires tool use so Claude can call your backend functions instead of guessing an answer.