How to Build a Claude-Powered Image Captioning Tool
Building a Claude-powered image captioning tool means sending an image to Claude's vision-enabled models alongside a text prompt and getting back a natural-language description, alt text, or structured tags. The core mechanics are simple: encode the image, attach it as a content block in a messages request, and tune the prompt for the output format you need (short alt text, detailed description, or JSON with categories).
This guide walks through the full implementation — image encoding, prompt design for different captioning use cases, batch processing for galleries or product catalogs, and the production details (rate limits, cost control, error handling) that separate a demo from something you'd actually ship.
What you're building
A typical captioning tool does one of three things:
- Accessibility alt text — short, descriptive text for screen readers
- SEO/metadata captions — longer descriptions optimized for search and CMS fields
- Structured tagging — JSON output with objects, colors, mood, or category labels for search/filtering
All three use the same underlying request shape — an image content block plus a text instruction — with different prompts and output formats.
Step 1: Encode the image
Claude's vision models accept images as base64-encoded data in the request body. You can also pass a URL in some integrations, but base64 is the most portable approach and works identically whether the image comes from an upload, a URL fetch, or local storage.
import fs from "fs";
function encodeImage(path) {
const buffer = fs.readFileSync(path);
return buffer.toString("base64");
}
const imageData = encodeImage("./product.jpg");Supported formats are JPEG, PNG, GIF, and WebP. Keep images under a few megabytes — resizing large uploads before sending them reduces latency and token cost without hurting caption quality for most use cases.
Step 2: Send the captioning request
The message body includes an image content block and a text block with your instruction. Here's a basic alt-text generator using SubToAPI's /v1/messages endpoint, which exposes Claude through a standard HTTPS API with your own sub_live_... key:
curl https://api.subtoapi.app/v1/messages \
-H "Authorization: Bearer $SUBTOAPI_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-sonnet-4-5",
"max_tokens": 200,
"messages": [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": "'"$(base64 -w0 product.jpg)"'"
}
},
{
"type": "text",
"text": "Write a concise, descriptive alt text for this image in one sentence. Focus on the main subject and relevant context, avoid starting with \"image of\"."
}
]
}
]
}'This pattern works the same way whether you're calling Anthropic directly or through SubToAPI — see the messages docs for the full request/response reference if you're integrating it into an existing backend.
Step 3: Design prompts for different caption types
Prompt wording has a bigger effect on output quality than model choice here. A few patterns that work well in practice:
Alt text (accessibility):
Write alt text for this image in under 125 characters. Describe only what's visually present. No speculation about context not shown.SEO description:
Write a 2-3 sentence product description for this image suitable for an e-commerce listing. Mention color, material, and style if visible.Structured tags:
Analyze this image and return JSON with keys: "objects" (array of strings), "dominant_colors" (array), "setting" (string), "mood" (string). Return only valid JSON, no explanation.For the structured case, parse the response and validate it before storing it — Claude is reliable at following JSON instructions, but you should still wrap parsing in a try/catch and have a fallback for malformed output.
Step 4: Batch captioning for galleries or catalogs
If you're captioning hundreds or thousands of images — a product catalog migration, a photo library, a CMS backfill — process them in small concurrent batches rather than one-by-one sequentially or all at once.
async function captionImage(base64Data, mediaType) {
const res = await fetch("https://api.subtoapi.app/v1/messages", {
method: "POST",
headers: {
"Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "claude-sonnet-4-5",
max_tokens: 150,
messages: [{
role: "user",
content: [
{ type: "image", source: { type: "base64", media_type: mediaType, data: base64Data } },
{ type: "text", text: "Write a one-sentence caption for this image." },
],
}],
}),
});
const json = await res.json();
return json.content[0].text;
}
async function captionBatch(images, concurrency = 5) {
const results = [];
for (let i = 0; i < images.length; i += concurrency) {
const chunk = images.slice(i, i + concurrency);
const chunkResults = await Promise.all(
chunk.map(img => captionImage(img.base64, img.mediaType))
);
results.push(...chunkResults);
}
return results;
}A concurrency of 3–5 is a reasonable starting point. Pushing higher increases throughput but also increases your risk of hitting rate limits, especially on a shared plan. Track failures per image and retry individually rather than re-running the whole batch.
Step 5: Handle streaming for live previews
If captions are generated interactively — a user uploads an image and watches text appear — use streaming instead of waiting for the full response. This matters more for longer, multi-sentence descriptions than for short alt text, but it noticeably improves perceived speed either way. See the streaming docs for the event format and how to consume server-sent events in a frontend.
Production considerations
A few things that matter once this moves past a prototype:
- Cost control: captioning is usually cheap per-image, but at catalog scale it adds up. Cap
max_tokenstightly for short caption types — there's no reason to allow 1000 tokens for alt text. - Image resizing: downscale large uploads before encoding. A 4000px product photo doesn't produce a better caption than a 1200px version, and it costs more in tokens and latency.
- Fallback handling: if the model returns an empty or malformed response, don't silently fail — log it and retry once before falling back to a manual review queue.
- Access control: if multiple team members or services call your captioning pipeline, separate API keys per service or team member make it easier to track usage and catch runaway loops. SubToAPI's pricing tiers include per-seat keys with usage visibility for exactly this.
If you're starting from scratch and want to skip the account setup, the quickstart guide covers getting a key and making your first request in a few minutes via signup.
What image formats does Claude support for captioning?
JPEG, PNG, GIF, and WebP, sent as base64-encoded data in an image content block. There's no native TIFF or BMP support, so convert those formats before sending.
How long should a generated caption be?
Depends on the use case — alt text should stay under roughly 125 characters for screen reader usability, while SEO or catalog descriptions can run 2–4 sentences. Control length explicitly in the prompt rather than relying on max_tokens alone, since the model may otherwise stop mid-sentence.
Can I get structured output instead of plain text captions?
Yes — instruct the model to return JSON with a specific schema (object names, colors, category, mood, etc.) and parse the response on your side. Validate the parsed JSON before storing it, since you should always handle the case where the model's output doesn't perfectly match the requested shape.