How to Use a Local LLM: A Practical Guide

"Using" a local LLM means three things in practice: getting a model running on your own machine, talking to it interactively to test it, and calling it programmatically from your scripts or apps. Most people get stuck on the first step and never make it to the third, which is where local LLMs actually become useful for day-to-day work.

The fastest path is to install a local runtime like Ollama or LM Studio, pull a quantized model that fits your hardware, and interact with it through a chat window or a local HTTP endpoint. This guide covers each step, plus the tradeoffs you'll hit once you try to use a local LLM for real projects instead of just experimenting.

Step 1: Pick a runtime

You don't need to write inference code yourself. Three tools cover almost every use case:

• Ollama — command-line first, exposes a local REST API on http://localhost:11434, good default choice for developers.
• LM Studio — GUI-based, easier for non-terminal users, also exposes an OpenAI-compatible local API.
• llama.cpp directly — more control, more setup, useful if you're tuning performance on specific hardware.

For most people learning how to use a local LLM, Ollama is the least friction. Install it, and you have both a CLI and an API in one binary.

curl -fsSL https://ollama.com/install.sh | sh

Step 2: Pull a model that fits your hardware

Local LLMs are constrained by RAM and, if you have one, GPU VRAM. Rough guidance:

• 7–8B parameter models (quantized) run comfortably on 16GB RAM, no GPU required, though slower on CPU.
• 13–14B models want 24GB+ RAM or a mid-range GPU.
• 70B+ models need a serious GPU setup or multiple cards — not realistic on a laptop.

Pull a model with Ollama:

ollama pull llama3.1:8b

Quantization (the q4, q5 suffixes you'll see on model pages) trades a small amount of accuracy for a much smaller memory footprint. Start with a quantized 7–8B model, confirm it works, then scale up if your hardware allows.

Step 3: Chat with it interactively

Once pulled, run it directly from the terminal:

ollama run llama3.1:8b

This drops you into an interactive prompt. It's the quickest way to sanity-check that the model is coherent, fast enough, and good at the kind of tasks you care about — summarization, code, general Q&A — before you build anything on top of it.

Step 4: Call it from code

The real value of a local LLM is using it inside a script, a CLI tool, or an app. Ollama and LM Studio both expose an HTTP API on localhost, so you can call it like any other model endpoint:

const res = await fetch("http://localhost:11434/api/generate", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "llama3.1:8b",
    prompt: "Summarize this changelog in three bullet points: ...",
    stream: false
  })
});

const data = await res.json();
console.log(data.response);

For streaming responses (useful for chat UIs), set stream: true and read the response body as newline-delimited JSON chunks instead of waiting for the full payload.

Step 5: Add structure — system prompts, context, tools

Local LLMs support the same patterns as hosted models:

• System prompts to set behavior and tone.
• Context windows — check the model card, since local models often ship with smaller context limits (4k–8k tokens) than hosted frontier models.
• Function/tool calling — supported by some models (like Llama 3.1) but implementation varies by runtime and isn't as consistent as with hosted APIs.

If you're building anything beyond a simple prompt-response loop, test tool calling and long-context behavior early. This is usually where local models show their limits compared to hosted alternatives.

When a local LLM isn't enough

Local LLMs are genuinely good for privacy-sensitive tasks, offline work, high-volume simple tasks where quality bar is low, and experimentation without API costs. They fall short when you need:

• Frontier-level reasoning or long, complex context handling
• Reliable, well-documented tool use and function calling
• Consistent output quality across a team, not just your one machine
• No local hardware/ops burden — someone always has to keep the runtime updated and the model current

In those cases, the practical move is to keep your local setup for prototyping and route production traffic to a hosted model like Claude. SubToAPI turns your existing Claude access into a standard HTTPS API — sub_live_... keys, streaming, tool use, and usage metadata — so you can swap your local fetch call for a hosted one without rewriting your app logic. The quickstart and messages docs mirror the same request/response shape you're already using locally, which makes the switch mostly a matter of changing the endpoint and auth header.

A common pattern: develop and test against a local model to iterate quickly and for free, then point the same integration at a hosted API for production quality, streaming (/docs/streaming) and tool calls (/docs/tools) that need to work reliably for every user, not just on your machine.

questions

Do I need a GPU to use a local LLM? No. Quantized 7–8B models run on CPU with 16GB of RAM, just slower than with a GPU. A GPU speeds up inference significantly and is necessary for larger models (13B+).

What's the difference between Ollama and LM Studio? Ollama is CLI-first and easier to script or integrate into apps via its local API. LM Studio has a GUI, which is friendlier for browsing and testing models manually, and also exposes an OpenAI-compatible local endpoint.

Can a local LLM replace a hosted API like Claude? For simple, privacy-sensitive, or offline tasks, yes. For complex reasoning, long context, or reliable tool use across a team, hosted models like Claude — accessible via SubToAPI — generally outperform local models running on consumer hardware.