How to Setup LLM Locally: A Developer's Guide

Setting up an LLM locally means installing a model runtime on your own machine (or server) and downloading model weights so you can run inference without sending data to a third-party API. The short version: pick a runtime like Ollama or LM Studio, download a quantized model that fits your hardware, and start it with a single command. The rest of this guide walks through the decisions that actually matter — hardware, model size, runtime choice, and how to test the setup once it's running.

Running an LLM locally makes sense when you need data privacy, want to experiment offline, or are building something that doesn't require the latest frontier model quality. It doesn't replace hosted models like Claude for production apps that need strong reasoning, long context, or tool use — but it's the right call for prototyping, local dev environments, and privacy-sensitive workloads.

Step 1: Check your hardware

Local LLM performance depends almost entirely on RAM (or VRAM if you have a GPU) and, secondarily, on CPU/GPU speed.

• CPU-only, 8–16GB RAM: you can run small models (3B–8B parameters, quantized) at usable speed.
• GPU with 8–12GB VRAM: comfortably run 7B–13B models with good throughput.
• GPU with 24GB+ VRAM: run 30B+ models or larger context windows.
• Apple Silicon (M1/M2/M3/M4): unified memory means a 16–32GB Mac can run 7B–13B models well thanks to Metal acceleration.

Quantization is what makes local LLMs feasible on consumer hardware. A 7B model at full precision (FP16) needs ~14GB of memory; the same model quantized to 4-bit (Q4) needs closer to 4–5GB, with a modest quality trade-off. Almost every local setup today uses quantized GGUF or AWQ weights rather than full-precision checkpoints.

Step 2: Pick a runtime

You don't need to write inference code yourself. A handful of tools handle model loading, quantization, and serving an HTTP API:

• Ollama — the fastest way to get started. Single binary, simple CLI, built-in model library, and an OpenAI-compatible local API on localhost:11434.
• LM Studio — a desktop GUI on top of llama.cpp. Good if you want a chat interface without touching a terminal.
• llama.cpp — the underlying C++ inference engine most tools build on. Use it directly if you want maximum control over quantization and performance flags.
• vLLM — built for GPU servers and high-throughput serving, not laptops. Use it if you're deploying a local model behind an internal API for multiple users.

For most developers, Ollama is the practical starting point because it handles model downloads, quantization format, and API serving in one tool.

Step 3: Install and pull a model

On macOS/Linux:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b

On Windows, download the installer from Ollama's site, then run the same ollama pull command from PowerShell.

Model name suffixes indicate size and quantization — llama3.1:8b is the 8-billion-parameter model at a default quantization level. If memory is tight, look for smaller variants (:8b-instruct-q4_0) or smaller model families (Phi-3, Gemma 2, Qwen2.5-7B).

Step 4: Run and test it

Once pulled, start a chat session directly:

ollama run llama3.1:8b

Or call the local HTTP API from your own code, which is what you'll actually use in an application:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Summarize the tradeoffs of running LLMs locally.",
  "stream": false
}'

From JavaScript:

const res = await fetch("http://localhost:11434/api/chat", {
  method: "POST",
  body: JSON.stringify({
    model: "llama3.1:8b",
    messages: [{ role: "user", content: "Explain quantization in one paragraph." }],
  }),
});
const data = await res.json();
console.log(data.message.content);

This gives you a working local LLM API you can point any client at, with no external network calls once the model is downloaded.

Step 5: Tune for your use case

• Context length: local models often default to a smaller context window than their max supported size to save memory. Set it explicitly with Ollama's num_ctx parameter if you need longer conversations.
• GPU offload: if you have a GPU but the runtime is defaulting to CPU, check driver installation (CUDA for NVIDIA, ROCm for AMD) — most tools auto-detect but occasionally need a nudge.
• System prompts: local models often need more explicit instructions than hosted frontier models to stay on task. Be more directive than you would with GPT-4 or Claude.

When local setup isn't enough

Local LLMs are great for experimentation, but they hit real limits: reasoning quality lags behind frontier hosted models, tool use and function calling are less reliable, and running larger models means buying or renting serious GPU hardware. If your application needs consistent tool use, streaming responses, and production-grade output quality, a hosted API is usually the better trade.

If you already have Claude access and want an HTTPS API without managing local infrastructure, SubToAPI turns that access into application API keys with streaming, tool use, and usage metadata — the same integration pattern as your local Ollama endpoint, but backed by a frontier model. Check the quickstart to see how the request shape compares to what you just built locally.

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "claude-sonnet-4", "messages": [{"role": "user", "content": "Hello"}]}'

Many teams run both: local models for offline dev and low-stakes tasks, and a hosted API for anything customer-facing. Pricing for the hosted route starts on the pricing page if you want to compare the cost of GPU hardware against a monthly API plan.

questions

Do I need a GPU to run an LLM locally? No. Quantized 7B–8B models run acceptably on a modern CPU with 16GB of RAM, though a GPU with 8GB+ VRAM gives noticeably faster responses.

What's the easiest tool to setup an LLM locally? Ollama is the easiest starting point — one install command, one ollama pull command, and a local HTTP API ready to use in minutes.

How much disk space do local LLMs need? Quantized models typically range from 4GB (7B, 4-bit) to 20GB+ (30B+ models). Plan for at least 20–30GB free if you want to try a few model sizes.