How to Set Up an LLM Locally: A Practical Walkthrough

Setting up a large language model locally means running the model's weights on your own hardware — your laptop, desktop, or a home server — instead of calling a hosted API over the internet. The core steps are: pick a model that fits your hardware, install a runtime that can load it, download the weights, and run a simple prompt through it to confirm it works. This guide walks through exactly that, plus the tradeoffs that usually convince people to move certain workloads to a hosted API later.

Running an LLM locally is worth doing when you care about privacy, offline access, zero per-token cost, or full control over the model. It's less worth it when you need the largest, most capable models, multi-user access, or production-grade reliability — those usually require either serious GPU hardware or a hosted API.

What "running an LLM locally" actually requires

You need three things:

1. A model — a set of weights, usually distributed in a quantized format like GGUF so it fits in consumer RAM/VRAM.
2. A runtime — software that loads the weights and exposes an inference loop (and often an HTTP API).
3. Enough hardware — RAM and/or a GPU with enough VRAM to hold the model plus its context.

As a rough guide for quantized models:

• 7–8B parameter models: 8–16 GB RAM, runs fine on a modern laptop CPU, faster with any GPU.
• 13–14B parameter models: 16–24 GB RAM, noticeably better with a GPU with 8+ GB VRAM.
• 30–34B parameter models: 32 GB+ RAM or a GPU with 16–24 GB VRAM.
• 70B+ parameter models: typically need a GPU with 40–48 GB VRAM, or heavy quantization with patience for slower CPU inference.

Step 1: Choose a runtime

For most people getting started, Ollama is the simplest path — it handles downloading, quantization formats, and gives you both a CLI and a local HTTP API out of the box.

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows: download the installer from ollama.com

Alternatives worth knowing:

• llama.cpp — the underlying engine many tools build on; more manual but very configurable.
• LM Studio — a GUI app, good if you prefer clicking over the terminal.
• vLLM — built for throughput on GPU servers, overkill for a single laptop but the standard choice if you're serving multiple users from your own hardware.

Step 2: Pull a model

With Ollama, pulling and running a model is one command:

ollama pull llama3.1:8b
ollama run llama3.1:8b

This drops you into an interactive prompt. Other solid starting points depending on your hardware and use case: mistral, qwen2.5, phi3, or codellama for coding tasks. Smaller models (1–3B) run on almost anything but lose reasoning quality; 7–8B is the sweet spot for most local setups.

Step 3: Call it like an API

Ollama exposes a local HTTP endpoint by default, so you can integrate it into scripts the same way you'd call any REST API:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Summarize the plot of Dune in two sentences.",
  "stream": false
}'

const res = await fetch("http://localhost:11434/api/generate", {
  method: "POST",
  body: JSON.stringify({
    model: "llama3.1:8b",
    prompt: "Write a haiku about databases.",
    stream: false
  })
});
const data = await res.json();
console.log(data.response);

This is enough for prototyping, personal tools, or scripts that don't need to leave your machine.

Step 4: Tune for your hardware

A few practical adjustments once the basic setup works:

• Quantization level — lower-bit quantizations (Q4, Q5) trade a small amount of accuracy for a big reduction in RAM/VRAM use. Start with Q4_K_M unless you have a strong reason not to.
• Context length — longer context windows consume more memory. Don't set it higher than you actually need.
• GPU offloading — if you have a GPU but limited VRAM, most runtimes let you offload only some layers to the GPU and keep the rest on CPU, which is slower but avoids out-of-memory errors.
• Batch size / concurrency — local setups are usually single-user; if you try to serve multiple requests at once, response times will degrade fast unless you're on real GPU hardware.

When local setups stop being enough

Local LLMs are great for experimentation, offline work, and privacy-sensitive prototyping. They start showing limits once you need:

• Model quality that only the largest hosted models currently provide.
• Multiple team members hitting the same backend reliably.
• Streaming responses, tool use, and structured usage tracking without building that infrastructure yourself.
• Predictable uptime without babysitting a machine.

At that point, many teams keep local models for offline/dev work and move production traffic to a hosted API. If your team already has Claude access, SubToAPI turns that access into a standard HTTPS API — you get application API keys (sub_live_...), streaming, tool use, and usage metadata in one dashboard, without standing up your own inference server. It's a way to get API-style access without managing GPUs or quantization tradeoffs. Setup takes a few minutes: see the quickstart, the messages endpoint docs, and details on streaming and tool use. Plans start at €9/month, listed on the pricing page, with a free trial at signup.

The two approaches aren't mutually exclusive — a lot of developers prototype locally with Ollama, then point production code at a hosted endpoint once they need reliability or team access.

Questions

Do I need a GPU to run an LLM locally? No. Quantized 7–8B models run acceptably on CPU with 16 GB of RAM, though a GPU significantly speeds up response time. A GPU becomes more important as model size grows past 13B parameters.

Which local model should I start with? Llama 3.1 8B or Mistral 7B are good defaults — capable, well-documented, and light enough to run on most modern laptops with Ollama or LM Studio.

Is a local LLM as good as a hosted one like Claude or GPT-4? Not currently. Open local models are improving fast, but the largest hosted models still lead on reasoning, coding, and instruction-following. Local setups are strongest for privacy, offline use, and low-cost experimentation.