How to Run an LLM Locally: A Practical Walkthrough

Running an LLM locally means downloading a model's weights and executing inference on your own machine, without sending prompts to a hosted API. The short version: pick a runtime (Ollama is the easiest starting point), pull a quantized model sized to your RAM/VRAM, and start chatting through a local CLI or HTTP endpoint. The rest of this guide covers the actual steps, the hardware math, and where local inference stops making sense.

Running locally gives you full control over data (nothing leaves your machine), no per-token costs, and the ability to work offline. The tradeoffs are real too: local models are almost always smaller and less capable than frontier hosted models, and you're responsible for the infrastructure — updates, GPU drivers, disk space for multiple model versions.

Step 1: Pick a runtime

You don't need to compile anything from source unless you want to. Three options cover almost every use case:

• Ollama — a single binary that wraps model downloading, quantization formats, and a local REST API. Best for getting started fast.
• llama.cpp — the underlying engine many tools build on. Lower-level, more control over CPU/GPU offload, threading, and context size.
• LM Studio — a desktop GUI for people who want a chat window and a model browser without touching a terminal.

For developers, Ollama is the practical default because it exposes a local HTTP API on localhost:11434 that behaves close enough to other chat completion APIs to slot into existing code.

Step 2: Check your hardware against model size

This is the step people skip and then wonder why everything is slow or crashes. Model size on disk roughly correlates with RAM/VRAM needed to run it, especially at 4-bit quantization:

| Model size | Quantized (4-bit) size | Minimum usable RAM | |---|---|---| | 3B | ~2 GB | 8 GB | | 7–8B | ~4.5 GB | 16 GB | | 13–14B | ~8 GB | 24–32 GB | | 34B+ | ~20 GB | 32–64 GB, GPU recommended |

If you have a discrete GPU with enough VRAM, inference will be dramatically faster than CPU-only. Without one, expect single-digit tokens/second on anything above 7B — usable for testing, painful for interactive use.

Step 3: Install and pull a model

With Ollama installed:

# install (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh

# pull a model
ollama pull llama3.1:8b

# run it interactively
ollama run llama3.1:8b

That gives you a chat prompt in the terminal immediately. For programmatic access, Ollama exposes a local API:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Summarize this changelog in three bullets."
}'

This is the same pattern you'd use to wire a local model into a script or app — swap the endpoint, keep the rest of your integration logic.

Step 4: Choose the right quantization

Model files come in different quantization levels (Q4_K_M, Q5_K_M, Q8_0, etc.). Lower quantization = smaller file and faster inference, at the cost of some output quality. For most day-to-day use, Q4_K_M is the sweet spot: noticeably smaller than full precision, minimal quality loss for general tasks. Reserve Q8 or full precision for cases where you're benchmarking output quality specifically.

Step 5: Decide what you're actually optimizing for

Local LLMs make sense when:

• Data cannot leave your infrastructure (compliance, contracts, sensitive internal docs)
• You're prototyping offline or in an air-gapped environment
• You want zero marginal cost for high-volume, low-stakes tasks (classification, extraction, drafting)

Local LLMs are the wrong tool when:

• You need frontier-level reasoning, long-context accuracy, or strong coding ability — open-weight models at sizes that fit on consumer hardware still lag behind the largest hosted models on hard tasks.
• You need to scale to many concurrent users. A single machine running one model instance doesn't parallelize the way a hosted API does.
• You want to ship a product with tool use, streaming, and multi-seat access control without building that infrastructure yourself.

For that last case, a lot of teams end up wanting the reliability and features of a managed API without giving up an existing Claude subscription they're already paying for. SubToAPI turns your Claude access into an HTTPS API with application keys (sub_live_...), streaming, tool use, and usage metadata — useful when local inference has proven the workflow but you need production-grade output quality and multi-user access. You can see the request shape in the quickstart and the messages docs.

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4",
    "max_tokens": 512,
    "messages": [{"role": "user", "content": "Draft a release note from this diff."}]
  }'

A realistic workflow

Most developers end up with a hybrid setup: local models for iteration, drafting, and anything sensitive, hosted models for tasks that need higher accuracy or need to run at scale. Prototype your prompt logic against Ollama locally where iteration is free, then point the same request shape at a hosted endpoint once you need better output quality or you're shipping to real users. If you're already running Claude interactively and want that same model behind an API for your app, signup takes a few minutes, and pricing starts at €9/month for solo use.

Troubleshooting common issues

• Model runs but is extremely slow: you're likely running a model too large for your available RAM/VRAM and it's swapping to disk. Try a smaller model or a lower quantization.
• Out of memory errors: reduce context window size, close other memory-heavy applications, or switch to a 4-bit quantized variant.
• Output quality is worse than expected: check the quantization level and consider that smaller open-weight models genuinely underperform larger hosted models on complex reasoning — this isn't a configuration problem, it's a model capability limit.

Do I need a GPU to run an LLM locally?

No. CPU-only inference works fine for smaller models (3B–8B) at modest speeds. A GPU with enough VRAM makes inference significantly faster and lets you run larger models, but it's not required to get started.

What's the easiest way to run an LLM locally?

Install Ollama and run ollama run llama3.1:8b. It handles downloading, quantization, and gives you both a terminal chat and a local API in one step.

Can a local LLM match Claude or GPT-4 quality?

Not currently for complex reasoning, long-context tasks, or coding at a high level. Open-weight models that fit on consumer hardware are good for drafting, extraction, and classification, but frontier hosted models still lead on hard tasks.