Setting Up a Local LLM: Hardware, Tools, and Tradeoffs
Setting up a local LLM means installing model-serving software on your own machine, downloading a quantized model file, and running inference without sending data to a cloud API. The whole process takes under 15 minutes on a modern laptop if you use a tool like Ollama, or longer if you compile llama.cpp yourself and want fine control over quantization and context length.
This guide walks through the actual setup steps — hardware requirements, software choices, model selection — and is honest about the tradeoffs: local LLMs give you privacy and zero per-token cost, but they trade off raw quality and speed compared to frontier hosted models.
Step 1: Check Your Hardware
Local inference is bottlenecked by memory bandwidth, not just raw compute. Before picking a model:
- RAM: 16GB minimum for 7B-parameter models at 4-bit quantization. 32GB+ if you want 13B–14B models comfortably.
- GPU VRAM: An 8GB GPU (RTX 3060/4060) handles 7B models well with GPU offload. 24GB (RTX 3090/4090) opens up 30B-class models.
- Apple Silicon: M1/M2/M3 Macs with 16GB+ unified memory run 7B–13B models surprisingly well thanks to shared memory architecture — no dedicated VRAM needed.
- CPU-only: Works, but expect 2–10 tokens/second on 7B models depending on core count. Fine for testing, painful for anything interactive.
If you don't have this hardware, skip ahead to the "when local isn't the right choice" section below.
Step 2: Pick a Serving Tool
Three realistic options, in order of setup friction:
Ollama — the fastest path. Single binary, handles model downloading, quantization, and a local HTTP API automatically.
curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.1That second command downloads the model on first run and drops you into a chat prompt. Ollama also exposes a REST API on localhost:11434 so you can call it from code.
LM Studio — a GUI wrapper around llama.cpp. Good if you want a chat interface plus model browsing without touching a terminal. Also exposes an OpenAI-compatible local server.
llama.cpp directly — most control, most setup work. You compile it, download GGUF model weights manually, and tune context size, batch size, and GPU layers yourself:
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make LLAMA_CUDA=1 # or LLAMA_METAL=1 on Mac
./main -m models/llama-3.1-8b.Q4_K_M.gguf -p "Explain quantization" -n 200For most developers, start with Ollama. Drop to llama.cpp only if you need a specific quantization format or an unsupported hardware backend.
Step 3: Choose a Model and Quantization Level
Model choice matters more than serving tool. Rough guidance for late-2024/2025-era open models:
- 7B–8B models (Llama 3.1 8B, Mistral 7B, Qwen2.5 7B): good general chat quality, run on almost anything with 16GB RAM.
- 13B–14B: noticeably better reasoning, needs 16–24GB RAM/VRAM.
- 30B+: approaches mid-tier hosted model quality, needs 24GB+ VRAM or heavy RAM offload with slower speed.
Quantization trades precision for size. Q4_K_M is the standard sweet spot — roughly 4 bits per parameter, small quality loss, big memory savings. Q8 is near-lossless but twice the size; Q2/Q3 save more memory but degrade output noticeably.
Step 4: Test It Like an API
Once running, Ollama and LM Studio both expose local HTTP endpoints so you can integrate the model into scripts the same way you'd call any LLM API:
curl http://localhost:11434/api/generate -d '{
"model": "llama3.1",
"prompt": "Summarize the benefits of quantization",
"stream": false
}'This is useful for prototyping — you can build your app against a local model, then swap in a hosted API later for production traffic without rewriting your request logic much, since most local servers mimic OpenAI-style chat completion formats.
When Local Isn't the Right Choice
Local setups make sense for privacy-sensitive prototyping, offline use, or avoiding per-token costs during heavy experimentation. They fall short when you need:
- Frontier-level reasoning or coding quality (open 7B–30B models still lag well behind top-tier hosted models on hard tasks)
- Reliable throughput without babysitting GPU memory
- Multi-user access with authentication, rate limits, and usage tracking
- Team-wide access without every developer configuring their own machine
If your actual goal is "I want an API key and predictable HTTPS access to a capable model," running Claude through your existing subscription via SubToAPI is often simpler than maintaining local infrastructure. It turns your Claude access into a standard API — you get an application key (sub_live_...), streaming, tool use, and usage metadata per key, with team seats if you need shared access. Setup is a signup flow at /signup, not a hardware audit.
const res = await fetch("https://api.subtoapi.app/v1/messages", {
method: "POST",
headers: {
"Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
model: "claude-3-5-sonnet-latest",
max_tokens: 500,
messages: [{ role: "user", content: "Summarize the benefits of quantization" }]
})
});Plans start at €9 for solo use, with team pricing at €19–€49/seat — see /pricing. For request formats and streaming setup, check /docs/messages and /docs/streaming.
A practical pattern many teams use: run a local LLM for offline dev/testing, and route production or team traffic through a hosted API. You get the best of both without over-investing in either.
Questions
Do I need a GPU to run a local LLM? No. CPU-only inference works for 7B models, just slower (2–10 tokens/sec). A GPU with 8GB+ VRAM or Apple Silicon unified memory makes it noticeably faster and enables larger models.
What's the easiest way to get started? Install Ollama and run ollama run llama3.1. It handles downloading, quantization, and serving automatically — no manual compilation needed.
Is a local LLM as good as a hosted one like Claude? Not currently for complex reasoning or coding tasks. Open 7B–30B models are solid for chat and simple tasks but generally lag behind frontier hosted models on harder work.