How to Train an LLM Locally: What You Actually Need

Training an LLM locally almost never means building GPT-4 from scratch on your desktop. It means one of three realistic things: fine-tuning an existing open-weight model on your own data, running a parameter-efficient fine-tune (LoRA/QLoRA) on consumer GPUs, or continuing pretraining on a small model for a narrow domain. This guide covers all three, with the hardware, tools, and steps you actually need.

If your goal is "get a model that talks like my product docs" or "classify support tickets the way my team does," you don't need a data center — you need the right base model, a clean dataset, and a fine-tuning method sized to your GPU. If your goal is production inference at scale rather than experimentation, local training is the wrong tool entirely; you'd be better served calling a hosted API.

Full Pretraining vs. Fine-Tuning: Pick the Right Target

Full pretraining from scratch requires trillions of tokens, thousands of GPU-hours, and a data pipeline most teams will never build. Skip it unless you're a research lab.

Fine-tuning an open-weight model (Llama 3, Mistral, Qwen, Gemma) is what almost everyone means by "train an LLM locally." You start from weights that already understand language and adjust them toward your task or tone.

Parameter-efficient fine-tuning (LoRA/QLoRA) trains a small set of adapter weights instead of the full model. This is the realistic path on a single consumer GPU — a 7B–8B model can be fine-tuned with LoRA on 16–24GB of VRAM, and QLoRA (4-bit quantized base weights) pushes that down further.

For most local training projects, QLoRA on a 7B–13B model is the sweet spot: good enough quality, achievable on one GPU, fast iteration.

Hardware You Actually Need

| Goal | Minimum GPU | Notes | |---|---|---| | QLoRA fine-tune, 7B model | 1x 12–16GB VRAM | RTX 3060/4060 Ti class works | | LoRA fine-tune, 7B model | 1x 24GB VRAM | RTX 3090/4090 or A5000 | | LoRA fine-tune, 13B–34B | 1x 48GB or multi-GPU | A6000, dual 4090 with sharding | | Full fine-tune, 7B+ | Multi-GPU, 80GB+ per card | A100/H100 territory |

CPU-only training is possible for tiny models or heavily quantized adapters, but expect training runs measured in days instead of hours. If you don't own suitable hardware, renting GPU time (Lambda, RunPod, Vast.ai) is usually cheaper than buying — you still run the same local training scripts, just on rented boxes.

Tools That Do the Heavy Lifting

You rarely write a training loop from scratch anymore. Pick one of these:

• Axolotl — YAML-config-driven fine-tuning, supports LoRA/QLoRA, handles most open-weight architectures out of the box.
• Unsloth — optimized kernels that cut VRAM use and training time significantly for LoRA/QLoRA on popular models.
• LLaMA-Factory — GUI plus CLI, good for teams that want fewer moving parts.
• Hugging Face PEFT + Transformers + bitsandbytes — the lower-level building blocks if you want full control.

Step-by-Step: Fine-Tuning a Model Locally

1. Pick a base model. Start with something already close to your task's language and size (e.g., Llama 3 8B Instruct for a chat-style assistant).
2. Prepare your dataset. Format as instruction/response pairs (JSONL is standard):

{"instruction": "Summarize this ticket", "input": "...", "output": "..."}

Aim for at least a few hundred clean, task-representative examples for LoRA fine-tuning; thousands if you want reliable generalization.

1. Install dependencies.

pip install torch transformers peft bitsandbytes accelerate datasets trl

1. Load the base model in 4-bit and attach a LoRA adapter:

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype="bfloat16")

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

lora_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)

1. Train with the trl library's SFTTrainer or a Hugging Face Trainer loop, pointing at your JSONL dataset.
2. Evaluate on held-out examples — don't trust loss curves alone; manually check outputs against real prompts.
3. Merge adapters (optional) back into the base weights if you want a single deployable checkpoint, or keep them separate for easy swapping.

When Local Training Isn't the Right Move

Local training makes sense when you need a model that behaves differently — a specific tone, a narrow domain vocabulary, or a classification task baked into the weights. It does not make sense when what you actually need is reliable inference infrastructure: streaming responses, usage tracking, team API keys, tool-calling support.

If your product just needs to call an LLM reliably from your backend — without managing GPUs, uptime, or fine-tuning pipelines — that's a different problem, and it's usually faster to solve with a hosted API. SubToAPI turns your existing Claude access into a standard HTTPS API with application keys (sub_live_...), streaming, tool use, and per-key usage metadata, so you're not maintaining local inference infrastructure just to ship a feature. Check the quickstart or pricing if that's closer to what you actually need.

Many teams do both: fine-tune locally for a specific narrow task, and use a hosted API for general-purpose chat and reasoning in the same product.

questions

Do I need a GPU to train an LLM locally? For anything beyond toy models, yes. QLoRA fine-tuning on a 7B model is feasible on a single 12–16GB consumer GPU; CPU-only training is possible but very slow for anything larger than small models.

How much data do I need to fine-tune a model? A few hundred high-quality, task-representative instruction/response pairs is enough to see meaningful behavior change with LoRA. Thousands of examples improve generalization but quality matters more than volume.

What's the difference between fine-tuning and training from scratch? Fine-tuning adjusts an existing pretrained model's weights (or a small adapter) toward your task and requires modest compute. Training from scratch builds a model's language understanding from raw text and requires massive datasets and GPU clusters — not practical for individuals or most teams.