How to Train Your Own LLM Locally (Realistically)

If you searched "how to train your own LLM locally," you probably want one of two things: either you want to fine-tune an existing open-weight model on your own data, or you're picturing something closer to building GPT-4 from scratch in your bedroom. The honest answer is that only the first one is realistic for almost everyone, and this guide focuses entirely on that.

Training a foundation model from scratch requires trillions of tokens, thousands of GPUs, and months of engineering time — it's not a "local" project. What you actually want is fine-tuning: taking a pretrained open model (Llama, Mistral, Qwen, Gemma, Phi) and adapting it to your task, tone, or dataset on hardware you own. This is genuinely achievable on a single consumer or workstation GPU, and this article walks through exactly how.

What "training" actually means here

There are three distinct things people lump under "training an LLM," and they have wildly different costs:

• Pretraining — teaching a model language from raw text, from random weights. Requires massive compute. Not local.
• Full fine-tuning — updating every parameter of an existing model on your dataset. Possible locally only for small models (1B–3B params) with a good GPU.
• Parameter-efficient fine-tuning (PEFT) — methods like LoRA and QLoRA that freeze the base model and train small adapter layers. This is what most local training actually means, and it works on a single GPU even for 7B–13B models.

If your goal is "make an LLM speak like my support docs" or "specialize a model for legal contracts," you want LoRA/QLoRA fine-tuning, not pretraining.

Hardware you actually need

Rough guidelines for QLoRA fine-tuning (4-bit quantized base model, LoRA adapters):

| Model size | Minimum VRAM | Realistic setup | |---|---|---| | 1–3B | 8 GB | Any modern gaming GPU | | 7B | 12–16 GB | RTX 3090/4090, or Apple Silicon with 32GB+ unified memory | | 13B | 24 GB | RTX 4090, A5000 | | 30B+ | 48 GB+ | Multi-GPU or cloud rental |

CPU-only training is possible but slow enough that it's rarely worth it beyond toy experiments. If you don't own a GPU with at least 12GB VRAM, renting one by the hour is usually cheaper than buying hardware you'll use a handful of times.

Step 1: Pick a base model

Start small. A 7B model fine-tuned well will often outperform a 13B model fine-tuned poorly, and it trains faster. Good starting points: Mistral-7B, Llama-3-8B, or Qwen2.5-7B, all available with permissive-enough licenses for most uses.

Step 2: Prepare your dataset

Format matters more than volume. Most fine-tuning frameworks expect instruction-style JSONL:

{"instruction": "Summarize the ticket below in one sentence.", "input": "Customer reports login failures...", "output": "Customer cannot log in due to expired session tokens."}

A few hundred high-quality, well-formatted examples usually beat tens of thousands of noisy ones. Deduplicate, check for consistent formatting, and hold out 10–15% for evaluation.

Step 3: Set up the training stack

The two most common local paths:

Hugging Face + PEFT — full control, more setup work.

pip install transformers peft bitsandbytes accelerate datasets

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

bnb_config = BitsAndBytesConfig(load_in_4bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    quantization_config=bnb_config,
    device_map="auto"
)

lora_config = LoraConfig(
    r=16, lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05, task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)

Axolotl or Unsloth — YAML-driven, less boilerplate, and Unsloth in particular is noticeably faster on single-GPU setups. If you're doing your first fine-tune, start here rather than hand-rolling the training loop.

Step 4: Train and monitor

Watch training loss and, more importantly, run manual spot checks against your held-out examples every few hundred steps. A dropping loss curve doesn't guarantee useful outputs — overfitting on small datasets is common. Two to three epochs is usually enough for LoRA; more often makes things worse, not better.

Step 5: Merge and serve locally

Once you're happy with the adapter, merge it into the base weights and quantize for inference:

python merge_lora.py --base_model mistral-7b --adapter ./lora-out --output ./merged-model

Then serve with llama.cpp or Ollama for local inference:

ollama create my-model -f Modelfile
ollama run my-model

This gets you a model running entirely on your machine, fine-tuned on your data, with no external API calls.

When local training isn't the right tool

Local fine-tuning is great for narrow, well-defined tasks — style adaptation, domain vocabulary, structured output formats. It's a poor fit if you need frontier-level reasoning, long-context handling, or reliable tool use, since open 7B–13B models still trail closed models like Claude on those dimensions, and closing that gap with fine-tuning alone is hard.

If what you actually need is a production-grade model behind a stable API — with streaming, tool calling, and usage metadata — rather than a custom-trained local model, it's often faster to skip the GPU rental and use an existing model through an API. SubToAPI turns a Claude subscription into an HTTPS API with application keys, streaming, and team seats, which is a reasonable middle ground if fine-tuning turns out to be more infrastructure than your project needs. Check the quickstart or pricing if that's a better fit than managing your own training pipeline.

questions

Do I need a GPU to train an LLM locally? Practically, yes. CPU training is possible for tiny models but too slow to be useful for anything above a few hundred million parameters. A single consumer GPU with 12GB+ VRAM is enough for QLoRA fine-tuning of 7B models.

How much data do I need to fine-tune a model? For task-specific fine-tuning, a few hundred to a few thousand high-quality, well-formatted examples is often enough. Quality and consistency of format matter more than raw volume.

Can I train a model as good as GPT-4 or Claude locally? No. Those are pretrained on massive infrastructure you can't replicate at home. Local training realistically means adapting an existing open model to a narrow task, not building a frontier general-purpose model.