How to Train a Local LLM: Methods That Work

"Training a local LLM" usually means one of two very different things: pretraining a model from scratch, or fine-tuning an existing open-weight model on your own data. Almost nobody outside a well-funded lab does the first one — pretraining a competitive model requires hundreds of GPUs and months of compute. What developers actually want, and what this guide covers, is fine-tuning: taking a model like Llama, Mistral, or Qwen and adapting it to a specific task, tone, or domain using a local GPU.

The short answer: pick a small-to-mid-size open model, use a parameter-efficient method like LoRA or QLoRA, prepare a clean instruction dataset of a few hundred to a few thousand examples, and train with a tool like Axolotl or Unsloth on a single consumer GPU (or a rented one). Below is the full workflow, including where full fine-tuning still makes sense and where it doesn't.

Decide what "training" actually solves

Before touching any code, be clear about the goal, because fine-tuning is not always the right tool:

• You want a specific output format or tone (structured JSON, a support-bot voice, a coding style) → fine-tuning works well.
• You want the model to know facts from your documents → retrieval (RAG) is usually cheaper and more accurate than training. Facts baked into weights go stale and can't be updated without retraining.
• You want faster/cheaper inference on a narrow task → fine-tuning a small model to match a bigger model's behavior on that task is a solid use case (distillation).
• You want a generally smarter model → you can't get there with a small local dataset. Use a stronger hosted model instead.

If your real need is "call a strong model from my app without managing infra," you don't need to train anything — a service like SubToAPI turns Claude access into a standard HTTPS API with streaming and tool use, which is often the faster path to production. See the quickstart if that's closer to what you're solving. But if you specifically need a customized, self-hosted model, keep reading.

Full fine-tuning vs. parameter-efficient fine-tuning

Full fine-tuning updates every weight in the model. It gives the most control but needs enough VRAM to hold the model, gradients, and optimizer states simultaneously — for a 7B model that's typically 80GB+ with standard optimizers, out of reach for most local setups.

LoRA (Low-Rank Adaptation) freezes the base model and trains small additional matrices injected into attention layers. It captures most of the benefit of full fine-tuning with a fraction of the memory.

QLoRA quantizes the base model to 4-bit and trains LoRA adapters on top, cutting memory further. This is what makes fine-tuning a 7B–13B model practical on a single 24GB GPU (e.g., an RTX 4090), and even 7B models on 12GB with tighter settings.

For almost all local training projects, start with QLoRA. Move to full fine-tuning only if you have measured evidence that LoRA isn't capturing the behavior you need.

Preparing your dataset

Dataset quality matters more than dataset size for fine-tuning. A few hundred well-written examples that consistently demonstrate the behavior you want will outperform tens of thousands of noisy, inconsistent ones.

Structure your data as instruction/response pairs, matching the chat template the base model expects:

{"messages": [
  {"role": "system", "content": "You are a support agent for Acme Cloud."},
  {"role": "user", "content": "My deploy is stuck at 90%."},
  {"role": "assistant", "content": "That's usually a health-check timeout. Check your /health endpoint returns 200 within 30s of boot."}
]}

Practical rules:

• Keep formatting consistent across every example — inconsistent structure is the most common cause of bad fine-tunes.
• Deduplicate aggressively. Repeated near-identical examples cause overfitting fast.
• Hold out 10–15% of the data for validation so you can catch overfitting during training.
• If you're distilling behavior from a stronger model, generate your training examples with that model, then review and clean them by hand — don't ship raw synthetic output untouched.

Choosing hardware and tooling

For local training, realistic hardware tiers look like this:

• 8–12GB VRAM: QLoRA on 3B–7B models, small batch sizes, gradient accumulation.
• 24GB VRAM (RTX 3090/4090): QLoRA on 7B–13B comfortably, LoRA on 7B.
• 48GB+ or multi-GPU: LoRA on larger models, or full fine-tuning on smaller ones.

If you don't have the hardware, renting a single GPU by the hour (RunPod, Lambda, Vast.ai) for a few hours is usually cheaper than buying one, and the workflow is identical — it's still "local" in the sense that you control the training run end to end.

Tooling that handles most of the boilerplate:

• Axolotl — YAML-config-driven fine-tuning, supports LoRA/QLoRA/full fine-tuning, widely used for open models.
• Unsloth — optimized kernels that cut training time and memory significantly for LoRA/QLoRA on popular architectures.
• Hugging Face peft + transformers — lower-level but gives full control if you want to write your own training loop.
• llama.cpp — not for training, but useful afterward to quantize and run your fine-tuned model efficiently.

A minimal Axolotl-style config for QLoRA looks roughly like:

base_model: mistralai/Mistral-7B-v0.1
load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
datasets:
  - path: ./data/train.jsonl
    type: chat_template
val_set_size: 0.1
num_epochs: 3
micro_batch_size: 2
gradient_accumulation_steps: 8
learning_rate: 0.0002

Run for a few epochs, watch validation loss, and stop before it starts climbing — that's your overfitting signal.

Evaluating the result

Loss curves tell you training is stable, not that the model is good. Build a small held-out test set of realistic prompts and manually score outputs (pass/fail against a rubric, or side-by-side against the base model) before deploying anything. For task-specific fine-tunes, automated checks — does the output parse as valid JSON, does it contain required fields, does it stay under a length limit — catch regressions faster than eyeballing text.

When to skip training entirely

If your actual goal is shipping a feature that calls an LLM reliably — with retries, streaming, and usage tracking — fine-tuning adds weeks of iteration for a benefit you may not need. Many teams get further faster by using a strong general model through an API and handling customization with prompting, tool use, and RAG. SubToAPI exposes Claude through a standard Messages-style API with streaming, tool calling, and per-key usage metadata, so you can prototype the product first and revisit local fine-tuning later if there's a clear, measured gap. Check pricing or the tools docs if that's a path worth comparing against.

Questions

Do I need a powerful GPU to fine-tune a local LLM? Not necessarily. QLoRA makes 7B–13B model fine-tuning feasible on a single 24GB consumer GPU, and smaller models run on 8–12GB. Renting a GPU by the hour is a practical alternative to buying hardware.

How much data do I need to fine-tune a model? Often a few hundred to a few thousand clean, consistently formatted examples outperform much larger noisy datasets. Quality and consistency matter more than volume for most fine-tuning tasks.

Should I fine-tune or use retrieval (RAG) for domain knowledge? Use RAG when the goal is giving the model access to facts or documents — it's cheaper and easier to update. Fine-tune when you need a specific output format, tone, or task behavior baked into the model's responses.