Fine-Tuning & Post-Training

When to fine-tune vs prompt vs RAG

Reach for the cheapest tool that works, in this order:

Prompting / in-context learning. Change behavior with instructions and a few examples. Zero training and instant iteration. Try this first.
RAG (RAG & Retrieval). Inject knowledge the model lacks. Use when facts are large, dynamic, or must be cited.
Fine-tuning. Change the model's default behavior: output format/schema adherence, tone/persona, a narrow skill, following domain conventions, or compressing a long prompt into learned weights (latency/cost win). Also for teaching structure or style that's hard to specify in words.

The classic mistake is fine-tuning to add facts. It is an expensive, leaky way to do what RAG does better, and it bakes in staleness. Fine-tune for form and behavior and retrieve for knowledge. These combine: a fine-tuned model plus RAG is common.

The post-training stack

Modern "post-training" (everything after base pretraining) has a few layers:

SFT (Supervised Fine-Tuning). Train on curated (prompt, ideal-response) pairs. The workhorse. It teaches the model to imitate demonstrations. Data quality dominates the outcome.
Preference optimization. Align to human/AI preferences over pairs of responses:
- RLHF (Reinforcement Learning from Human Feedback). Train a reward model on preference data, then optimize the policy with RL (e.g., PPO). It is powerful but complex and unstable to run.
- DPO (Direct Preference Optimization). Skips the separate reward model and RL loop and optimizes directly on preference pairs with a simple classification-style loss. It is far easier to run than PPO and is the common default for open-model alignment. Variants like ORPO, KTO, and SimPO trade off data format and stability.
- RLAIF / Constitutional-style. Use an AI model to generate the preference labels, which reduces human labeling cost.
- RLVR (RL from Verifiable Rewards). For tasks with checkable answers (math, code, tool use), reward correctness directly. Central to 2025 to 2026 reasoning-model training.

Parameter-efficient fine-tuning (PEFT): LoRA / QLoRA

Full fine-tuning updates every weight, which requires enormous memory and compute and a full model copy per task. PEFT methods freeze the base model and train a small number of new parameters instead.

LoRA (Low-Rank Adaptation). Freeze the base weights and inject small trainable low-rank matrices into (typically) the attention/projection layers. You train and ship a few megabytes of "adapter" instead of a full model, can host many adapters over one base, and lose little quality on most tasks. The dominant PEFT method.
QLoRA. LoRA on top of a quantized (commonly 4-bit, NF4) frozen base, so the base's memory footprint collapses. This is what lets people fine-tune large models on a single consumer/prosumer GPU. You pay a little training-speed and precision cost for a large memory saving. Variants like DoRA push quality closer to full fine-tuning.
PEFT more broadly (adapters, prefix/prompt tuning) is packaged in Hugging Face's peft library.

Tooling

Tool	Best at	Watch out
Unsloth	Fast, cheap single-GPU tuning: custom kernels give large speedups and big VRAM cuts for LoRA/QLoRA, plus ready notebooks	Single-GPU focus (multi-GPU historically more limited)
Axolotl	Reproducible, scalable training jobs without writing loops: config-driven (YAML), SFT/DPO/etc. across many models and multi-GPU	Broad config surface, so a learning curve
Hugging Face TRL	Custom pipelines and new algorithms: reference SFT, DPO, GRPO/PPO, and reward modeling; integrates with `peft`/`accelerate`	More code and moving parts than the wrappers
LLaMA-Factory	Broad model/method coverage with a UI	Many knobs; verify recipes for your model
torchtune	Native-PyTorch fine-tuning recipes	Development wound down in 2025 (maintenance-only), so treat it as a reference, not a live default
MLX-LM	Fine-tuning on Apple Silicon (MLX)	Mac-only, smaller-scale
veRL	Scaled RLVR/GRPO/PPO post-training (HybridFlow); the go-to for reasoning-model RL at scale	Heavier infra (Ray-based); overkill for simple SFT/LoRA

GPU / precision constraints

The practical question is always "will it fit in my GPU's memory?" A rough guide:

Just running the model (inference). Weights need about 2 bytes × parameters at FP16, so a 7B model is ≈ 14 GB, plus some headroom for the KV-cache. Quantizing to 4-bit roughly quarters that, so the same 7B fits in ~4 GB.
Full fine-tuning. Much heavier: you need room for the weights plus the gradients, optimizer state, and activations, which together run several times the model's size. This is why full fine-tuning usually needs a multi-GPU A100/H100-class setup.
QLoRA (the cheap path). Loads the base model in 4-bit and trains only small adapters on top, so a mid-size model fine-tunes on a single 24 GB card. This is what puts fine-tuning within reach of one consumer GPU.

Precision formats you'll run into: BF16 (the training default), FP8 (newer, for faster training/inference on Hopper/Blackwell), and 4-bit NF4/INT4 (QLoRA and low-memory inference). The lower you go, the more accuracy you trade for memory, so validate on your eval set.