Introduction
The question comes up in every AI architecture conversation: should we fine-tune a model or use RAG?
It's the wrong question. Fine-tuning and RAG solve different problems, they're not competing strategies. The real question is: what is actually causing your model to underperform? Is it a knowledge problem (missing information), a behavior problem (wrong tone, format, or style), or a reasoning problem (the base model isn't capable of the task)?
RAG solves knowledge problems. It injects external, up-to-date information into the model's context at query time. It's runtime retrieval.
Fine-tuning solves behavior problems. It changes how the model responds — its style, its output format, its reasoning patterns — by training on examples of the behavior you want.
Prompting solves a surprising number of both. Before reaching for either, the correct answer is often more carefully designed prompts with better examples and clearer instructions.
This post is for engineers and ML practitioners who need to make the right call for their specific use case. We'll cover what fine-tuning actually changes inside the model, how LoRA makes it practical at production scale, when each approach is the correct choice, and the production considerations that determine whether a fine-tuned model is an asset or a maintenance burden.
What Fine-Tuning Actually Changes
When you fine-tune a language model, you're adjusting the model's weights — the billions of numerical parameters that determine how the model responds to input. Training on new examples updates those weights to make the model more likely to produce the types of outputs you showed it and less likely to produce outputs that differ.
This is fundamentally different from RAG, which doesn't change the model at all. RAG changes the model's input (what information is in the context). Fine-tuning changes the model itself (how it processes and responds to any input).
What fine-tuning improves:
- Output format consistency — models that reliably produce JSON, specific markdown formats, or structured templates
- Domain-specific tone and style — a customer service bot that always uses company language, a medical summarizer that uses clinical terminology correctly
- Task specialization — models that have internalized specialized reasoning patterns (legal document analysis, code generation in a specific framework)
- Reduced prompt length — behaviors that require lengthy few-shot examples in a prompt can be internalized through fine-tuning, reducing per-request token cost
What fine-tuning doesn't improve:
- Knowledge freshness — fine-tuned knowledge is frozen at training time. A model fine-tuned on your product docs in January won't know about the March product update.
- Factual grounding — fine-tuning can teach a model how to reason, but it still hallucinates facts it doesn't have in training
- Long-tail knowledge — fine-tuning requires substantial data. For niche domains with limited examples, RAG with retrieval is more reliable
The Three Fine-Tuning Approaches
Full Fine-Tuning (Rarely Practical in 2026)
Full fine-tuning updates all model weights. For a 70B parameter model at fp16, that's 140GB of model weights, gradients, and optimizer state — easily 400-600GB of GPU memory during training. At $30,000 per H100, the economics rarely work outside of large AI labs.
Full fine-tuning also risks catastrophic forgetting: the new training overwrites general capabilities as the model specializes. A general-purpose model becomes narrow.
For most teams, full fine-tuning is theoretical.
Parameter-Efficient Fine-Tuning (PEFT)
PEFT methods update only a small fraction of the model's parameters — leaving most weights frozen and training only small adapter layers. The result: comparable quality improvement at a fraction of the compute and memory cost.
The dominant PEFT method is LoRA (Low-Rank Adaptation).
LoRA: The Practical Fine-Tuning Method
LoRA's insight: large weight matrices have low intrinsic rank during fine-tuning. The updates needed to specialize a model for a task are much lower-dimensional than the full weight matrix. Instead of updating a full weight matrix W (e.g., 4096×4096), LoRA decomposes the update into two small matrices: W + ΔW, where ΔW = A × B, and A and B have much smaller dimensions (4096×r and r×4096, where rank r is typically 4-64).
This reduces trainable parameters from millions to thousands while capturing nearly all the task-specific information needed.
QLoRA: quantized LoRA. The base model weights are quantized to 4-bit, and LoRA adapters are trained on top. This enables fine-tuning 70B models on consumer hardware (two A100 40GB GPUs, or even a single A100 80GB). The quality degradation from 4-bit quantization is typically small enough that QLoRA results are competitive with full fine-tuning for most tasks.
# Fine-tuning with PEFT + LoRA
# pip install transformers peft datasets accelerate bitsandbytes
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer
import torch
# Load base model in 4-bit for QLoRA
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3-8B-Instruct",
load_in_4bit=True,
torch_dtype=torch.float16,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3-8B-Instruct")
tokenizer.pad_token = tokenizer.eos_token
# Configure LoRA adapters
lora_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=16, # Rank — higher = more parameters, more capacity
lora_alpha=32, # Scaling factor (usually 2× rank)
target_modules=["q_proj", "v_proj", "k_proj", "o_proj"], # Which layers to adapt
lora_dropout=0.05,
bias="none",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 3,407,872 || all params: 8,033,669,120 || trainable%: 0.04242
# Training
training_args = TrainingArguments(
output_dir="./fine-tuned-llama",
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4, # Effective batch size = 16
warmup_steps=100,
learning_rate=2e-4,
fp16=True,
logging_steps=25,
save_strategy="epoch",
report_to="tensorboard",
)
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=train_dataset, # Your formatted dataset
tokenizer=tokenizer,
max_seq_length=2048,
)
trainer.train()
# Save adapter weights only (not the full base model)
model.save_pretrained("./lora-adapters")
The trainable parameter count — 0.04% of the full model — illustrates LoRA's efficiency. You're training 3.4M parameters instead of 8 billion, yet achieving meaningful task-specific improvement.
Dataset Quality: The Actual Bottleneck
The most common fine-tuning mistake: not enough data, or data that's the wrong shape. LoRA can learn from hundreds of examples — but those examples must precisely demonstrate the behavior you want.
Quality > quantity. 200 carefully curated examples of the exact input/output pattern you want the model to learn will outperform 2,000 noisy examples with inconsistent formatting, mixed tones, or off-target outputs.
Dataset format for instruction fine-tuning:
# Training data format: instruction/input/output triplets
training_examples = [
{
"instruction": "Classify this customer message by severity and category",
"input": "My account has been locked and I can't log in. I have an urgent presentation in 2 hours and need access immediately.",
"output": '{"severity": "critical", "category": "account_access", "urgency": "immediate", "sentiment": "distressed", "recommended_action": "escalate_to_agent"}',
},
{
"instruction": "Classify this customer message by severity and category",
"input": "Can you update my billing address?",
"output": '{"severity": "low", "category": "billing", "urgency": "routine", "sentiment": "neutral", "recommended_action": "self_service"}',
},
# ...200-500 more examples
]
# Format into chat template
def format_example(example):
return f"""<|system|>
You are a customer support classifier. Always output valid JSON.
<|user|>
{example['instruction']}
{example['input']}
<|assistant|>
{example['output']}"""
Data collection strategies:
- Distillation from larger models: generate examples using Claude Opus or GPT-4, then fine-tune a smaller model to replicate that behavior. Cost-effective way to transfer capability.
- Human annotation: for high-stakes tasks, have domain experts label examples. Slower and more expensive, but higher quality for nuanced domains.
- Synthetic data generation: for tasks with clear rules (format conversion, classification, templated generation), generate data programmatically.
Fine-Tuning vs RAG vs Prompting: The Decision Matrix
| Criterion | Better Prompt | RAG | Fine-Tuning |
|---|---|---|---|
| Missing recent knowledge | ✗ | ✓ | ✗ |
| Knowledge base > context window | ✗ | ✓ | ✗ |
| Consistent output format | ✓ (few-shot) | ✗ | ✓✓ |
| Domain-specific style/tone | ✓ (system prompt) | ✗ | ✓✓ |
| Reduce per-request token cost | ✗ | ✗ | ✓ |
| Internalize reasoning patterns | ✗ | ✗ | ✓ |
| Low latency (no retrieval step) | ✓ | ✗ | ✓ |
| Knowledge is verifiable/citable | ✗ | ✓ | ✗ |
| Handles data you can't share externally | ✗ | ✗ | ✓ |
The right architecture for most production AI applications combines all three: a carefully designed system prompt establishes persona and constraints, RAG injects relevant factual context, and a fine-tuned adapter shapes the output format and style. They're layers, not alternatives.
DPO: Alignment Fine-Tuning Without Reinforcement Learning
RLHF (Reinforcement Learning from Human Feedback) was the original method for aligning model behavior — teaching models to be helpful, harmless, and honest based on human preference signals. RLHF is complex: it requires training a separate reward model, managing a reinforcement learning loop, and dealing with training instability.
DPO (Direct Preference Optimization), introduced in 2023 and now the dominant alignment method for small-team fine-tuning, simplifies this to a standard supervised learning problem. Instead of a reward model and RL loop, DPO directly optimizes on preference data: pairs of (chosen, rejected) responses to the same prompt.
from trl import DPOTrainer, DPOConfig
# Preference dataset format
preference_data = [
{
"prompt": "How do I center a div in CSS?",
"chosen": "Use flexbox: set `display: flex; justify-content: center; align-items: center;` on the parent. This is the modern, reliable approach that works across all current browsers.",
"rejected": "You can use margin: auto but that only works sometimes. There's also table-cell but nobody does that anymore. The old float trick doesn't really work. Honestly just Google it.",
},
# ... more preference pairs
]
dpo_config = DPOConfig(
beta=0.1, # Temperature for the DPO objective — controls how strongly to prefer chosen vs rejected
learning_rate=5e-7,
num_train_epochs=1,
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
)
dpo_trainer = DPOTrainer(
model=model,
ref_model=ref_model, # A frozen copy of the base model
args=dpo_config,
train_dataset=preference_data,
tokenizer=tokenizer,
)
dpo_trainer.train()
DPO is particularly effective for: reducing verbosity, improving response quality on open-ended tasks, adjusting model persona, and reducing refusal rates on edge cases the base model over-refuses.
Serving Fine-Tuned Models
The LoRA adapter weights are typically small (50-200MB for rank-16 adapters on a 7-8B model) compared to the base model (14-16GB at fp16). Production serving strategies:
Single adapter: merge the LoRA weights into the base model once training is complete. model.merge_and_unload() in PEFT produces a single model file that serves like a normal model — no adapter-handling overhead at inference time.
Multiple adapters, shared base: vLLM supports serving a single base model with multiple LoRA adapters loaded simultaneously. Requests specify which adapter to use. Memory efficient: the 14GB base model is loaded once, and each adapter adds 50-200MB.
# vLLM with multiple LoRA adapters
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3-8B-Instruct \
--enable-lora \
--lora-modules customer-service=./adapters/customer-service \
code-review=./adapters/code-review \
medical-notes=./adapters/medical-notes \
--max-loras 3
Request routing specifies the adapter at the API level — different endpoints or different model parameter values route to the appropriate adapter, all served from the same GPU.
Production Considerations
Evaluation before deployment: fine-tuning can improve specific tasks while degrading general capability. Define an evaluation set before training and measure both target task performance and a general capability benchmark (MMLU, or a custom benchmark reflecting your use case mix) after each training run.
Training and evaluation data contamination: if your evaluation set contains any examples from training, your metrics are meaningless. Use strict train/eval splits and verify there's no leakage.
Model drift over time: fine-tuned models can become stale as the underlying domain changes. Schedule periodic re-training runs as new data accumulates. Track performance on a fixed evaluation set over time to detect degradation.
Cost accounting: for teams considering fine-tuning to reduce API costs (calling a fine-tuned 7B model instead of a frontier model), the break-even calculation needs to account for training costs, serving infrastructure, and the quality gap. A fine-tuned 7B model rarely matches GPT-4 or Claude Opus 4 on complex reasoning — only on narrow tasks where the training data is high quality and the task is well-defined.
Evaluation: Measuring Whether Fine-Tuning Actually Worked
Fine-tuning without rigorous evaluation is expensive guessing. Before training, define the evaluation criteria precisely. After training, measure against them on a held-out test set that has no overlap with training data.
Automatic metrics for structured outputs: if your fine-tuned model produces JSON, code, or templated text, exact-match accuracy on the eval set is your primary metric. "Did the model produce valid JSON?" and "Does the JSON contain the required fields?" are binary, automatable checks.
LLM-as-judge for open-ended outputs: for tasks where quality is hard to measure programmatically — summarization, tone, style, helpfulness — use a larger frontier model (Claude Opus 4 or GPT-4o) to evaluate outputs against a rubric. This approach scales to thousands of examples without human annotation. Define the rubric precisely: don't ask "is this good?" but "on a scale of 1-5, does this response use the company's brand voice as described in [criteria]?"
def evaluate_fine_tuned_model(
eval_dataset: list[dict],
model_name: str,
judge_model: str = "claude-opus-4-6",
) -> dict:
"""
Evaluate fine-tuned model on held-out test set.
Returns accuracy, quality scores, and failure analysis.
"""
results = []
for example in eval_dataset:
# Get fine-tuned model output
prediction = run_fine_tuned_model(model_name, example["instruction"], example["input"])
# Exact match check for structured outputs
exact_match = prediction.strip() == example["output"].strip()
# LLM judge for quality
judge_prompt = f"""
Task: {example["instruction"]}
Input: {example["input"]}
Expected output: {example["output"]}
Actual output: {prediction}
Rate the actual output on these criteria (1-5 each):
1. Format correctness (follows the required format exactly)
2. Content accuracy (facts/logic are correct)
3. Style compliance (matches the expected tone and language)
Output JSON: {{"format": N, "accuracy": N, "style": N, "reasoning": "..."}}
"""
judge_response = run_judge_model(judge_model, judge_prompt)
scores = json.loads(judge_response)
results.append({
"exact_match": exact_match,
**scores,
"prediction": prediction,
"expected": example["output"],
})
return {
"exact_match_accuracy": sum(r["exact_match"] for r in results) / len(results),
"avg_format_score": sum(r["format"] for r in results) / len(results),
"avg_accuracy_score": sum(r["accuracy"] for r in results) / len(results),
"avg_style_score": sum(r["style"] for r in results) / len(results),
"failure_cases": [r for r in results if not r["exact_match"]],
}
Regression testing: a fine-tuned model might improve at the target task while regressing on general capabilities. Always run both task-specific eval and a general capability benchmark. If general capability drops more than 3-5% on your benchmark, investigate before deploying — the training data or learning rate may be causing catastrophic forgetting.
A/B testing in production: for models serving real users, shadow traffic testing or gradual rollout (5% → 25% → 100%) is the gold standard. Real user behavior reveals failure modes that offline eval misses.
Common Fine-Tuning Mistakes
Training on too little data without checking eval quality: 50 examples often isn't enough. 200-500 is a more reliable minimum. If your eval accuracy isn't improving after 3 epochs on 50 examples, the limiting factor is data quantity, not model capacity.
Inconsistent output format in training data: if some examples produce JSON with double quotes and others with single quotes, some with trailing commas and some without, the model learns inconsistency. Normalize all training data formatting before training.
Using production traffic directly as training data without filtering: real user requests include adversarial inputs, edge cases, and off-topic queries. Training on raw production traffic teaches the model those behaviors too. Filter aggressively for clean, canonical examples of the target behavior.
Setting the learning rate too high: high learning rates cause catastrophic forgetting — the model "unlearns" general capabilities to specialize in the training task. For LoRA fine-tuning, 1e-4 to 2e-4 is a typical starting range. Reduce by half if you see general capability degradation.
Forgetting to convert the model to inference format: LoRA adapters trained on 4-bit quantized models must be either merged (model.merge_and_unload()) or served via a LoRA-aware serving stack (vLLM with --enable-lora). Deploying the adapter-only weights without the base model doesn't work.
Conclusion
Fine-tuning is not a silver bullet and it's not always the answer. For most teams, the diagnostic hierarchy is: try a better prompt first (system prompt, chain-of-thought, few-shot examples). If knowledge is the gap, add RAG. If behavior is consistently wrong regardless of what information you provide, fine-tuning addresses the root cause.
LoRA and QLoRA have removed the GPU infrastructure barrier that made fine-tuning impractical for most teams. Training a task-specific adapter on a 7B base model on a rented cloud GPU is now a routine operation, not a research project.
The teams that use fine-tuning well treat it as part of a system — not a replacement for careful prompt design, good data, and rigorous evaluation.
Sources & References
About the Author
Toc Am
Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.
Published: 2026-05-05 · Written with AI assistance, reviewed by Toc Am.
Get These In Your Inbox
Weekly deep-dives on AI engineering, no fluff. Join the newsletter →
Or grab the book ($39, ~100 pages) · Buy me a coffee
☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter
No comments:
Post a Comment