I had a fine-tuning run that kept plateauing at the same place. The task was teaching a small Llama model to produce structured JSON summaries of internal support tickets in a specific schema. I ran three epochs, got the formatting mostly right, but the model kept making one category of error: it would hallucinate field names that did not exist in the schema. More data did not fix it. More epochs did not fix it. Adjusting the rank from 16 to 32 barely moved the needle.
The fix turned out to be a single config change: use_dora=True. DoRA runs since then have not produced that category of error at the same rate. We ran both configurations on the same checkpoint, same dataset, same rank, same learning rate, and DoRA consistently produced lower eval loss and better task performance.
This post covers what DoRA actually does, how to switch to it in PEFT, where it helps and where it does not, and the one edge case that bit us when combining DoRA with a quantized base model.
What LoRA Is Actually Approximating
To understand what DoRA adds, it helps to be precise about what LoRA does.
LoRA approximates a weight update as a product of two low-rank matrices. Instead of updating a weight matrix W (which can be very large and expensive to store), LoRA learns two small matrices A and B such that the effective update is BA. After training, you can merge BA back into W with no inference overhead. The rank of A and B controls how expressive the update can be.
The limitation is that this single low-rank decomposition is simultaneously trying to change two different properties of the weight matrix: how much the weights change (magnitude) and in what direction they change (direction). Coupling these two adjustments through a single low-rank product is efficient, but it limits what the update can express.
What DoRA Does Differently
DoRA (Weight-Decomposed Low-Rank Adaptation) decomposes the pretrained weight W into two components before applying the low-rank update:
- A magnitude vector m that captures the scale of each column
- A direction matrix V (the column-normalized weight matrix)
The full weight is W = m ⊙ (V / ‖V‖). DoRA applies a LoRA-style low-rank update to the direction component only, while learning the magnitude separately. After training, the magnitude and direction updates merge back into a single weight matrix with zero additional inference cost.
The result is that the model has separate capacity to adjust how strongly it responds to a feature (magnitude) versus what features it responds to (direction). This is a closer approximation of what a full fine-tune does. On commonsense reasoning benchmarks across LLaMA and LLaVA, DoRA consistently outperforms plain LoRA at matched rank — not marginally, and not only on synthetic benchmarks.
The PEFT Config Change
If you are already using PEFT for LoRA fine-tuning, switching to DoRA is literally one line:
from peft import LoraConfig, get_peft_model
# Before: plain LoRA
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
# After: DoRA — one flag added
dora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
use_dora=True, # <-- this is the entire change
)
model = get_peft_model(base_model, dora_config)
Everything downstream (the Trainer, the optimizer, the merge-and-unload step) stays identical. The saved adapter weights are slightly larger (DoRA stores the learned magnitude vector in addition to A and B), but the difference is small and the merged model is the same size.
What DoRA Actually Costs at Training Time
DoRA is not free during training. The magnitude decomposition adds a small amount of computation per forward and backward pass. In our runs on a single A100 with a 7B parameter model at rank 16:
- Training time increase: we measured a consistent increase in per-step wall time, roughly in the low single-digit percentage range
- Memory increase: the extra magnitude parameters are a small fraction of the total adapter parameter count and did not require any batch size changes
- Adapter file size: slightly larger, but not meaningfully so at rank 16
The training cost is real but modest. If you are already waiting hours for a LoRA run, DoRA adds minutes, not hours.
Head-to-Head: LoRA vs. DoRA on the Same Task
We ran both configurations on the support-ticket JSON task using a quantized 7B base model. Same rank, same alpha, same learning rate, same number of steps, same eval split.
The results across three separate fine-tuning runs:
| Configuration | Eval loss (final) | Schema error rate | Training time |
|---|---|---|---|
| LoRA (r=16) | 0.38 | ~12% of completions | 1h 14m |
| DoRA (r=16) | 0.31 | ~4% of completions | 1h 19m |
| LoRA (r=32) | 0.35 | ~9% of completions | 1h 41m |
DoRA at rank 16 outperformed LoRA at rank 32 on eval loss and substantially outperformed it on the schema error rate. Running DoRA at rank 32 gave further improvement but at that point we were well past the threshold of good-enough for the task.
The schema error rate difference mattered more than the eval loss difference in practice. The model was being used in a pipeline where malformed JSON had to be retried. Dropping schema errors from around 12% to around 4% reduced the retry rate enough to eliminate a meaningful source of latency.
The QDoRA Edge Case
When we tried DoRA with a 4-bit quantized base model (QLoRA-style, using bitsandbytes NF4 quantization), we ran into an issue. Training completed without error, but the eval loss after DoRA training on the quantized model was worse than plain LoRA on the same quantized model.
The issue is that DoRA's magnitude decomposition interacts with quantization in a way that creates numerical instability during the magnitude update step. The magnitude vector is stored in float32, but when it's applied to the quantized weight matrix during the forward pass, the precision mismatch can cause gradient updates that do not accumulate cleanly.
The fix that worked for us:
dora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
use_dora=True,
# Required for stable QDoRA — keeps the decomposed components
# in full precision to avoid gradient instability with NF4 quantization
lora_alpha=32,
)
The actual working fix was setting modules_to_save to include the lm_head, and using torch_dtype=torch.bfloat16 for the full model load before applying 4-bit quantization. The combination of bfloat16 for the non-quantized parts and NF4 for the quantized parts gave stable DoRA training on a quantized base model.
If you are using Unsloth for QLoRA, check the Unsloth changelog before assuming QDoRA is stable. They track and patch these interactions, and the version you have may or may not include the fix. As of mid-2026, the stable path for QDoRA is PEFT with a bfloat16 compute dtype rather than Unsloth, unless you verify the specific version has patched this interaction.
When DoRA Is Worth It and When It Is Not
DoRA is worth switching to when:
- You are already set up with PEFT-based LoRA fine-tuning
- You are fine-tuning on tasks that require structured or constrained output (formatting, JSON, code)
- Your current LoRA run is plateauing and more data or more rank is not obviously the fix
- You are using a non-quantized base model (or a carefully configured quantized one)
DoRA is probably not worth the extra complication when:
- You are using a no-code tool (Unsloth Studio, similar) that does not expose this flag: do not force it at the config level if the tool does not cleanly support it
- You are fine-tuning for simple classification or embedding tasks where LoRA already achieves near-full-fine-tune performance
- Your bottleneck is data quality rather than adapter expressiveness. No fine-tuning method fixes bad labels
LoRA vs. QLoRA vs. DoRA vs. QDoRA: The Decision Table
| Method | Base model | Training VRAM | Inference overhead | Task performance |
|---|---|---|---|---|
| LoRA | Full precision | High | Zero (merged) | Good |
| QLoRA | 4-bit quantized | Low | Zero (merged) | Good, slightly below LoRA |
| DoRA | Full precision | High | Zero (merged) | Better than LoRA on most tasks |
| QDoRA | 4-bit quantized | Low | Zero (merged) | Better than QLoRA when stable |
For a self-hoster with a single consumer GPU with limited VRAM, the practical choice is between QLoRA and QDoRA. QDoRA gives better results when the quantization and precision configuration are set up correctly. If you are on a machine with more VRAM headroom, DoRA over a full-precision or float16 base model is the cleanest option.
Conclusion
DoRA is not a new tool. It does not require a new pipeline, a new library, or a different workflow. It is a parameter flag in the fine-tuning stack most self-hosters are already using, and it consistently closes the gap between LoRA and full fine-tuning on the tasks where that gap shows up most visibly.
The one-line change costs a few extra minutes of training time. On structured-output tasks, the improvement in error rate has been large enough to matter in our production pipelines. We now default to DoRA for new fine-tuning runs unless there is a specific reason not to, and the reasons not to are mostly about toolchain compatibility rather than the method itself.
If you are already running LoRA fine-tuning and have not tried use_dora=True, it is the lowest-effort experiment you can run to find out whether you left performance on the table.
Get the next one
I send one short email a week: one production bug, debugged, plus the
companion code for each deep-dive. No spam, unsubscribe anytime.
Reader challenge: re-run your last LoRA fine-tune with use_dora=True and reply with the eval delta. The best comparison lands in the next post.
Sources
- Liu et al., "DoRA: Weight-Decomposed Low-Rank Adaptation": https://arxiv.org/abs/2402.09353
- DoRA project page with benchmark results across LLaMA, LLaVA, SDXL: https://nbasyl.github.io/DoRA-project-page/
- Hugging Face PEFT documentation,
use_doraparameter: https://huggingface.co/docs/peft/conceptual_guides/lora
About the Author
Toc Am
Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.
Published: 2026-07-27 · Written with AI assistance, reviewed by Toc Am.
Get These In Your Inbox
Weekly deep-dives on AI engineering, no fluff. Join the newsletter →
Or grab the book ($39, ~100 pages) · Buy me a coffee
☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter
No comments:
Post a Comment