| Decision | What is supported | Planning use |
|---|---|---|
| Trainable share | LoRA trains 0.5-5% of parameters; full fine-tuning trains about 100% | Expect a much smaller set of weights to optimise, store and track |
| Rank | A higher rank can improve quality for some tasks but increases memory use and compute | Start modest and raise rank only if results stall |
| Layer targeting | LoRA can be applied selectively to certain layers or components | Choose target layers deliberately instead of adapting everything |
| Adapters | Adapters are modular and easy to version control, and several task-specific modules can be swapped as needed | Keep one adapter per task on a shared base model |
Figures from Modal’s LoRA and QLoRA comparison and Exxact’s LoRA overview.
This lora fine tuning explained guide starts with the key point: LoRA freezes the base model and trains small added matrices, so far fewer parameters change. Quality depends mainly on rank and on which layers you target.
If you are deciding whether to adapt a model at all, the wider overview on how to fine tune an llm sets out the options. For the choice between adapting weights and retrieving context, see fine tuning vs rag.
What LoRA changes in practice
LoRA changes how much you train, how much you keep, and how easily you can switch tasks. According to Modal, LoRA trains 0.5-5% of a model’s parameters, while full fine-tuning updates every single parameter in the base model. That gap drives most of the practical benefits.
- Smaller optimisation problem. Only the added matrices receive updates, so the trainable set is a small slice of the model and not the whole of it.
- Task-specific outputs. Each training run produces its own adapter instead of a modified copy of the model, and multiple LoRA modules can be trained for different tasks and swapped out as needed.
- Cleaner experiment tracking. LoRA adapters are modular and easy to version control, which suits repeated runs with different settings.
LoRA suits you if you want several specialised behaviours from one base model and need to keep the training footprint small. If you expect to change the model’s behaviour as broadly as possible and can afford training everything, full fine-tuning keeps every parameter free to move.
LoRA versus full fine-tuning
LoRA and full fine-tuning differ in what is allowed to change. Full fine-tuning updates every parameter, so every weight can move. LoRA keeps the original weights frozen and learns only the added low-rank matrices, which limits the form the update can take.
The trade-off is straightforward. Memory and compute for training follow the number of trainable parameters, so training a small fraction costs less than training everything. Whether the restriction hurts depends on the task, and rank is the lever you have. Higher rank can improve quality for some tasks, but it increases memory use and compute.
- Choose LoRA when you need several task variants, want cheap experiments, or must keep training within a limited memory budget.
- Choose full fine-tuning when a single model must change as freely as possible and the resources are available to train all parameters.
- Raise rank before switching approach if a LoRA run underperforms, since rank is the direct quality-versus-cost control.
Adapter weights: storing, swapping and merging
Adapter weights are the trained LoRA matrices, kept separately from the base model. Because the base stays frozen and untouched, one copy of it can serve many adapters, and each adapter represents one task or experiment.
You can also fold an adapter into the model. The LoRA weights can be merged with the original weights when required, which lets you use the result without separate adapter layers. This works because the adapted weight is the base weight plus the product of the two small matrices, so merging means adding that product into the base weights.
- Keep adapters separate while you are still comparing runs or serving several tasks from one base model.
- Merge once you have settled on one adapter and want a single set of weights with no extra layers.
The low-rank mechanism
Low-rank adaptation represents a weight update as the product of two small matrices instead of one full-size matrix. The adapted weight matrix is calculated as W’ = W + A·B, where W is the frozen base weight and A and B are the low-rank matrices.
Picture W as a large grid of numbers. A is tall and narrow, B is short and wide, and multiplying them gives a grid the same size as W. The shared narrow dimension of A and B is the rank. A small rank means fewer numbers to store and train. A larger rank gives the update more room to express changes.
That is why LoRA adds so few trainable parameters. As one beginner’s guide describes it, LoRA adds a small number of trainable parameters while keeping the original model parameters frozen. In a transformer, the small low-rank matrices are added to the attention mechanism.
Rank, alpha and target layers
Three settings shape a LoRA run: rank, alpha scaling and the target layers. In the PEFT library they live in one configuration object, and the hands-on route is covered in the qlora fine tuning tutorial.
Rank
Rank sets the size of the added matrices and therefore how much capacity the adapter has. The cost of a larger rank is more memory and compute, so treat it as a dial to turn up only when a smaller value underperforms.
Alpha scaling
Alpha is the scaling setting that sits alongside rank in the configuration. One worked BERT example uses r=8 with lora_alpha=16. Treat that as an example pairing, not a universal default, and change rank and alpha deliberately rather than together.
Target layers
The same example targets the query and value layers: it applies LoRA to BERT’s query and value layers so that only the added LoRA parameters are trained. Because LoRA can be applied selectively, you can start with attention projections like these and widen the list only if quality needs it. Each extra target adds more trainable parameters.
Frequently asked questions
Which library provides LoRA support?
The Hugging Face PEFT library supports LoRA fine-tuning, as MindStudio notes. Typical code imports LoraConfig and get_peft_model from peft.
Why is the LoRA learning rate often higher than in full training?
One worked example uses a learning rate of 2e-4, described as slightly higher because only the LoRA layers are trained. Treat it as a starting point for that example, not a rule.
Is LoRA the same thing as low-rank adaptation?
Yes. LoRA is the abbreviation of low-rank adaptation, the method of adding small trainable matrices to selected layers while the base weights stay frozen.
Does LoRA change the original model file?
No. The base weights remain frozen and the training output is the adapter, so the original model is untouched unless you choose to merge the adapter into it.
