AI & ML / Training & fine-tuning / 06_lora_and_peft.md

LoRA and parameter-efficient fine-tuning

Updated 6 interview angles 5 min read source
On this page8
  1. The insight
  2. The knobs that matter
  3. QLoRA: the one that made it accessible
  4. Serving: the property that decides architectures
  5. The other PEFT methods
  6. When not to fine-tune at all
  7. Related
  8. Interview angle

LoRA and parameter-efficient fine-tuning

Full fine-tuning of a 70B model needs roughly a terabyte of GPU memory once you count optimiser state and gradients. PEFT is the family of methods that get most of the benefit by training a tiny fraction of the parameters, and LoRA is the one that won.

The insight

The weight update a fine-tune learns is low-rank — it does not need the full expressiveness of the matrix it modifies. So instead of learning ΔW directly, learn two thin matrices whose product approximates it:

text 
W' = W + BA        W: d×k frozen
                   B: d×r   A: r×k     r ≪ min(d, k)

At r = 16 on a 4096×4096 matrix, that is 131k trainable parameters instead of 16.7M — about 0.8%.

python 
from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    # scaling: effective LR is alpha/r
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)
model = get_peft_model(base, config)
model.print_trainable_parameters()
# trainable: 4,194,304 || all: 6,742,609,920 || trainable%: 0.062

B is initialised to zero, so BA = 0 and the adapted model starts exactly equal to the base model. Training therefore begins from a known-good point rather than a perturbed one, which is why LoRA is so stable.

The knobs that matter

Parameter Effect
r capacity. 8-16 for style, 32-64 for new knowledge
lora_alpha scaling; the effective rate is alpha / r
target_modules which matrices get adapters
lora_dropout regularisation on the adapter path

target_modules is where people under-reach. Attention projections only (q_proj, v_proj) is the classic recipe and it is conservative; including the MLP projections costs more memory and reliably does better when you are teaching the model something it does not know.

Gotcha: raising r without raising lora_alpha lowers the effective learning rate, because the scaling is alpha / r. People bump r to fix underfitting, get a worse result, and conclude LoRA cannot learn the task.

QLoRA: the one that made it accessible

Quantise the frozen base to 4-bit, keep the adapters in 16-bit, and backprop through the quantised weights. A 70B model fine-tunes on a single 48GB card.

python 
model = AutoModelForCausalLM.from_pretrained(
    name,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.bfloat16,
        bnb_4bit_quant_type="nf4",       # normal-float, for normal weights
        bnb_4bit_use_double_quant=True,  # quantise the quantisation constants
    ),
)

The base is frozen, so its quantisation error is fixed and the adapters learn around it — which is why quality holds up far better than quantising a fine-tuned model afterwards.

Serving: the property that decides architectures

Two options, and the second is why LoRA dominates in production:

python 
# Merge: zero inference overhead, one model per task.
merged = model.merge_and_unload()

# Keep separate: one base in memory, swap adapters per request.
model.set_adapter("customer_support")

One base model plus N adapters at ~50MB each means serving a hundred fine-tunes from one GPU. Full fine-tuning means a hundred model copies. For any multi-tenant or per-customer product that is not a performance detail, it is the difference between a viable business and an impossible one.

Merging is right when a task is hot enough to deserve a dedicated deployment; keeping adapters separate is right when there are many and each is cold.

The other PEFT methods

Method Idea Status
LoRA / QLoRA low-rank update the default
DoRA decomposes magnitude and direction better, slightly slower
Prefix / P-tuning learned virtual tokens mostly superseded
(IA)³ learned rescaling vectors very few parameters
BitFit train only biases a baseline, not a plan

Naming LoRA and QLoRA and knowing DoRA exists is the right depth. The others are largely historical.

When not to fine-tune at all

The question behind the question. Fine-tuning teaches form — a tone, a schema, a task shape. It does not reliably teach facts, and it cannot teach facts that change.

  • Needs current or private data → retrieval, not fine-tuning.
  • Needs a consistent output format → try structured output and a prompt first.
  • Fewer than a few hundred good examples → prompt, and collect more.

See Fine-Tuning vs RAG vs Prompt Engineering, which is the decision in full.

Interview angle 6

  • “What is LoRA?” - instead of learning the full weight update, learn two thin matrices whose product approximates it. The update is low-rank, so r = 16 trains under 1% of the parameters. B starts at zero, so the adapted model begins identical to the base, which is why it trains so stably.
  • “What does lora_alpha do?” - scales the adapter output; the effective learning rate is alpha / r. Raising r without raising alpha lowers it, which is why bumping r to fix underfitting often makes results worse.
  • “Which modules do you adapt?” - attention q_proj and v_proj is the conservative default and fine for style. Include the MLP projections when teaching genuinely new capability; it costs memory and reliably does better.
  • “What is QLoRA?” - a 4-bit frozen base with 16-bit adapters, backpropagating through the quantised weights. It puts 70B fine-tuning on one 48GB card, and quality holds up because the base is frozen so its quantisation error is fixed rather than compounding.
  • “Why does LoRA matter for serving?” - one base model plus many ~50MB adapters, swapped per request. A hundred customer fine-tunes fit on one GPU where full fine-tuning would need a hundred model copies. Merge instead when a single task is hot enough to justify its own deployment.
  • “When would you not fine-tune?” - when you need facts rather than form. Fine-tuning teaches tone, schema and task shape; it does not reliably teach knowledge and cannot teach knowledge that changes. That is retrieval.
Previous
Distillation and synthetic data