Skip to content
LiveNext 22:06:47
topfive.ai

Definition

What is LoRA?

LoRA is LoRA (low-rank adaptation) is a fine-tuning method that trains a small set of added weights while leaving the original model frozen, making customization far cheaper and the result a small file you can swap in.

LoRA, short for low-rank adaptation, is a way to customize an AI model without retraining all of it. Instead of changing billions of numbers, you train a small add-on that sits on top of the original model.

How it works

A model's behavior lives in its parameters, stored as model weights. Normal fine-tuning updates all of them, which needs a lot of memory and produces a full-size copy of the model for every variation.

LoRA freezes the original weights and adds a pair of small matrices next to selected layers. Only those small matrices are trained. The word "low-rank" refers to a mathematical trick: the change you want is approximated by multiplying two thin matrices, which has far fewer numbers than a full matrix.

Think of a stage production. The theater, the set and the cast stay the same. LoRA is a costume change: a small, swappable layer that gives the same performers a new look.

Why it matters to you

  • Cheap to train. Because so few numbers are updated, LoRA can run on a single consumer graphics card for many models.
  • Small to share. A LoRA file is often tens or hundreds of megabytes instead of many gigabytes.
  • Swappable. You can keep one base model and load different LoRAs for different jobs, such as a brand's visual style, a character, or a writing voice.
  • Reversible. The original model is untouched, so you can remove the add-on at any time.

LoRA is especially popular with open image generators, where people train a style or a subject from a few dozen pictures. It is also used with open-weight language models to teach a format, tone or specialty.

What it can and cannot do

LoRA is good at shaping style, format and narrow skills. It is less suited to teaching a model large amounts of brand-new knowledge, and a poorly made one can make the model worse or produce odd results. Quality depends on the examples you train on, so a small, clean set beats a large, messy one.

A related method, QLoRA, combines LoRA with quantization so the frozen base model is stored at lower precision, which cuts the memory needed even further.

LoRA is one flavor of post-training customization. It differs from distillation, which trains a smaller model to imitate a larger one, and from prompting, which changes nothing inside the model at all.

Questions people ask

What does LoRA stand for?

Low-rank adaptation. It adds small trainable matrices to a frozen model instead of retraining the whole thing.

Is LoRA the same as fine-tuning?

It is a kind of fine-tuning. Full fine-tuning updates every weight, while LoRA trains only a small add-on, which is cheaper and easier to share.

LoRA in the news