A LoRA adapter or a full fine-tune?

A small language model trained for one domain starts from a base model that already reads and writes the language. The work is to teach it the institution's tasks: its forms, its categories, its report formats, the tools it calls. There are two broad ways to do that. Train a small set of added weights, an adapter, and leave the base untouched; or update every weight in the model. The choice changes the cost of training, how the model is served, what it may forget, and how each task is tested.

What LoRA changes

Low-rank adaptation (LoRA) is a widely used way to train an adapter. The original paper describes it as a method that “freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture.” In plain terms: the base model stays exactly as it was, and next to some of its weight matrices sits a pair of much smaller matrices that learn the change the task needs. Their size is set by the rank, which bounds how much the adapter can move the model.

The adapter is a small file beside the base. For production, the paper notes that the change can be merged into the base weights, so a merged model adds no inference latency; switching to another task means removing one adapter's change and adding another's.

When one adapter per task is enough

An adapter fits when the base model already understands the material and what is missing is the institution's way of handling it:

• extracting the fields of a claim form into the institution's own record format;
• sorting complaints into its categories, with its definitions;
• writing a section of a credit memo in its template and tone;
• calling a tool, such as a SQL query or a JSON request, with the right parameters.

Each of these is mostly format, labels, and judgment calls the expert can grade. A separate adapter per task keeps each one small, versioned, and replaceable on its own. There is a second reason to prefer it: the study “LoRA Learns Less and Forgets Less” found that LoRA “better maintains the base model's performance on tasks outside the target domain.” When the model still has to read general text well, forgetting less matters.

When the whole model has to move

The same study is also the clearest warning. On programming and mathematics, in both instruction fine-tuning and continued pre-training, it reports that “in the standard low-rank settings, LoRA substantially underperforms full finetuning.” An adapter learns less. When the distance between what the base knows and what the task needs is large, a small change may not cover it.

Two situations justify going further:

1. The field's language is missing. If the base model does not know the domain's vocabulary and documents, such as a sector's regulations, its product terms, or its technical shorthand, continued pre-training on in-domain text comes first. Gururangan and colleagues found that “a second phase of pretraining in-domain (domain-adaptive pretraining) leads to performance gains.” We use it only where that language is actually missing, measured before training.
2. The adapter fails the test set. If an adapter, after its rank and data have been tried, still fails tasks on the expert's test set, a full fine-tune is the next candidate.

A full fine-tune has costs an adapter avoids. It needs more memory and compute to train, each variant is a full copy of the model, and the general abilities the work still relies on need their own regression test, because more of the model moved. Hardware for either route is sized in the Decision Blueprint, from the model and the tasks.

Many adapters, one base

Adapters change how a model is served. Several tasks can share one base model in memory, each request using its own adapter. The S-LoRA work, for example, describes a serving system that “stores all adapters in the main memory” and claims it can “serve thousands of LoRA adapters on a single GPU or across multiple GPUs.” Within one institution, that means a new task is a new adapter, not a new model to host, and a task can be rolled back or retired without touching the others.

The boundary still holds between institutions. An adapter trained on one customer's data belongs to that customer and never serves, or trains, another customer's system.

Test each task with the expert's test set

Before any training, the institution's expert grades a test set for each task: real-shaped cases with the answers the expert would give. Every candidate, an adapter at a given rank, a full fine-tune, or the base model with a careful prompt, is measured on the same set. A task goes live only when it passes, and only that task goes live. A small model trained this way often matches a large general model on its own task; often, not always, which is why the test set decides and not the method.

Changing knowledge stays in retrieval, not in the weights, and policy and authority stay outside the model. The adapter teaches a task; it does not grant permission to act.

The base model's license still applies

An adapter is trained on top of a base model, and the base model's license continues to apply to everything built on it: what uses it allows, what attribution it requires, and whether the result may be redistributed. That is checked before the base is chosen, not after the first adapter works.

Start with the smallest change that passes the expert's test set. Move the whole model only when the evidence says the small change cannot do the work.