Retrieval or fine-tuning: ask about the data first

Retrieval-augmented generation and fine-tuning are not competing answers. They are answers to different questions. The question is the part most people get wrong, and once the question is right, the answer is usually obvious — and sometimes the answer is “do neither”.

Four questions about your data

Before deciding between retrieval and fine-tuning, answer these four. Each one points to one of the two approaches more clearly than the comparison itself does.

How fast does the data change?

If the data changes weekly or faster, retrieval. Fine-tuning a model takes hours to days; the model is stale before training finishes. If the data changes yearly or slower, fine-tuning is on the table. If the data changes monthly, either can work, and the decision moves to the next question.

Does the customer need to cite the source?

If yes, retrieval, full stop. A retrieval pipeline returns the chunk it used; the model's answer can be traced back to a specific passage in a specific document. A fine-tuned model cannot tell you which training example produced its answer, even when you ask it to. Cite-or-don't-cite is a hard constraint, and it is the one most often skipped in the comparison.

Is the format of the answer fixed?

If the customer needs answers in a specific format — a particular JSON schema, a fixed report template, a regulated document layout — fine-tuning earns its keep. Retrieval can produce the right content but the format consistency degrades over volume. A fine-tuned model can hold a format with very high consistency. This is the case where fine-tuning is straightforwardly better.

How sensitive is the data?

If the data is sensitive enough that even a model trained on it presents a leakage concern, retrieval — with the data sitting outside the model — is safer. Modern fine-tuning has private-tuning options, but the reasoning about leakage is harder. Retrieval keeps the data in a place where the customer's existing access controls already apply.

When neither is right

There is a third option that gets skipped: do neither. A surprising number of cases where teams reach for retrieval or fine-tuning are cases where a small amount of structured prompting against a base model works fine, and adding either retrieval or fine-tuning is overkill.

The signs that you are in this case: the data fits in the prompt, the format is loose, the customer does not need citations, and the volume is low. A careful prompt and a small validator on the output can be the whole design, and it is the cheapest one to operate and the easiest one to explain.

When both are right

There is a fourth case: do both. Fine-tune the model for format and behavior; use retrieval for content. This is the right architecture for any agent that needs to produce a structured output (fine-tuning) grounded in changing data (retrieval).

A monthly management pack is the canonical shape. The format is fixed — the controller has a template, and deviations from it get rejected. The content is the actual numbers, which change every month. The design fine-tunes on the format and reads the numbers via retrieval.

What we do not do

We do not fine-tune for raw knowledge. The cost-benefit is wrong; retrieval is faster, cheaper, and easier to update. Even where fine-tuning on a corpus of operational runbooks edges out retrieval on a held-out set, the fine-tuned version is the wrong thing to ship: the first time a runbook is updated the model is wrong, and the operations team has no way to know which answers are stale. A benchmark margin does not survive contact with a document that changed.

We also do not stack layers by default (fine-tune, retrieve, then prompt-engineer the output). Every added layer has to earn its place on the held-out evaluation, because each one makes the system harder for the on-call engineer to read. A layer that cannot show its gain is a maze, not a system.

The decision, condensed

Ask the four questions. If you have not answered all four, you are not ready to choose. If you have, the choice is usually obvious.

The next post is on running vector stores in production, which is half of what makes the retrieval option work and the part that gets skipped most often.