RAG or Fine-Tuning? Choose the Boring Answer First

Teams often arrive at this question having already decided the answer is interesting. Usually it is not. Most problems that appear to need fine-tuning need better retrieval, and a meaningful share of problems that appear to need retrieval need a better prompt and a longer context window.

It helps to separate two things that get conflated: what the model knows, and how the model behaves.

Two different problems

Retrieval changes what the model knows at the moment of the request. Fine-tuning changes how the model behaves across all requests. They are not competing solutions to one problem; they are solutions to different problems that happen to share a delivery mechanism.

• Your knowledge changes often, or is large, or must be attributed. That is retrieval. A policy document updated weekly has no business inside model weights.
• Your format, tone, or task structure is unusual and consistent. That is a candidate for fine-tuning. Teaching a model your house style, or a domain-specific output schema, is behaviour.
• You need both: retrieval for the facts, fine-tuning for the shape. This is common in mature systems and rare in first versions.

Start further back than you think

Before either, exhaust the cheap options. Modern context windows are large enough that stuffing the ten most relevant documents into the prompt is a legitimate architecture for a surprising number of use cases. It is trivially debuggable, has no infrastructure, and gives you an honest baseline.

That baseline matters. Without it you cannot tell whether your retrieval pipeline is adding value or merely adding failure modes. We have seen elaborate vector search stacks underperform a well-constructed prompt over a curated document set, and nobody noticed because the baseline was never measured.

When retrieval is the answer, the retrieval is the work

Teams underestimate this consistently. The generation step is largely solved and improves for free as models improve. The retrieval step is your problem, and it is where quality is won or lost.

1. Chunking is a modelling decision, not a default. Splitting on a fixed token count severs arguments mid-sentence. Split on document structure (sections, clauses, headings) and keep enough surrounding context that a retrieved fragment is self-explanatory.
2. Hybrid search beats pure vectors for most corpora. Semantic similarity is poor at exact identifiers, product codes and names. Combining it with keyword search recovers precisely the cases that embarrass a demo.
3. Re-ranking earns its cost. Retrieve broadly, then re-rank the candidates against the actual question. This is usually the highest-return change available once a naive pipeline exists.
4. Measure retrieval separately from generation. If you only measure the final answer, you cannot tell whether a failure was a retrieval miss or a generation error, and those have opposite fixes.

When fine-tuning is worth it

Fine-tuning makes sense when you have a consistent task, enough examples of it done well, and a behaviour that prompting struggles to hold. Structured extraction into an unusual schema is a good example: a few thousand well-labelled examples will typically outperform a long prompt, run faster and cost less per call.

The honest trade-off is operational. A fine-tuned model is a versioned artefact with a lifecycle: it needs retraining when the base model is deprecated, re-evaluation when your data drifts, and a rollback story. That is a real cost, and it is worth paying only when the gain is one you have measured rather than assumed.

A decision order that holds up

In practice we work through the options in roughly this order, stopping at the first that meets the bar:

1. A clear prompt with curated context in the window.
2. That, plus keyword retrieval over a well-structured corpus.
3. That, plus semantic retrieval and re-ranking.
4. That, plus fine-tuning for output shape and task-specific behaviour.

Most production systems we build settle at step two or three. The teams that jumped straight to step four almost always arrive back at step one eventually, having spent a quarter discovering that their retrieval was the problem all along.