Choosing between fine-tuning and retrieval-augmented generation starts with a diagnosis: what is the system failing to do? An assistant that cannot find a current policy has a different problem from a classifier that consistently uses the wrong output format. Both failures can look like a weak model when viewed only through a final answer.
This guide offers an original decision worksheet for a fictional documentation assistant. It does not claim that one approach is universally cheaper, faster, or more accurate. The goal is to define experiments that reveal whether a proposed change addresses the observed failure, under your own quality, access, and operating constraints.
Understand the two mechanisms
Fine-tuning continues training a pretrained model on task-specific data, as described in the Hugging Face training guide. The resulting model can behave differently because its parameters have changed. The effect depends on the training method, examples, base model, and evaluation task; the word fine-tuned is not itself evidence of improvement.
RAG adds retrieved information to a generation workflow. Microsoft's RAG overview describes grounding responses in supplied content. Retrieval can make relevant documents available at answer time, but does not guarantee that the right passage will be found, interpreted correctly, or cited accurately.
The approaches can be combined. A fine-tuned generator may still use retrieved documents, and a retrieval system may use a separately trained component. Keep those component decisions distinct so that an experiment changes one identifiable part of the application.
Establish a prompt-only baseline
Before building either approach, save a small set of representative questions and the outputs you expect. Include ordinary requests, ambiguous requests, missing information, and requests for restricted content. Record what a simple prompt with the available information does today. This baseline makes the proposed investment answerable: what specific failure should the new system reduce?
Use a written output contract. For the fictional assistant, require an answer, supporting document identifiers when applicable, and an explicit unknown when the evidence is insufficient. Keep invalid formats, unsupported statements, and access failures as separate observations. A single aggregate score can hide an important regression.
Do not treat the examples used while editing prompts as an independent final assessment. Reserve additional cases for reviewing the selected configuration. The size and coverage of that assessment should reflect the consequences of errors, rather than a universal number borrowed from another team's task.
Diagnose missing or changing information
Consider a question about a policy that changed yesterday. First ask whether the correct version was available to the application. If it was absent, changing the generator's behavior cannot make that missing source available by itself. A document-ingestion and retrieval experiment is a reasonable next candidate.
Trace the stages separately. Did ingestion accept the document? Did access filtering permit it for this user? Did retrieval return the relevant passage? Did the generator use that passage correctly? If the passage never reached generation, evaluate retrieval before changing generation settings.
Treat source freshness as an operational property. Define how updates enter the index, how obsolete versions are identified, and what the application says while an update is pending. RAG does not automatically make a document current or ensure that every result is authorized.
Diagnose behavior and formatting failures
Now consider an assistant that sees all necessary information but repeatedly returns an invalid category or a response that violates a well-defined format. Improve the instructions and output validation first. If the failure persists and you have suitable examples of the desired behavior, evaluate a training experiment against the same baseline.
Review the examples for inconsistent labels, accidental sensitive data, and contradictions in the desired output. A training process cannot resolve an organizational policy that the examples themselves do not express consistently. Keep the labeling policy and dataset versions with the experiment record.
Avoid claiming that a certain number of examples guarantees a useful model. Data requirements depend on the task and method. Select a documented training approach, inspect its requirements, and measure the result on cases that were not used to guide the change.
Compare candidate experiments
| Observed problem | Candidate investigation | Evidence to collect |
|---|---|---|
| Relevant policy is absent | Ingestion and retrieval | Source version and retrieved passages |
| Passage is present but answer is unsupported | Prompt, generation and review | Claim-to-passage checks |
| Output labels violate a stable policy | Instructions, parser and possibly training | Invalid labels and category errors |
| Source is restricted for this user | Authorization and filtering | Permitted and denied access cases |
| Response is too slow | End-to-end profiling | Stage timings under comparable load |
These are starting points for investigation, not automatic architecture selections. An authorization failure belongs to application controls even if a different model happens to avoid the restricted answer in one test. Likewise, a faster answer that drops required evidence has not necessarily improved the workflow.
Measure total operating effort
List costs before assigning estimates. A retrieval design may need document parsing, indexing, access filtering, refresh jobs, evaluation, and generation. A fine-tuning design may need example preparation, training, artifact management, deployment, evaluation, and later retraining. A combined design carries responsibilities from both.
Use your actual provider documentation and measurements for prices, supported methods, model availability, and capacity. Do not assume that a model with changed weights has the same serving cost or latency as the baseline. Keep request size, context size, hardware or service configuration, and concurrency comparable when interpreting timing results.
Also measure reviewer effort. Count corrections, unresolved cases, and the time needed to locate supporting evidence. An implementation that reduces one infrastructure bill while increasing manual investigation may be a poor fit for the intended use. State which effort your comparison includes and which it leaves out.
Plan a bounded comparison
For the fictional documentation assistant, choose a retrieval baseline and one alternative change. Save the questions and permitted document snapshot. Define acceptance conditions before examining the alternative's outputs. Require that denied content remains denied and that unknown answers remain visible.
Run the configurations on the same cases. Compare answer support, missing information, invalid output, stage timing, and review effort. Inspect individual changed answers rather than relying only on average scores. If a result improves on common questions but regresses on restricted requests, report both observations.
If the retrieval baseline meets the requirements, stop and document why a more complex design is unnecessary for this version. If it fails, use the failure trace to choose the next change. The next step might be better document parsing, a clearer labeling policy, another prompt, or a training experiment; it is not predetermined by the name of the architecture.
Record the decision and its limits
A useful decision note names the task, input snapshot, baseline, alternative, acceptance conditions, observed errors, operating assumptions, and remaining uncertainties. Include a rollback plan and a trigger for reassessing the choice, such as a new document source or a changed output policy.
For practical next steps, use the RAG evaluation guide to separate retrieval and generation errors, and the classifier worksheet to make label comparisons inspectable. Choose the smallest change supported by your evidence, then keep evaluating as the surrounding data and workflow change.