RAG vs fine-tuning vs small models: which one fixes your problem

Use RAG when the model gives wrong or outdated answers because it can't see your documents. Use fine-tuning when it knows enough but keeps doing the task badly, such as mislabeling tickets or breaking the output format. Use a fine-tuned small model when that task runs at high volume or the data can't leave your environment. Most business problems start with a good prompt and retrieval, so try those first.

#What each option actually changes

The three terms get mixed up because they answer different questions.

RAG (retrieval-augmented generation) changes what the model sees. At question time, the system searches your files, picks the relevant passages and hands them to the model with an instruction to answer from them and cite them. The model itself is untouched. Our RAG guide covers the pipeline and has a shorter version of this comparison.

Fine-tuning changes how the model behaves. You continue training it on labeled examples, so it learns one task in your format with your categories. It's a poor place to store facts. They get blended into the weights with no source and no easy way to remove them.

A small model is a deployment choice layered on top. You take an open-weight model of a few billion parameters, fine-tune it on one job and run it in your own cloud account or on your own hardware. The trade-offs are in SLM vs LLM for business, and on-premise LLM covers the hosting side.

So the first two are techniques, and the third is a technique plus a hosting decision. They combine well. A fine-tuned small model can sort incoming tickets while a RAG layer answers questions from the help center.

#Pick by symptom

Start with what you see going wrong. The technique comes second.

Read down the third column. Nearly every first fix is a prompt change or a retrieval step, and fine-tuning only appears once those have been measured and found short.

#RAG vs fine-tuning vs small model, side by side

#Examples from business documents

Employee policy questions. HR gets the same ones every week: parental leave, expense limits. The documents change every year. RAG fits, with permissions mirrored from SharePoint or Drive and a "can't find it" fallback. Fine-tuning would freeze the policy at training time and give staff no source to check.

Supplier invoice extraction. A finance team receives PDFs from several hundred suppliers and needs vendor, invoice number, dates, line items and tax. Retrieval doesn't help, because nothing needs looking up. At low volume, a strong prompt on a large model does the job. At tens of thousands of invoices a month with stable layouts, a small model fine-tuned on your posted ERP values can cut cost and latency, as long as a validation layer and a review queue sit around it. Extraction at that volume is where small models earn their keep.

Contract clause drafting. Lawyers want first drafts in the firm's voice, built from approved clauses. The clauses are knowledge, so a retrieval step supplies them. The voice is behavior, and a few approved examples in the prompt usually capture it. Fine-tuning only enters if the style stays inconsistent after that. A lawyer approves every draft either way.

#Cost drivers, without the price tags

We don't quote prices here because they swing with document volume, model choice, hosting and how clean your sources are. What follows is what moves each bill.

RAG costs sit mostly before and after the model. Before: connectors, parsing scanned PDFs and tables, chunking, and keeping permissions in sync. After: the retrieved text adds tokens to every question, and someone has to maintain the test set and the sources. A messy knowledge base raises the build cost far more than the model does.

Fine-tuning costs sit in the data. Labeled examples need people, consistent category rules and a held-out test set. Training runs are the small part. Every change in your business, such as a new product line or invoice layout, means collecting examples and retraining. So when someone asks about LLM fine-tuning cost, the honest answer is that the training run is rarely the expensive line.

A small model adds hosting: GPUs in your cloud account or your own data center, monitoring, a rollback plan and someone on call. At low volume that fixed cost makes it dearer than an API. At high and steady volume it flips.

Key point: Retrieval fixes what the model knows, and fine-tuning fixes what it does. Work out which problem you have, fix the cheaper one first and measure on your own cases before paying for training.

#The order we try things in

1. Write a clear prompt with a handful of approved examples and a defined output format. Measure it on a test set of real cases.
2. Add retrieval if errors come from missing or outdated knowledge. Score retrieval separately from the final answer.
3. Add validation in code for anything exact: totals, dates, allowed values.
4. Fine-tune only if a measured gap remains on a narrow, stable task and you have enough clean labeled examples.
5. Move to a small model when volume, latency or data residency justifies hosting it, and benchmark it against the large model first.

Skipping steps 1 to 3 is the usual mistake. Teams jump to training because it sounds more serious, then discover the problem was a missing document.

#Hiring a RAG development company or custom LLM team

If you're comparing RAG development services, or a firm offering custom LLM development, put these questions to them before signing.

• Will they build a test set from your own questions, and show retrieval scores separately from answer scores?
• How are permissions enforced, and who sees which passages?
• What happens when the answer isn't in the documents?
• Who owns the prompts, the index and the evaluation set at handover?
• If they recommend fine-tuning, which measured gap does it close?

A vendor that proposes training before running a baseline is guessing. You can see how we scope these systems under knowledge assistants and private AI models. If you want a straight view on which one your use case needs, book a strategy call and bring a sample of your documents or labeled data.

#Frequently asked questions

#Is RAG better than fine-tuning?

Neither is better in general. RAG wins when the problem is missing or changing knowledge, because it keeps answers current and cited. Fine-tuning wins when the model has the knowledge but handles a repeated task poorly, such as labeling or formatting.

#Can you use RAG and fine-tuning together?

Yes, and many production systems do. A fine-tuned model handles a fixed task like classification, while a RAG layer answers questions from your documents. Treat them as separate components, each with its own test set.

#How much data does fine-tuning need?

It depends on the task. For classification with a modest set of categories, a few thousand labeled examples is a common starting point, and consistent extraction can need fewer. Your held-out test results show whether more data would help, so measure before collecting more.

#What drives the cost of fine-tuning an LLM?

Mostly the labeled data and the upkeep: people to label, a test set, repeated evaluation and retraining when your business changes. Hosting adds a fixed cost if you run a small model yourself. The training run itself is usually a minor part of the total.

#Do we need a custom model at all?

Often not. A well-prompted hosted model plus retrieval covers many business cases. A custom or fine-tuned model earns its place when the task is narrow, volume is high, accuracy on your own data is measurably short, or data can't leave your environment.