What is RAG? How retrieval-augmented generation works on company documents

A RAG knowledge assistant for company documents answers questions by searching your own files at the moment a question is asked, passing the most relevant passages to a language model, and returning an answer with links to its sources. Done well, it gives staff a fast answer they can check in seconds, drawn from the current version of a policy, contract or help article. Done badly, it gives confident answers that are wrong, so most of the work goes into retrieval, permissions and testing.

#What retrieval-augmented generation means in plain language

Think of an open-book exam. The language model has not memorized your company's documents. Each time someone asks a question, the system finds a handful of relevant excerpts and hands them to the model with an instruction: answer using only these, and say where each point came from.

That design has three practical consequences:

• Answers stay current. Update the travel policy in SharePoint and the next answer reflects it. Nothing is retrained.
• Answers are checkable. Every claim links to the paragraph it came from.
• Content can be removed. Delete or restrict a document and it stops appearing in answers.

This is why RAG suits company knowledge better than training a model on your documents. Fine-tuning is good at teaching a model a task, such as sorting tickets into categories, and a poor way to store facts that change and need a source. Our page on private AI models covers where fine-tuned models fit.

#RAG vs fine-tuning

Teams often ask whether the model should be "trained on our data." RAG supplies facts at question time. Fine-tuning changes how the model behaves. If the problem is "it doesn't know our content," try retrieval first. If it keeps doing the task the wrong way, such as mislabeling tickets, fine-tuning is the next step, and plenty of systems use both. The full comparison, with a symptom-by-symptom guide and cost drivers, is in RAG vs fine-tuning.

#The parts of a RAG knowledge assistant

Every RAG system follows the same pipeline. Answer quality depends on how carefully each stage is built.

#Connectors and sync

Connectors pull content from where it already lives: Google Drive, SharePoint, Confluence, Notion, Zendesk or Intercom help centers, and internal databases. They must pick up edits and deletions, and carry metadata with the text: owner, last-modified date, document type and access permissions. Without that metadata, later stages cannot filter out stale content or enforce access.

#Chunking

Documents are split into passages small enough to search precisely. Splitting at fixed character counts is a common shortcut and a common source of bad answers: a refund rule gets cut in half, or a table loses its header row. Better chunking follows the document's structure (headings, sections, list items, table rows) and keeps the section title attached to each passage. Scanned PDFs and slide decks need layout-aware parsing first.

#Hybrid search

Two kinds of search run side by side. Vector (semantic) search finds passages with similar meaning, so "time off for a new baby" matches a section titled "Parental leave." Keyword search finds exact terms that semantic search often misses: product codes, error numbers, contract IDs, names. Combining both finds the right passage far more often than either alone.

#Re-ranking

Search returns a few dozen candidates. A re-ranking model reads each one against the question and reorders them by relevance, so only the best handful reach the language model. This often fixes cases where the right passage was found but ranked too low to be used.

#Permission-aware retrieval

The assistant answers only from content the person asking is already allowed to see. Access rules are synced from each source, and results are filtered by the user's identity before any text reaches the model.

#Cited answers

The model writes a short answer with citations to specific passages, each linking to the source document. When it cannot find an answer, it says so and points to the team that owns the topic.

#How to prevent made-up answers

Language models will produce a fluent answer to almost anything. The system's job is to let them answer only when the evidence is there.

• Answer only from retrieved passages. The instructions forbid outside knowledge for company-specific questions.
• Set a relevance threshold. If no passage scores above it, the assistant replies "I could not find this in the documents you have access to" and suggests who to ask.
• Require and check citations. An automated step confirms that each cited passage supports the sentence it is attached to, and flags answers that fail.
• Clean up the sources. If three versions of the expense policy exist, the assistant may quote the old one. Archive superseded documents and prefer recent content.
• Start narrow. An assistant for IT and HR policies is far easier to make reliable than one covering the whole company drive.
• Route sensitive topics to people. Questions about legal disputes, individual pay or disciplinary matters get a pointer to the right contact.

Key point: Most wrong answers trace back to retrieval. The right passage was never found, was split badly, or an outdated version ranked higher. Fix retrieval first and the answers usually follow.

#How to evaluate quality with a test set of real questions

Ten questions in a demo tell you very little. Build a test set and score it before launch and after every change.

1. Collect real questions. Pull 100 to 300 from support tickets, the #ask-hr Slack channel, IT helpdesk emails and intranet search logs, with their original phrasing.
2. Record the expected answer and source. A subject-matter owner writes the correct answer and the document section it comes from.
3. Add questions that should be declined. Include questions with no answer in the documents, and questions a test user lacks access to.
4. Score retrieval separately. Check whether the correct source appears in the top results. If it does not, no prompt change will fix the answer.
5. Grade the answers. Correct, partly correct, wrong, or correctly declined. People should grade at least a sample. Automated grading helps at scale once you have checked it against human grades.
6. Agree a launch threshold. The business owner sets the acceptable pass rate and names errors that are unacceptable at any rate, such as quoting a restricted document.
7. Re-run on every change. New sources, a different chunk size, a new model version or a prompt edit can all lower quality silently.
8. Keep growing the set. Review thumbs-down ratings and unanswered questions weekly and add the useful ones.

#Security and permissions

For most companies, permissions decide whether an assistant can launch at all.

• Mirror source permissions. The model never sees a passage the user could not open themselves.
• Use real identity. The assistant knows who is asking through your single sign-on. A shared service account that reads everything defeats the purpose.
• Watch the sync gap. If permissions sync hourly, someone removed from a folder may get answers from it for up to an hour. For sensitive sources, check access at query time.
• Exclude what should never be indexed. Board materials, HR case files and payroll exports are often better left out entirely.
• Know where data goes. Decide where the index lives, which model provider processes passages, and their retention terms. The whole system can run in your own cloud account if policy requires it.
• Treat document text as data. A document can contain text that looks like instructions. The assistant never follows them, and any action beyond answering, such as creating a ticket, needs a person's approval.
• Keep an audit trail. Log each question, the passages retrieved, the answer and the user. Restrict access to those logs and set a retention period.

#Common failure modes

#When a RAG assistant is the wrong tool

RAG answers questions from text. It fits poorly when the knowledge is not written down, when answers require calculations across many records, or when the documents are few enough to fit in a single prompt. If the real need is to act, such as resetting a password or issuing a refund, you need an agent with controlled system access and approvals. The same retrieval layer often powers customer support agents that answer from your help center and hand off to a person with context.

If you want to see how this would work on your own documents, read how we build knowledge assistants or book a strategy call. We will look at your sources, permissions and most common questions, and tell you plainly whether a knowledge assistant is worth building.

#Frequently asked questions

#How is a RAG knowledge assistant different from ChatGPT or Copilot?

Microsoft 365 Copilot is itself permission-aware and works well across content inside Microsoft 365. A custom assistant earns its place when your knowledge also lives outside Microsoft (a helpdesk, a database, a document system), when you need an evaluation set built from your own questions, or when you need control over the model and where it is hosted. General chat assistants without connectors answer only from their training data and the files you give them in a session.

#Does it need to be trained on our documents?

No. The model reads relevant passages at question time, so nothing is trained on your data. Answers stay current, and you can remove content by deleting or restricting the source document.

#What happens when the assistant does not know the answer?

It says so and points to the person or team who owns the topic. Unanswered questions are logged and reviewed, which shows you where documentation is missing.

#Can it run inside our own cloud?

Yes. The index, search components and model can all be deployed in your own AWS, Google Cloud or Azure account. Some teams use hosted models under enterprise data terms instead, depending on their data and policies.

#Do you build RAG assistants?

Yes. Our knowledge assistants work covers connectors, permission-aware retrieval, cited answers and a test set built from your team's real questions. If you are weighing a RAG development company against building in-house, the same test set is the fair comparison: ask each option to answer your own questions and show the sources.