An assistant that answers questions based on a company's procedures, contracts, and notes often seems simpler than it really is. The shortcut is tempting: feed the documents into a model, then ask questions. Yet this does not make the model a reliable repository of the organization's documentary assets. The system must first know which document to consult, which version is authoritative, and what the user is allowed to see.
This is precisely the role of RAG, or retrieval-augmented generation. The term refers to an architecture in which the system searches a corpus for passages before asking the model to formulate an answer. The principle was formalized in research on retrieval-augmented generation. In an enterprise setting, its value does not lie in the acronym: it avoids confusing a model's general knowledge with the rules specific to an organization.
A well-designed RAG system therefore does not promise that the machine “knows” everything. It establishes a retrieval pipeline. A question comes in, the system identifies the relevant passages, sends them to the model with instructions, and then presents an answer that must point back to its sources. This is a fundamental difference. The answer can be useful because it is verifiable, not because it sounds convincing.
RAG addresses a search problem, not every AI problem
Consider a sales department looking for the procedure that applies to an exceptional discount. The relevant document may be a note updated the day before, buried among very similar files. Without search, a general-purpose assistant has no reason to know this rule. With RAG, it can retrieve the passage, provide its title, and help rephrase it. It does not replace the source; it reduces the time needed to find it again.
This mechanism explains why RAG is often preferable to retraining for documents that change. Updating a procedure should not trigger a model development project. It should be enough to re-index the document, archive the old version, and make the new one available. The model remains a language component; the corpus remains the reference. For an initial scope, AI project scoping primarily serves to establish this boundary: which data informs the answer, who maintains it, and what outcome will be considered acceptable.
RAG does not eliminate errors. If the search returns the wrong passage, if the document is obsolete, or if the question goes beyond the available sources, the model may produce an unreliable answer. It must therefore be able to say that it does not know, display the excerpts used, and refrain from answering outside its scope. The NIST AI Risk Management Framework reinforces a useful discipline: manage risks during design, deployment, and evaluation rather than discovering them once the tool has been made available to everyone.
The quality of the corpus determines the quality of the answers
The word “document” conceals several challenges. A scanned PDF, a price table, a contradictory procedure, and a mailbox cannot be split up in the same way. Dividing them into fragments that are too short causes conditions and exceptions to disappear. Keeping fragments too long dilutes the search and overloads the context sent to the model. There is no universal configuration; the real corpus and real questions must be tested.
Then comes the issue that demonstrations readily avoid: access rights. An HR assistant must not retrieve a compensation note simply because it is semantically similar to a general question about salaries. Permissions must be applied before generation, at the document level and, where necessary, at the fragment level. The CNIL's recommendations on developing AI systems call for data protection to be integrated by design. In a RAG system, this means identified sources, defined roles, and retention of audit trails appropriate to the use case.
Testing cannot be limited to a few questions written by the project team. A representative set of questions must be assembled: those the assistant should answer, those it should refer to a human, and those it should not be allowed to access. The quality of retrieval, the answer's fidelity to the retrieved passage, and the clarity of the sources can then be measured. A brief answer with a precise citation is better than an elegant synthesis that cannot be verified.
Start with an identifiable source of truth
The first suitable RAG project does not cover “all of the company's knowledge.” It starts with a bounded corpus that is already useful and that a responsible owner agrees to keep up to date. Product documentation, support procedures, or a quality reference system: the right candidate is one where searching takes time and an incorrect answer can be detected.
This approach also leaves room for infrastructure choices. Depending on the sensitivity of the data, the expected level of control, and the tools already in place, the hosting and connectors will differ. The issues addressed by sovereign hosting of models and data do not disappear because the system is limited to document search. They must be decided before connecting to the corpus, not afterward.
RAG becomes valuable when it puts company information at the heart of the answer and makes it possible to verify that answer. To achieve this, the question is not merely “which model should we choose?” It is more demanding: which source of truth do we want to make searchable, by whom, and with what evidence?