RAG Architecture: building verifiable search over enterprise data

Production RAG consists of an indexing pipeline and an answer pipeline. Indexing receives documents, extracts structure, splits content, enriches metadata, computes embeddings and stores versions in a search index. The answer pipeline transforms a question, retrieves candidates, reranks them, assembles context and only then calls the LLM.

This separation is essential for reproducibility. An answer should cite a specific document version and passage, while a deleted source must disappear after reindexing. Without versioning and lineage, it is impossible to determine whether the model failed or retrieval supplied stale text.

The parser should preserve headings, tables, lists, page numbers and section relationships. Flattening everything into plain text destroys the structure of a contract or policy. Chunking should follow semantic boundaries with a size ceiling instead of cutting every document at the same character count.

Each chunk carries a document ID, version, section, source URL, access labels, timestamps and checksum. An embedding is a derived artifact, not the source of truth, so it can be recomputed when the model changes. Reindexing must be idempotent and remove old-version chunks to prevent contradictory duplicates.

Vector search captures semantic similarity but can miss an exact SKU, contract number or rare term. Full-text search handles exact matches but is weaker on paraphrases. Hybrid search runs both and merges candidates, improving recall across mixed enterprise queries.

A reranker evaluates a smaller candidate set against the actual question. It improves precision but adds latency and cost, so retrieve a moderate pool, rerank it and send only the top passages to the model. A relevance threshold lets the system answer honestly when the corpus does not contain enough evidence.

The LLM receives the question, instructions and bounded context with stable source identifiers. It answers from those passages and returns citations that the application validates before display. A citation should resolve to a document and location the current user can access, not merely a model-generated filename.

Apply ACL filtering before retrieval, not after generation. Otherwise restricted content has already entered the prompt and can influence the answer. Treat instructions found inside documents as data, not system commands. Logs must not retain secret context without the same access and retention policy.

An evaluation set should contain real questions, expected sources, acceptable answers and unanswerable cases. Evaluate retrieval separately with recall at K, precision and expected-passage rank. Evaluate generation for grounding, completeness, citation quality and correct refusal.

In production, monitor empty results, low relevance scores, citation clicks, user feedback, latency and cost. Compare every chunking, embedding, reranker or prompt change on the same regression set. Otherwise RAG optimization becomes a sequence of impressive demo questions rather than engineering.