RAG — retrieval-augmented generation — is how business AI answers from your documents instead of hallucinating. The concept is simple; the gap between naive RAG (a weekend tutorial) and RAG that survives real users is where projects live or die. This is the implementation guide. (Deciding between RAG and fine-tuning? That comparison first.)
The pipeline, and where each stage goes wrong
1. Ingestion and chunking. Documents split into retrievable pieces. Naive fixed-size chunking slices tables in half and separates answers from their context. Production chunking respects structure — sections, tables kept whole, headings attached — and stores metadata (source, date, department) for filtering. Most "RAG is inaccurate" complaints trace back here.
2. Retrieval. Embedding search alone misses exact terms (product codes, names); keyword search alone misses paraphrases. Production systems run hybrid retrieval (both, merged) plus a reranker that reorders candidates by true relevance. This stage decides your accuracy ceiling — generation cannot cite what retrieval never found.
3. Grounded generation. The model answers only from retrieved context, cites sources, and says "not in the knowledge base" when retrieval comes back thin — the honesty behavior users trust. Prompting for refusal is as important as prompting for answers.
4. Evaluation. A test set of real questions with known-correct sources measures retrieval hit rate, answer groundedness, and refusal correctness — run on every change. 50 good questions from actual users beat 500 synthetic ones.
The failure modes that sink builds
- Stale indexes. Documents change; pipelines must re-ingest on schedule or the system confidently cites last year's policy.
- Permission leaks. If HR docs and public docs share an index without access filtering, the chatbot becomes a data breach. Metadata filtering is security, not decoration.
- Answering everything. A RAG system without refusal behavior degrades into a normal hallucinating chatbot precisely when retrieval fails.
Costs and timelines (India, 2026)
A production RAG system — hybrid retrieval, reranking, evals, refresh pipeline — runs ₹2–6 lakh and 4–10 weeks depending on document mess. The document cleanup is often a third of the effort; budget for it rather than resenting it.
RAG quality is measurable, so demand measurement: any builder should show retrieval hit rates on your documents before you commit. Send us a sample of yours and we will do exactly that — or start with the cost calculator.
Frequently asked questions
What is RAG and why do businesses use it?
Retrieval-augmented generation: the AI retrieves relevant passages from your own documents and answers only from them, with citations. It grounds LLMs in your policies, catalogs, and knowledge instead of hallucinating from generic training data.
How much does RAG implementation cost?
A production system — structure-aware chunking, hybrid retrieval with reranking, evaluation sets, and refresh pipelines — runs ₹2–6 lakh in India in 2026, over 4–10 weeks. Document cleanup is often a third of the effort.
Why do naive RAG implementations fail?
Fixed-size chunking slicing tables and context apart, embedding-only retrieval missing exact terms like product codes, no refusal behavior when retrieval comes back thin, stale indexes citing outdated policies, and missing permission filtering that leaks restricted documents.
What is hybrid retrieval and do I need it?
Running semantic (embedding) and keyword search together, merging results, then reranking by true relevance. Yes — embeddings miss exact identifiers, keywords miss paraphrases; production accuracy needs both. Retrieval quality sets your system's ceiling.