Designing Reliable RAG Pipelines: Retrieval, Grounding, and Failure Trade-offs

16th August, 2026

I used to get the same frustrated question from a product manager: “We have the docs in the database — why does the model keep making stuff up?” That tension — documents exist, model invents — is the real engineering problem RAG tries to solve. In this post I lay out a concrete mental model for RAG, show a clear data-flow example, explain what I’ve implemented in my projects, and enumerate where this all breaks and how I mitigate it in practice.

A short mental model

Think of RAG as two cooperating systems, not a single oracle:

- Retrieval: a filter that maps a query to a small set of evidence (documents or passages). It answers “what could be relevant?” - Generation (grounding): a synthesizer that composes an answer given the retrieved evidence and the prompt. It answers “how should we phrase a response using the evidence?”

Retrieval reduces the model’s search space; grounding constrains generation by surfacing provenance and applying rules (citation, answer format, thresholds). If either step is weak, you get hallucinations or omission.

Simple architecture / data-flow example

User query -> Query encoder -> Vector search (pgvector) -> Top-K passages -> Reranker + score threshold -> Context assembly (citations + prompt template) -> LLM -> Post-process (citations injected, answer filter) -> Response

Component responsibilities (mini table)

| Component | Responsibility | |---|---| | Encoder & vector DB (pgvector) | Fast semantic match; supports tenant scope/isolation | | Reranker | Improves precision for top results; helps reduce noise | | Context assembler | Builds prompt with numbered citations and instruction schema | | Generator (LLM) | Produces the final text constrained by the prompt | | Post-processor | Checks answer against thresholds, formats citations, logs audit events |

A concrete example: a meeting-copilot flow

1. Ingest meeting transcript -> chunk and embed each chunk. 2. On user query (“What did we decide about pricing?”) the system vector-searches chunks scoped to that meeting and tenant. 3. Reranker sorts the top 20 by relevance; we keep top 4 that pass a score threshold. 4. The assembler produces a prompt like: “Answer concisely. Use only the numbered citations. If none support the claim, say: ‘I don’t have a supported answer.’” 5. LLM returns answer + inline citation tokens; post-processor maps tokens to citation numbers, and records the provenance and audit event.

This exact flow is what I aim to make repeatable across products: explicit scope (tenant/meeting), embeddings, vector search, citation wiring, and evaluation hooks for precision/recall.

What I built around this idea

I’ve implemented several pieces tied to this pattern (evidence: code and projects in my portfolio) and I’ll be explicit about implemented vs still gated:

- ARIL (implemented pieces): tenant-scoped knowledge bases; documents -> chunks -> embeddings pipeline; pgvector-based retrieval; citation wiring; evaluation metrics; protected routes and tenant isolation. These are in a local monorepo with tests and fixtures. (Deployment gate: real production Postgres migration is still an environment gate.) [4]

- Listenly (implemented pieces): a local-first meeting copilot that assembles grounded context windows, numbers citations in summaries, and produces session summaries and recent-turn windows. (Operational gates: real meeting recordings and durable cloud session storage are intentionally gated.) [4]

In both projects I implemented evaluation hooks (store retrieved hits, gold answers, and compute precision@k) and audit logs that record which documents were used for an answer. Those afford human-in-the-loop review and targeted fixes.

Where this breaks (failure modes and mitigations)

1) Retrieval misses the evidence - Failure: Relevant passages were never retrieved (bad chunks, stale embeddings, wrong scope). - Mitigation: monitor recall on human-annotated queries; use larger K and reranker; log false negative cases for re-chunking and re-embedding.

2) Incorrect or noisy documents get retrieved - Failure: Low-quality or adversarial content ranks highly, contaminating the prompt. - Mitigation: add document quality signals (source trust score, age), apply filters, and run a safety/rate check on retrieved docs before assembly.

3) Model ignores evidence and hallucinates - Failure: Even with correct context present, the LLM fabricates unsupported claims. - Mitigation: constrain the prompt (explicit instruction to only use citations), include a “verbatim source snippet” appendix, and require the model to produce citation tokens for claims. If confidence is low, fall back to “I don’t know” or human review gate.

4) Prompt injection inside documents - Failure: Retrieved text contains attacker-crafted instructions that override the system prompt. - Mitigation: sanitize retrieved text (strip system-like directives), escape special tokens, and run a small adversarial classifier against retrieved passages.

5) Stale knowledge / data drift - Failure: Cached embeddings or old docs cause outdated answers. - Mitigation: timebox documents (TTL), incremental re-embedding, and expose last-updated timestamps in responses.

6) Privacy / multi-tenant leakage - Failure: Cross-tenant retrieval or accidental exposure of tenant data. - Mitigation: enforce tenant-scoped DB queries, schema-level isolation where possible, and fail-closed behavior for unknown tenant IDs. Audit trails and protected routes help detect misuse.

A practical checklist (5–8 checks) you can run before releasing a RAG feature

1. Tenant isolation: queries include tenant ID and vector DB enforces scoping. 2. Embedding health: run embedding-drift checks and sample similarity spot checks weekly. 3. Citation wiring: responses always include citation tokens mapped back to source IDs. 4. Thresholds: define minimum retrieval/confidence thresholds and a human-fallback path. 5. Audit & observability: every RAG answer must log retrieved doc IDs, scores, and the final prompt snapshot. 6. Safety filters: run a safety classifier on retrieved docs and assembled prompt. 7. Repro tests: fixture-based tests that assert that a known question retrieves expected doc IDs and that the generated answer cites those docs.

Where to put your effort first

- Make retrieval observable. If you can’t see which passages were chosen, you can’t debug. Logging the top-k hits and exposing a debug UI is the single most helpful investment. - Add small rules that make hallucination failures obvious (“If no supporting citation, respond with ‘I don’t know’”). Those rules reduce silent failures and give you time to iterate on retrieval quality.

Conclusion

RAG is powerful when treated as a system of components with clear responsibilities and observability. The most expensive step is diagnosing retrieval failures; design for logging, citation, and safe fallbacks first. In my work (ARIL and Listenly) I prioritized tenant-scoped retrieval, citation wiring, and audit events so that every answer can be traced back to evidence — the pragmatic steps that cut down hallucinations even when the LLM is imperfect.

References

[1] Model Context Protocol — Anthropic: https://www.anthropic.com/news/model-context-protocol

[2] Retrieval-augmented approaches papers (example): https://arxiv.org/abs/2005.11401

[3] RAG-related research: https://arxiv.org/abs/2307.03172

[4] My portfolio (code & projects, including ARIL and Listenly): https://github.com/deepanshuvermaa/my-portfolio

[5] Museum of Failure (examples and tests): https://github.com/deepanshuvermaa/museum-of-failure