Designing an Evidence-First Research Agent: Retrieval, Provenance, and Synthesis

16th August, 2026

I once watched a colleague paste five papers into a chat and ask “what should I cite for X?” The agent confidently produced a paragraph of prose citing three papers I’d never heard of. The prose read well, but when we traced it back there were no quoted passages, only loose paraphrases and a chain of inference. That’s the tension: for research work you need verifiable evidence—exact spans, provenance, and access to the original—rather than polished, unverifiable summaries.

In this post I describe a mental model and a concrete architecture for an evidence-first research agent. I then explain what I implemented in my projects (notably Universal Scraper and ARIL), what remains an operational gate, common failure modes and mitigations, and a short checklist you can use to get started.

Why evidence-first?

Summaries hide uncertainty and provenance. For research decisions you want: - The original text span that supports a claim (quote or excerpt). - Metadata: URL, timestamp, author, document id, chunk id, and retrieval score. - The exact query used and any relevance/explainability signals.

Mental model

Think of the agent as two cooperating layers: 1) Retrieval & provenance: return ranked evidence items (snippets + metadata). No paraphrase unless explicitly requested. Store linkage back to full documents. 2) Reasoning & synthesis: operate over the returned evidence items, produce a labeled synthesis that references evidence IDs. Keep the chain of reasoning explicit and auditable.

A simple data-flow

source -> extractor -> document store -> chunker/embeddings -> vector index -> retriever -> evidence filter -> reasoner

Small architecture table

| Component | Responsibility | |---|---| | Scraper / Extractor | Fetch pages, extract structured text & metadata, store raw HTML/JSON | | Document Store | Keep raw documents, canonical IDs, timestamps, consent flags | | Chunker & Embedder | Create chunks with offsets and embeddings (pgvector in my stack) | | Retriever | Return top-K evidence items with scores and provenance | | Evidence Filter | Enforce consent, freshness, adversarial checks, and format snippets | | Reasoner / Agent | Synthesize answers referencing evidence IDs (not replacing them) |

Example data flow (short): - Universal Scraper runs a three-stage pipeline to extract article content and metadata. The extractor stores raw HTML plus a normalized document record. [see Evidence] - ARIL ingests the document, chunks it (char offsets preserved), computes embeddings, and stores vectors in pgvector. The retriever returns top-N chunks and their document metadata. - The agent presents a ranked list of quoted snippets and a synthesis that annotates which snippet supports each claim.

What I built around this idea

Implemented (code & local systems): - Universal Scraper: a three-stage extraction pipeline with fixture tests and live-safe runs for Product Hunt, Finsmes, and billing leads. It stores raw content and structured outputs suitable for indexing. (Implemented.) - ARIL (local monorepo): tenant-scoped knowledge bases, documents, chunks, embeddings, pgvector retrieval, citations, evaluation metrics, and protected routes. This is the core retrieval/provenance layer used to serve evidence to the agent. (Implemented in monorepo.) - Listenly / grounding pieces: tools for assembling grounded context windows and enumerating numbered citations so synthesized text can point back to evidence. (Implemented locally.)

Operational / deployment gates (not yet live): - ARIL: a real production Postgres migration is an environment gate—data locality and tenancy hardenings are implemented but live migration to production Postgres is gated. - Universal Scraper: scraping still needs to respect robots, rate limits, and provider blocks; selector drift and provider-level blocking are operational realities that must be handled in deployment. - Voice and messaging projects (Amazon Voice Agent, Google WhatsApp Scraper): have many offline features (mock runtime, consent flows, audit trails) but live provider acceptance, streaming and transfer remain gates.

What’s implemented vs. what’s gated (quick summary table)

| Feature | Implemented | Gate | |---|---:|---| | document extraction & raw storage | ✅ | none | | chunking + embeddings + pgvector retrieval | ✅ | none | | tenant-scoped KB + routes | ✅ | Postgres migration | | auto-send messaging (WhatsApp) | ✅ (review-only) | approved-send gate |

Where this breaks (failure modes + mitigations)

1) Hallucinated synthesis without linked evidence - Symptom: agent writes claims without pointing to snippets. - Mitigation: enforce policy: every claim must list evidence IDs; reject chains with low overlap between claims and evidence.

2) Stale or invalid evidence (page changed or removed) - Symptom: snippet no longer exists at URL. - Mitigation: store raw HTML and char offsets at ingestion time; display stored excerpt and a freshness timestamp; re-fetch on-demand.

3) Scraper selector drift / provider blocking - Symptom: extractor returns empty or malformed documents. - Mitigation: three-stage extraction with fixture tests, monitoring for schema changes, rate-limited retries, and a fall-back to human review when confidence drops.

4) Privacy and consent violations - Symptom: the agent surfaces PII or content that lacked consent. - Mitigation: tag ingested documents with consent and data-policy flags; evidence filter enforces consent rules before returning snippets; audit events recorded.

5) Embedding mismatch and retrieval failure - Symptom: retrieval misses relevant passages or returns semantically related but non-supporting text. - Mitigation: use hybrid search (BM25 + embeddings), tune chunk size and overlap, and add evaluation metrics to track recall on example queries.

A practical checklist (5–8 checks)

1. Store raw documents and normalized metadata (URL, timestamp, author) at ingestion time. 2. Preserve chunk-to-document mapping and character offsets for every snippet. 3. Return evidence as (snippet, doc_id, chunk_id, score, URL, fetch_time) — never paraphrase by default. 4. Implement consent and audit flags; filter evidence by policy before display. 5. Use hybrid retrieval and measure recall with a small labeled testset. 6. Keep a human-in-loop approval gate for any external publish/send action. 7. Record retrieval and synthesis evaluation metrics and replay logs for post-hoc analysis. 8. Monitor extractor health with fixture tests and alerts for selector drift.

Where to start in code

If you’re following my approach: extract and store raw pages first, build a small vector index (pgvector works well), and attach robust metadata. Use the retrieval layer to return snippets and make your agent restricted to operate over those snippets only, emitting evidence IDs in the output. The ARIL monorepo shows these patterns in a tenant-aware structure; Universal Scraper demonstrates robust, testable extraction pipelines. See the linked repos below for implementation details.

Conclusion

For serious research workflows, the agent’s job is to surface verifiable evidence and let humans (or auditable downstream systems) decide. Treat retrieval and provenance as first-class; synthesis is a convenience layer that must remain transparent, cited, and auditable. My work on Universal Scraper and ARIL follows this principle: collect raw truth, index it carefully, and make every claim traceable back to a stored snippet.

References

[1] Anthropic — Model Context Protocol announcement: https://www.anthropic.com/news/model-context-protocol

[2] Model Context Protocol introduction: https://modelcontextprotocol.io/introduction

[3] On retrieval-augmented generation and grounding ideas (survey / background): https://arxiv.org/abs/2005.11401

[4] Agent and evaluation research context: https://arxiv.org/abs/2307.03172

[5] My portfolio repository (projects & code): https://github.com/deepanshuvermaa/my-portfolio

[6] Universal Scraper evidence and pipeline: https://github.com/deepanshuvermaa/museum-of-failure (see scraper components)

[7] ARIL monorepo and retrieval experiments: https://github.com/deepanshuvermaa/air-canvas (tenant-scoped KBs and retrieval patterns)

(Archive references used for background inspiration: https://archive.li/4bx3g, https://archive.li/WNYo6, https://archive.li/3xst4, https://archive.li/cbMOU)