What Vector Databases Really Store — and Which Responsibilities Remain Upstream

16th August, 2026

A small engineering tension

I once built a customer-facing RAG prototype that returned plausible but wrong answers. The vector store happily returned semantically similar chunks, but the LLM confidently invented facts with no indication of provenance. The team’s question was simple: “Is the vector DB broken?” The short answer: no—what’s broken is our mental model of what a vector database stores and what responsibilities we still have elsewhere.

A clear mental model

Think of a vector database as a fast, approximate similarity index plus a thin metadata store. It keeps numeric representations (embeddings) that capture semantic signals and pointers to source text and metadata. It does not contain truth, business logic, or guarantees about the completeness, recency or correctness of the text behind those vectors.

What it stores (essentials)

- Embeddings: fixed-length numeric vectors produced by an embedding model. These are the indexed items. - Identifiers/pointers: stable IDs or URIs that map the vector back to source text or documents. - Metadata: document-level data such as tenant-id, chunk offsets, timestamps, authorship, and small tags used to filter results. - Auxiliary attributes: quality scores, embedding version, and precomputed scalar values for hybrid search.

What it does not solve (responsibilities left to you)

- Truth or provenance validation: vectors can point to text, but they don’t prove that text is correct or up-to-date. - Logical reasoning or multi-step workflows: the vector store is a lookup mechanism, not an orchestrator. - Conversation state and business rules: manage session state, gating, or consent outside the vector index. - Indexing strategy decisions: chunking, embedding versioning, and what gets stored are upstream design choices.

A concrete ingestion + retrieval flow

Below is the pattern I use and teach. It separates responsibilities so the vector DB remains simple and auditable.

1) Ingestion pipeline

- Source (scrapers / uploads) -> chunker (size, overlap, heuristics) -> embedder (model vX) -> write to vector DB with metadata (tenant, doc_id, chunk_index, embed_version).

2) Query pipeline

- User query -> embed using same embedder vX -> ANN search (top-K) -> apply metadata filters (tenant namespace, freshness) -> re-rank/score -> assemble context with citations -> prompt model (with instructions to cite or not hallucinate) -> return answer + citation pointers.

Small architecture table

| Component | Key responsibility | |---|---| | Chunker | Produce recoverable text chunks with offsets | | Embedder | Produce consistent embedding vectors; versioned | | Vector DB | ANN index, metadata store, retrieval API | | Reranker | Optional — improves ordering, uses scoring logic | | LLM & Prompt | Consumes assembled context and citation pointers |

What I built around this idea

I applied this pattern in ARIL (my local monorepo). Implemented features include tenant-scoped knowledge bases, documents, chunking, embeddings, pgvector-backed retrieval, citation pointers, evaluation metrics, and protected routes — all designed to reflect the responsibilities above. Evaluation metrics helped detect regressions from embedding model changes or chunking tweaks.

Implemented vs gates

- Implemented: tenant-scoped KBs, document/chunk lifecycle, embedding generation and storage, pgvector retrieval, citations, evaluation metrics, authentication-protected routes. - Environment/deployment gates: real production Postgres migration is an environment gate (so ARIL runs locally with those features but migration to a managed production DB is pending).

Related portfolio pieces that used the same idea

- Listenly: local-first meeting copilot relies on grounded context assembly and citation numbering — same retrieval patterns. - Universal Scraper and Google WhatsApp Scraper: source pipelines that feed chunkers and embeddings; both remain review-only with send gates.

Where this breaks (failure modes and mitigations)

1) Embedding drift / model mismatch

- Failure: updating the embedder yields vectors not comparable to earlier ones; searches return poor neighbors. - Mitigation: version embeddings and reindex in batches; keep embedding_version in metadata and fall back to reencode on read or reindex gradually.

2) Stale or incorrect source text

- Failure: vector points to old or incorrect documentation; the LLM amplifies errors. - Mitigation: add timestamps, freshness filters, and a pipeline to invalidate or flag outdated chunks; include provenance in prompts.

3) Tenant leakage or incorrect scoping

- Failure: users see results from another tenant due to filtering bugs. - Mitigation: enforce tenant namespace at the vector DB layer (logical isolation), validate at read-time, and add tests for fail-closed behavior (as I did in ARIL).

4) Chunking causes lost context or split facts

- Failure: an important sentence is split across chunks; retrieval returns half the fact and the model hallucinates the rest. - Mitigation: tune chunk size and overlap, preserve sentence boundaries, and store chunk offsets so the system can surface surrounding text when needed.

5) Similarity is semantics, not citation quality

- Failure: semantically similar but untrusted sources surface first. - Mitigation: hybrid scoring that mixes semantic similarity with business trust signals (domain authority, recency, manual ratings) stored as metadata.

A practical checklist (5–8 checks before you call it production-ready)

1) Embedding versioning present and reindex plan documented. 2) Tenant and auth gating enforced at write and read paths (fail-closed tests). 3) Chunking strategy logged (size, overlap) and reversible mapping to source text. 4) Metadata: timestamps, source_id, embed_version, trust_score populated for each vector. 5) Citations: every retrieved chunk includes a pointer to the original doc and byte offsets. 6) Monitoring & evaluation: nightly retrieval checks and sample QA tests to detect drift. 7) Encryption for at-rest embeddings and access controls for keys.

Conclusion

Vector databases are a high-performance similarity primitive. They shine at retrieving semantically relevant chunks, but they do not absolve you from designing chunking, embedding versioning, provenance, access control, freshness, or orchestration. When you build with clear separation of concerns — ingestion, index, retrieval, re-ranking, and LLM prompting — you gain predictable behavior and operational controls. In ARIL I implemented those pieces locally with tenant isolation, citations, and evaluation metrics; the remaining gate is environment-level migration to a production Postgres instance.

References

[1] Model Context Protocol — Anthropic: https://www.anthropic.com/news/model-context-protocol

[2] Model Context Protocol introduction: https://modelcontextprotocol.io/introduction

[3] A paper on retrieval/embedding techniques (background): https://arxiv.org/abs/2005.11401

[4] A recent arXiv on retrieval-augmented generation and related techniques: https://arxiv.org/abs/2307.03172

[5] My portfolio and ARIL evidence: https://github.com/deepanshuvermaa/my-portfolio

[6] Related projects (examples of pipelines and scraping used to feed KBs): https://github.com/deepanshuvermaa/universal-scraper