Testing LLM Systems: Evaluation Architecture, Failure Modes, and Operational Gates

16th August, 2026

I once watched a demo where an invoice-extraction LLM gave a confident tax credit decision that turned out to be impossible for the vendor’s jurisdiction. It “sounded good” — fluent, structured, and fast — but it would have failed a downstream legal gate. That kind of false confidence is why I treat LLM evaluation as engineering, not ceremony.

In this post I share the mental model and a concrete architecture I use to test LLM systems beyond surface metrics. I’ll show what I’ve built around these ideas in my projects, where the implementation stops and operational gates remain, four-plus failure modes with mitigations, and a short practical checklist you can apply quickly.

Why “it sounds good” is dangerous

LLMs optimize for plausibility, not truth. That means standard metrics (perplexity, BLEU, even human-fluency ratings) can hide failures: wrong facts with good form, made-up citations, brittle retrieval, and prompt sensitivity. My mental model splits evaluation into three orthogonal layers:

- Unit behavior: deterministic checks for core logic (parsers, schema, boundary conditions). - Retrieval and grounding: whether context used by the model is correct, complete, and cited. - System and safety: integration, adversarial inputs, and approval/operational gates.

Architecture / data-flow example

Here’s a compact architecture I use for an LLM-backed pipeline (invoice extraction and ITC decision as an example):

Component | Role ---|--- Uploader/ETL | Accept PDFs, images; run OCR + lightweight parser. Chunker + Embedding | Break into chunks, vectorize, store in pgvector-backed retrieval (tenant-scoped). Retriever | k-NN + taxonomic filters to collect grounding passages. Prompt Composer | Constructs prompt with citations, context window control, and MCP-like metadata [1][2]. Model | LLM call with safety-hardened system messages. Post-process & Validator | Schema checks, numeric reconciliation, cross-field rules (unit tests). Approval Gate / Auditor UI | Human review, consent, audit trail, encrypted keys (for voice or provider calls).

Data flow: inputs -> OCR -> chunk/embed -> retrieve -> prompt compose -> model -> validate -> human audit/approval -> (gated) action.

A small example: invoice extraction validation

1) The model extracts amount = 12,345.67 and taxable = true. 2) Post-processer verifies that sum(line_items) == amount +/- tolerance. 3) Retriever fetches supporting paragraphs (invoice header, tax line) and the system ensures the model included citation anchors. 4) If numeric reconciliation fails, the system raises an exception and sends to human review.

What I built around this idea

I applied these patterns across several codebases; here’s what’s implemented and what is still an operational gate:

Project | Implemented | Deployment/operational gates ---|---|--- ARIL (local monorepo) | Tenant-scoped knowledge bases, document chunking, embeddings, pgvector retrieval, citations, evaluation metrics, protected routes. | Real production Postgres migration is currently an environment gate. Amazon Voice Agent | Provider-agnostic mock-first voice runtime; multilingual detection; consented memory; audit events; approval-gated calls; encrypted provider keys; health probes; autonomous test reports; operator UI. | Live streaming, barge-in, transfer, and real provider acceptance remain gates. Go2 GST | Invoice extraction, ITC decision corpus coverage, fixture tests for statutory calculations. | Live/send of sensitive documents and external tax submission flows remain review-gated. Listenly | Local-first meeting copilot with grounded context assembly, citation numbering, session summaries. | Real meeting recordings and durable cloud sessions remain gates. Universal Scraper & Google WhatsApp Scraper | Three-stage extraction pipeline, adversarial safety suite, lead-service matching, review-only send gates; Growth Engine UI and durable local review storage. | Scraping subject to robots/rate-limits/provider blocking; approved-send remains gated.

Where this breaks (failure modes and mitigations)

1) Hallucinated citations — model fabricates a supporting paragraph that doesn’t exist. - Mitigation: Require retrieval anchors; store passage fingerprints and fail if model-citation mismatch exceeds threshold. Add a post-check that opens the exact chunk to a human if mismatch.

2) Retrieval drift / stale index — embeddings or selectors return irrelevant chunks (selector drift). - Mitigation: Regular fixture tests against known queries, continuous re-indexing schedule with unit tests, and conservative time-to-live for embeddings.

3) Overfitting to eval set — models optimized to pass the eval but break in production. - Mitigation: Maintain holdout adversarial tests and use dynamic red-team inputs; rotate evaluation suites and run blind human reviews.

4) Latency/timeouts and partial outputs — long chain calls or streaming failures lead to truncated results. - Mitigation: Timebox prompts, require idempotent retryable post-processors, and implement graceful degradation (e.g., fall back to summary-only with audit flag).

5) Privacy leaks or credential exposure in prompts — sensitive data accidentally included in LLM context. - Mitigation: Sensitive-field scrubbing, encrypted provider keys, and strict tenancy isolation (implemented in ARIL and Voice Agent) plus audit events for every call.

A practical checklist (5–8 checks)

1) Unit test non-ML pieces: parser, numeric reconciliation, schema validation. 2) Retrieval sanity: every model answer that references facts must include chunk IDs and pass chunk-existence checks. 3) Adversarial suite: daily run of targeted negative examples that historically trigger hallucinations. 4) Approval gates: require human sign-off for high-risk actions (payments, outbound messages). 5) Monitoring: log model latency, token counts, author attribution, and mismatch rates for citations. 6) Canary policy: route X% traffic to a stricter policy model + human audit before full rollout. 7) Secrets & tenancy: enforce encrypted keys and tenant-scoped data boundaries. 8) Maintenance: scheduled re-index + regression test for retrieval selectors.

Where evaluation tooling came from

I lean on two practical ideas when composing prompts and context: explicit context metadata (inspired by Model Context Protocol thinking) and deterministic post-processing gates. See model context ideas for background [1][2]. I also rely heavily on fixture-driven tests and local-first designs so human reviewers can reproduce model inputs offline.

Conclusion

Good LLM evaluation treats the model as one component in a system: validate inputs, ground outputs, fail safely, and keep humans in the loop where risk is material. The pattern I use — chunk/embed/retrieve + MCP-like metadata + validator + approval gate — has helped me surface problems that “it sounds good” would have missed. Start small: unit-test the non-ML parts, require citations for facts, and add one approval gate for high-risk actions.

References

[1] Model Context Protocol intro — https://modelcontextprotocol.io/introduction

[2] Anthropic on model context ideas — https://www.anthropic.com/news/model-context-protocol

[3] Attention Is All You Need (context on transformers) — https://arxiv.org/abs/1706.03762

[4] Synthetic/Adversarial evaluation literature — https://arxiv.org/abs/2005.11401

Portfolio evidence (code & projects)

[5] My portfolio repository — https://github.com/deepanshuvermaa/my-portfolio

[6] ARIL, Go2 GST, and related repos — https://github.com/deepanshuvermaa/go2-gst

[7] Go2 Payroll and statutory calculation examples — https://github.com/deepanshuvermaa/go2-payroll

[8] Trading engine and other projects (museum of failure) — https://github.com/deepanshuvermaa/trading-engine