Designing Layered Decision Systems for Assistants That Defer and Gate Actions

16th August, 2026

Designing Layered Decision Systems for Assistants That Defer and Gate Actions

I once watched a prototype voice assistant confidently read a partial account number back to a user and keep going when the user said, "That doesn't sound right." That moment crystallized the core engineering tension I work on: users want helpful, human-like assistants, but correctness, privacy, and safety demand conservative behavior. In this post I lay out a three-layer decision model, a compact retrieval-to-action architecture, a concrete data flow, and what I implemented in ARIL, Amazon Voice Agent, and Listenly — plus a deployable checklist for gating real-world actions.

Below I share a mental model, an architecture I use, an example data flow, what I’ve implemented in recent projects, where this breaks, and a practical checklist you can run before giving an assistant authority over real-world actions.

Mental model: three decision layers

I find it useful to think of an assistant as three layered decision systems stacked on top of each other:

- Retrieval + Evidence: fetch candidate facts and the provenance (documents, embeddings, timestamps). Confidence comes from signal quality: number of supporting docs, freshness, and semantic match score. - Generator + Attribution: the LLM composes an answer from retrieved evidence. This layer should be forced to cite and expose uncertainty (e.g., "Based on document X, it appears..."). - Action/Gate + Audit: before performing sensitive actions (calls, writes, payments), a policy layer applies approval gates, consent checks, and audit logging. If confidence/policy fail, escalate to a human or return a safe refusal.

Those layers let you reason about two orthogonal properties: factual correctness and operational safety. You can be conservative on one and permissive on the other, depending on risk.

Architecture and data flow (example)

Here’s a compact architecture I use; imagine this as a voice or chat assistant that can answer questions and place calls.

1. Inbound: user utterance or chat message -> language detection and intent classifier. 2. Context assembly: retrieve tenant-scoped KB chunks, recent-memory window, and meeting transcripts (where available); compute vector search using pgvector. 3. Evidence scoring: rank documents by semantic score + freshness + explicit trust tags. 4. LLM prompt: pass a structured prompt with numbered citations, a model context protocol-like context block, and explicit instructions to cite and to refuse when insufficient evidence [1][2]. 5. Safety & policy: run response through safety checks — PII leakage detection, consent checks (is user allowed to see this), action gating (is this a request to place a call or send a message?). 6. Action/Response: either (a) return a citation-backed answer, (b) present a refusal with escalation options, or (c) open an approval flow (human-in-loop) and log the event. 7. Audit & metrics: persist the full exchange with audit events, approval decisions, and evaluation metrics for offline review.

Small table: key signals and where they’re used

| Signal | Layer | Use | |---|---:|---| | Vector score, doc age | Evidence | Confidence + freshness filtering | | Consent flags | Policy | Block or allow memory/PII access | | Approval state | Gate | Enable action (e.g., outbound call) | | Audit trail | All | Post-hoc review, compliance |

Concrete example: "Transfer $500 to vendor X"

- Intent classifier tag: high-risk-financial. - Retrieve account docs and consent flags. If consent absent or vector evidence below threshold, the policy layer returns: "I can't complete that transfer — would you like to request approval?" If approved, record approval event and proceed. All steps logged.

What I built around this idea

I’ve implemented many of these pieces across recent projects; here’s what’s actually in code and what remains a deployment/operational gate:

- Implemented: - ARIL local monorepo: tenant-scoped knowledge bases, documents -> chunking -> embeddings -> pgvector retrieval, citations, evaluation metrics, and protected routes for multi-tenant access control [3]. This is the retrieval + evidence backbone I use. - Amazon Voice Agent: provider-agnostic mock-first voice runtime, multilingual detection, consented memory flags, audit events, approval-gated calls, encrypted provider keys, health probes, and a full operator UI for approvals and review. The runtime is designed so actions are gated until an approval event is present. - Listenly: local-first meeting copilot with grounded context assembly, session summaries, recent-turn windows, and citation numbering for answers built from meeting materials.

- Operational gates (not yet run in open production): - ARIL: real production Postgres migration is still an environment gate — the system runs locally and in staging but a live migration has not been completed. - Amazon Voice Agent: live streaming, barge-in, transfer, and real provider acceptance remain gated by provider approvals. - Listenly: real meeting recordings and durable cloud sessions are currently gated; the assistant works locally with fixture data.

I treat these gates as deliberate: until the approval and audit flows are exercised end-to-end in an environment that has regulatory and provider sign-offs, the system defaults to dry-run or review-only modes.

Where this breaks (failure modes + mitigations)

1. Hallucination from poor retrieval - Failure: LLM confidently invents a citation or fabricates details when retrieval fails. - Mitigation: force the model to only use numbered retrieved documents, suppress free-form knowledge, and add a confidence threshold to refuse when evidence is weak.

2. Stale or contradictory documents - Failure: the assistant cites an old policy or conflicting vendor info. - Mitigation: include document timestamps and trust scores in ranking; prefer recent authoritative sources and show provenance in the answer.

3. Privacy leakage via memory - Failure: assistant repeats sensitive info persisted in memory without consent. - Mitigation: consented memory flags, PII detectors before output, and fail-closed defaults for high-sensitivity types.

4. Authorization / action misuse - Failure: assistant executes an action (call, payment) without proper approval. - Mitigation: approval-gated flows that require an explicit human operator or out-of-band token; encrypt provider keys; log every attempt as an audit event.

5. Context window and truncation - Failure: long context causes relevant evidence to be trimmed, degrading answers. - Mitigation: context prioritization (recent-turn + high-confidence docs), model context protocol-style blocks to structure context [1][2], and explicit citation numbering to ensure provenance survives truncation.

A practical checklist (before giving an assistant action authority)

1. Provenance: every fact must link to a source with a timestamp and tenant ID. 2. Confidence thresholds: define numeric cutoffs for "answer" vs "defer". 3. Consent controls: require explicit opt-in for storing/using personal data. 4. Approval gates: require human approval for destructive or high-risk actions. 5. Audit logging: immutable event logs for all decision points and approvals. 6. Adversarial tests: run a safety suite of prompts and edge cases. 7. Fail-closed defaults: on any failure in checks, refuse or escalate. 8. Monitoring & drift: track retrieval hit rates, citation accuracy, and top failure modes.

Conclusion

Designing an assistant that "knows its limits" means engineering for uncertainty: assemble and surface evidence, force the model to attribute, and gate actions behind policy and human approval. I build systems that are cautious by default and instrumented for review, so the assistant can help without creating new risks. When you combine tenant-scoped retrieval, citation-first answers, consented memory, and approval-gated actions, you get a predictable operational surface — but you must keep the deployment gates and audit plumbing in place until end-to-end safety is proven.

References

[1] Model Context Protocol — introduction and ideas: https://modelcontextprotocol.io/introduction

[2] Anthropic on prompting and context best practices: https://www.anthropic.com/news/model-context-protocol

[3] ARIL (tenant-scoped KB, embeddings, pgvector retrieval) — my portfolio repo: https://github.com/deepanshuvermaa/my-portfolio

[4] Amazon Voice Agent (mock-first voice runtime, consented memory, audit events): https://github.com/deepanshuvermaa/my-portfolio

[5] Listenly (local-first meeting copilot, grounded context assembly): https://github.com/deepanshuvermaa/my-portfolio

[6] On hallucination and grounding in NLG (research context): https://arxiv.org/abs/2005.11401

[7] Retrieval-augmented generation and evaluation techniques: https://arxiv.org/abs/2307.03172