When to Decompose AI Assistants: Engineering Tradeoffs, Latency, and Hallucinations

16th August, 2026

Opening tension

I once had an engineering debate with a teammate: should we split a conversational product into several small “agents” (ASR, intent agent, retrieval agent, policy agent, tool agent, TTS) or keep one big controller calling an LLM with plugins? On paper decomposition yields clear ownership and specialization, but in practice it introduced new latency, brittle interfaces, and surprising cross-agent hallucinations. That tension—specialization versus coordination—is the practical boundary I will explore here.

Mental model: agents as roles in a play

Think of a multi-agent AI system as a small theater company. Each actor (agent) has a role and a script:

- Some actors are specialists (speech-to-text, retrieval, payment tool); they do a narrow job well. - A director (router/arbiter) assigns lines and coordinates timing. - An audience (user) drives the scene with input that may cause improvisation.

Good decomposition reduces cognitive load per component, improves testability, and lets you scale different parts independently. Bad decomposition creates lots of handoffs, repeated context, and the need for a stronger director.

Architecture and data-flow example

Here’s a concrete architecture I use as a mental template for voice + knowledge-enabled assistants:

1. Edge: Client audio capture, local prefilter (silence detection), consent check. 2. ASR agent: produces transcripts, emits token-level timing. 3. Router/Orchestrator: decides which agents to call based on intent, context, and policy. 4. Retrieval agent: fetches tenant-scoped docs/embeddings (ARIL-style vector DB) and returns citations. 5. LM agent: composes user-visible text, uses retrieval context and tool outputs. 6. Policy/Approval agent: enforces consent, gating (e.g., approval required for outbound calls). 7. TTS/Voice runtime: renders response; includes health checks and encrypted provider keys (Amazon Voice Agent pattern). 8. Telemetry + Audit: logs events, audit trails, and evaluation metrics.

Dataflow example (simplified):

1. Audio -> ASR agent -> "I want to reorder my last delivery" 2. Router checks session memory and intent model -> routes to Retrieval + OrderTool. 3. Retrieval returns candidate order details with citation blocks from tenant KB (ARIL). 4. LM agent composes a clarification prompt enriched by citations; Policy agent checks consent for calling PaymentTool. 5. If approved, OrderTool executes; Telemetry records the transaction attempt; TTS speaks confirmation.

A compact table of agents and responsibilities

| Agent | Responsibility | Key contract | |---|---:|---| | ASR | Speech→text, timestamps | Output transcripts + confidence | Retrieval | Vector/store lookup | Return documents + metadata/citations | LM | Compose text, hallucination guard | Use only provided context + citation protocol | Policy | Consent, gating | Decision + human-approval hooks | Tool | External actions (payments, SMS) | Idempotent call, deterministic response | Telemetry | Audits, metrics | Immutable events

What decomposition helps

- Ownership and testing: narrow interfaces let teams write fixture tests and mocks (I applied this pattern in Amazon Voice Agent with a provider-agnostic mock-first voice runtime). - Specialization: you can tune retrieval for embeddings (ARIL tenant-scoped vectors), while separately optimizing latency for ASR. - Safety and governance: central policy agents can enforce consent and approval gates consistently (used in the WhatsApp Scraper and Amazon Voice Agent designs).

What decomposition hurts

- Interface drift: many small agents mean many contracts to keep in sync; mismatched context formats lead to silent failures. - Performance: round trips add latency; each agent may repeat parts of the context, ballooning token costs. - Coordinated correctness: when multiple agents each try to “help” (e.g., multiple agents rewriting the prompt), you can get contradictory outputs or hallucination amplification. - Operational surface: more services to deploy, monitor, and secure (encrypted keys, health probes, tenant isolation).

What I built around this idea

I applied these patterns across several projects; here’s a candid mapping of implemented pieces vs gates:

- Implemented: Amazon Voice Agent — provider-agnostic mock-first voice runtime, multilingual detection, consented memory, audit events, approval-gated calls, encrypted provider keys, health probes, autonomous test reports, and operator UI. (Live streaming and real provider acceptance are still deployment gates.) - Implemented: ARIL monorepo — tenant-scoped knowledge bases, documents, chunks, embeddings, pgvector retrieval, citations, and evaluation metrics. (Production Postgres migration is an environment gate.) - Implemented patterns repeated elsewhere: consent/review gates in Google WhatsApp Scraper, grounded context assembly in Listenly, and three-stage extraction pipeline in Universal Scraper. Many components are functionally tested and UI-driven but remain review-only or blocked from live external interactions.

I emphasize what is still a gate: live provider acceptance, external sending of messages, and durable cloud persistence are guarded by review or environment policies rather than missing code.

Where this breaks (failure modes and mitigations)

1) Interface drift and schema mismatch - Symptom: Retrieval returns documents with unexpected metadata; LM ignores citations. - Mitigation: Contract tests, backward-compatible adapters, and automated schema checks during CI.

2) Latency accumulation - Symptom: Multiple serial calls add noticeable delay in voice interactions. - Mitigation: Parallelize independent calls (ASR + context prefetch), cache embeddings, and set streaming pipelines (ASR streaming -> incremental retrieval).

3) Cross-agent hallucinations - Symptom: Retrieval returns irrelevant docs; LM hallucinates a citation that doesn’t exist. - Mitigation: Citation-first prompt patterns, evidence scoring, and post-generation verification (use a small verifier agent to check claims against sources) [1].

4) Policy/consent bypass - Symptom: Tool agent performs sensitive action without proper approval under race conditions. - Mitigation: Fail-closed policy engine, idempotent tools with transaction logs, and strong audit trails (approval gating and operator UI).

5) Operational complexity and secrets - Symptom: Encrypted keys leaked or provider changes cause outages. - Mitigation: Centralized secret rotation, provider-agnostic mocks for testing, and health probes with graceful degradation.

A practical checklist (5–8 checks)

- Define clear contracts for each agent (input shape, outputs, error codes). - Implement contract tests and CI schema checks for every agent boundary. - Enforce a policy/approval agent; make it fail-closed for sensitive actions. - Use citation/evidence protocols and a small verifier for assertions.[1] - Parallelize independent agents where possible; measure tail latency end-to-end. - Centralize telemetry and immutable audit events for replay and debugging. - Mock external providers first and require human approval to enable live providers. - Plan for migration gates: separate code completion from environment deployment (DB, provider onboarding).

Conclusion

Decomposing an AI system into agents is powerful: you gain testability, governance, and specialization. But you also add coordination costs, latency, and new failure modes. The right balance is situational—use decomposition where it reduces cognitive and operational load, and centralize where coordination complexity outweighs specialization gains. In practice, that means strong contracts, policy-first designs, verifier agents for claims, and deployment gates for risky external effects.

References

[1] Model Context Protocol and evidence/citation best practices: https://modelcontextprotocol.io/introduction

[2] Anthropic — Model Context Protocol announcement and safety guidance: https://www.anthropic.com/news/model-context-protocol

[3] Multi-agent and coordination research (example papers on emergent coordination): https://arxiv.org/abs/2005.11401

[4] Tool and multi-agent emergent behavior research: https://arxiv.org/abs/2307.03172

[5] My portfolio and project repositories: https://github.com/deepanshuvermaa/my-portfolio

[6] Museum of Failure (operational lessons): https://github.com/deepanshuvermaa/museum-of-failure

[7] Go2 Payroll and Go2 GST (example production-focused modules and tests): https://github.com/deepanshuvermaa/go2-payroll, https://github.com/deepanshuvermaa/go2-gst

[8] Trading engine (design patterns for gating and safe defaults): https://github.com/deepanshuvermaa/trading-engine