Tradeoffs: Fine-Tuning, Retrieval (RAG), and Prompting for VAT/GST Assistants
16th August, 2026
Tradeoffs: Fine-Tuning, Retrieval (RAG), and Prompting for VAT/GST Assistants
I was building a VAT/GST assistant for accountants and hit a concrete engineering fork: the system must decide whether an invoice supports Input Tax Credit (ITC), cite the statutory text, and reflect rule changes within days. I had a small labelled invoice corpus and a growing set of regulatory PDFs — enough to test options, not enough to make tradeoffs trivial. Do I bake rules into a fine-tuned model, ground answers with retrieval and citations, or extract behavior from prompts? Each alters latency, update velocity, hallucination risk, and operational complexity. Below I lay out my mental model, the RAG pipeline I implemented in ARIL, where I used fine-tuning in Go2 GST, and a checklist to choose the right approach.
That scenario forces a choice between three levers: fine-tuning a model to bake domain knowledge in, using retrieval (RAG) to ground answers in documents, or obsessing over prompt design. Each option affects cost, latency, update speed, and hallucination risk.
My goal here is to give a clear mental model and a concrete architecture example, then show what I built around these ideas and pragmatic tradeoffs.
Mental model: three levers and the axes they move
Think of each approach as moving points along these axes: Accuracy for edge / domain rules, Updatability (how fast can new facts be applied), Hallucination risk, Latency/Cost per query, and Implementation complexity.
- Fine-tuning: pushes accuracy for narrowly defined tasks (classification, extraction) and reduces latency at inference time, but is costly to retrain, brittle across model changes, and slower to update as rules change. - Retrieval (RAG): keeps the core model generic but grounds answers in documents. Strong on updatability (replace docs), lower hallucination if citation and answer-filtering are enforced, but adds query-time complexity and latency. - Better prompts: lowest engineering friction — you can iterate quickly — but prompts alone can’t fix missing facts in the model; they can reduce some hallucinations and change behavior, but are sensitive to model updates and context-length limits.
A compact tradeoff table
| Approach | Strength | Weakness | |---|---:|---| | Fine-tune | High task accuracy for seen patterns | Expensive to update; needs labelled data | | Retrieval (RAG) | Fast updates, explicit citations | More infra; token-budget management | | Prompting | Fast iteration; cheap experiments | Limited by base model knowledge; fragile |
Concrete architecture / data-flow example (RAG with evaluation)
Here is the flow I use in ARIL for grounded answers:
1. Ingest: PDFs, invoices, statutes -> text extraction -> chunking into passages. 2. Embed: Passages -> embedding model -> store vectors in pgvector. 3. Retrieve: Query -> query embedding -> top-K by vector similarity (pgvector). 4. Filter & score: Use heuristics (date windows, tenant scope) and a lightweight reranker to reduce noisy passages. 5. Prompt assembly: Selected passages + instruction template + user question -> LLM. 6. Post-process: Answer + numbered citations; run small hallucination checks and consistency tests. 7. Evaluate: Store input/response/citations and metric outputs for offline evaluation.
This pipeline gives updatability (swap documents, re-index) and citationable results. It’s the pattern I wired into the ARIL monorepo: tenant-scoped KBs, chunking, embeddings, pgvector retrieval, citations and evaluation metrics [1].
When I still prefer fine-tuning
I reach for fine-tuning when: I have a clear task (e.g., invoice field extraction), sufficient labeled examples, and a need for very low-latency or deterministic outputs. In Go2 GST I used specialized extraction pipelines and a labelled ITC decision corpus; for those deterministic classification/extraction steps I favor fine-tuning or supervised models to improve precision on recurring patterns [2].
When prompts are the right first step
If you’re exploring, collecting data, or your domain is small deviations from general language, craft templates and evaluate them against a held-out set. Use prompt-scoring to identify failure modes before investing in fine-tuning or RAG.
What I built around this idea
- ARIL monorepo: I implemented tenant-scoped knowledge bases, document chunking, embeddings, pgvector retrieval, citation numbering, evaluation metrics, and protected routes. That pipeline is designed to support RAG scenarios and per-tenant isolation. A real production Postgres migration is still an environment gate (not completed) [1].
- Go2 GST: I built invoice extraction and an ITC decision corpus to support deterministic classification and coverage testing. Those components are implemented and exercised via fixture tests, but full live deployment into a billing pipeline and any automated message sending remain review gates (no external sends) [2].
Explicit: implementations listed above are local or testable in CI. Deployment to production Postgres, sending messages, or enabling live provider interactions remain operational gates and are not done.
Where this breaks (failure modes + mitigations)
1) Retrieval returns irrelevant or outdated passages. Mitigation: add freshness metadata, tenant-scoped filters, and a reranker + human-review feedback loop. Re-index when statutes change and mark passages with source and date.
2) Embedding drift / semantic mismatch between queries and docs. Mitigation: monitor retrieval recall@k, use hybrid filters (BM25 + vectors), and periodically re-embed the corpus with upgraded embedding models.
3) Hallucinations even after retrieval (model ignores passages). Mitigation: use strict prompt templates that force source quoting, apply post-generation verification against retrieved text, and fall back to “I don’t know” when confidence is low.
4) Cost/latency spikes from large context windows. Mitigation: chunk aggressively, apply passage scoring to limit tokens, cache recent results, and consider fine-tuning a small distilled model for low-latency paths.
5) Regulatory / privacy leaks (exposing private invoice data in prompts). Mitigation: tenant isolation, encrypt provider keys, redact or minimally include PII in prompts, and audit prompt logs [1].
A practical checklist (5–8 checks before choosing)
1. Define the success metric: precision/recall, hallucination rate, SLA latency. 2. Inventory data: labelled examples for fine-tune? Source docs for retrieval? 3. Run a prompt baseline: measure failures and collect error types. 4. Prototype RAG: ingest docs, run retrieval +/- BM25, measure recall@k. 5. Test a small fine-tune on the extraction task if you have >500–1k examples. 6. Add citation & post-check rules before presenting answers. 7. Build logging & evaluation: store (input, retrieved, response, metrics). 8. Plan updates: how will you refresh embeddings, re-run training, and deploy schema changes?
Conclusion
There’s no one-size-fits-all: fine-tuning wins when you need deterministic extraction and have labeled data; RAG wins when facts change often and citations matter; prompts win for quick iteration. I typically start with prompt baselines, add RAG for grounding and updatability, and selectively fine-tune for extraction modules where determinism and latency matter. In my projects (ARIL and Go2 GST) I combine these patterns: RAG for regulatory grounding, supervised models for structured extraction, and template-driven prompts to keep behavior stable while we iterate [1][2].
References
[1] ARIL monorepo (tenant-scoped KBs, embeddings, pgvector retrieval, citations, evaluation): https://github.com/deepanshuvermaa/my-portfolio
[2] Go2 GST (invoice extraction and ITC decision corpus): https://github.com/deepanshuvermaa/go2-gst
[3] Model Context Protocol (guidelines for structured context to models): https://modelcontextprotocol.io/introduction
[4] Anthropic model context / best-practices: https://www.anthropic.com/news/model-context-protocol
[5] Retrieval / RAG related research discussions: https://arxiv.org/abs/2005.11401
[6] Fine-tuning and related language model evaluation: https://arxiv.org/abs/2307.03172
Additional project evidence: https://github.com/deepanshuvermaa/trading-engine