Fine-Tuning 100M–3B LMs for Low-Latency, Auditable Web Extraction

16th August, 2026

I remember the moment clearly: a Product Hunt run returned dozens of messy HTML pages, and our heuristic extractor missed structured fields or returned garbled values. We could throw the latest 100B model at the problem, but latency, cost, and data governance made that impossible. The tension was simple — improve precision without blowing up CPU/GPU cost, latency, or our legal exposure. That’s the constraint set I optimized for when experimenting with fine-tuning small language models for extraction inside the Universal Scraper pipeline.

Why small models? The constraints were practical: limited GPU time for fine-tuning, sub-second inference targets for operator UI flows, and locally auditable behavior for safety reviews. Small models (100M–3B params) are cheaper to fine-tune and easier to run at inference on local or modest cloud instances. They are not magic; they require deliberate data engineering and architectural choices to be useful in production-like settings.

Mental model

Treat the fine-tuned small model as a stateful transformer that maps noisy, candidate-rich inputs to high-precision structured outputs, with three supporting systems:

- A retrieval/context layer that supplies grounding (document or chunk-level evidence). - A compact model tuned to the extraction task (field classification, normalization, or slot filling). - Deterministic post-processing and validation (regexes, unit checks, fallback heuristics).

The model’s job is constrained: it disambiguates and normalizes candidates, not invent missing facts. Grounding + strict output schemas keep the model’s freedom constrained and make small models effective.

High-level architecture / data flow

1) Scrape stage (HTML fetch, canonicalize) — our Universal Scraper runs a three-stage pipeline: raw fetch, sanitizer, candidate extraction [see “What I built”]. 2) Candidate generation — rule-based selectors and cheap NLP heuristics produce candidate spans for each field. 3) Context assembly — we chunk nearby text, attach metadata (URL, selector, timestamp), and optionally fetch embeddings from a local pgvector store for similar examples. 4) Fine-tuned inference — the small model runs on the assembled context + prompt template and returns a constrained JSON output. 5) Post-process + validate — deterministic checks and fallback heuristics accept/reject or mark for review. 6) Persist & audit — accepted outputs are saved with citations and fixture-testable audit trails.

A compact table of components

| Component | Purpose | |---|---| | Scraper (stage 1–3) | Produce cleaned text and candidates | | Context layer (ARIL-like KB) | Attach embeddings, citations, tenant-scoped docs [6] | | Fine-tuned small model | Map context -> structured output | | Post-processing | Validate and normalize outputs | | Fixture tests / audit | Regression tests and review gates |

Practical fine-tuning techniques I used

- Parameter-efficient fine-tuning (PEFT) — adapters or LoRA-style low-rank updates keep GPU memory low and speed up iterations. This is indispensable if you want many small, task-specific variants without rehosting an entire model. - Synthetic and semi-supervised labels — convert high-confidence heuristic outputs and templated paraphrases into training examples to enlarge datasets without manual labeling. - Strong prompt templates + constrained decode — instruct the model to output strict JSON with deterministic keys; combine with a grammar-based validator to reject malformed generations. - Retrieval augmentation — prepend nearest-neighbor chunks or exemplar I/O pairs to the prompt so the small model sees comparable cases; this often beats larger untuned models for edge cases [1]. - Fixture-driven regression — build a suite of scraping examples (HTML fixtures) so fine-tuning iterations are validated against real selector drift cases.

What I built around this idea

I integrated local fine-tuning and inference experiments into the Universal Scraper extraction pipeline. Implemented pieces:

- Three-stage extraction pipeline: raw fetch → sanitizer → candidate extraction, with fixture tests and live-safe Product Hunt / Finsmes / billing-lead runs. These runs exercise selector robustness and grounding. - Local scripts for PEFT-style fine-tuning, with training data serializers that pull examples from the fixture suite and from high-confidence heuristic outputs. - Context assembly tied into a tenant-scoped local KB (ARIL-style) so the model receives embeddings and citations when available [6]. - Audit trails and approval gates: every inferred field stores provenance and goes through review flows in our UI before downstream use.

Operational gates and what’s not yet production:

- Model hosting at scale (multi-tenant GPU inference, autoscaling, SLA) remains an operational gate. - Any outbound messaging based on extracted leads is intentionally review-only; we do not send messages without explicit human approval (consent gates remain) — Universal Scraper remains subject to robots, rate limits, and selector drift.

Where this breaks (failure modes + mitigations)

1) Selector drift & webpage layout changes - Failure: Heuristics miss fields and the model never sees new patterns. - Mitigation: Continuous fixture sampling, scheduled re-scrapes, and lightweight active learning that surfaces low-confidence examples for human labeling.

2) Small-data overfitting - Failure: Model memorizes training quirks and fails on slightly different phrasing. - Mitigation: Augment training with paraphrases, adversarial negatives, and holdout fixtures; use early stopping and validation on recent live fixtures.

3) Hallucination or invented values - Failure: Model fabricates normalized values when context is missing. - Mitigation: Strict output schemas, grammar validation, and a “no-answer” token that forces fallback to deterministic heuristics or reviewer queues.

4) Latency / resource limits - Failure: Inference spikes increase operator wait time or exhaust GPU budget. - Mitigation: Use quantized models or CPU-optimized runtimes, cache inference results for repeat pages, and route heavy jobs to batch queues.

A practical checklist (5–8 checks)

1) Do you have 200–1,000 high-quality labeled examples for each critical field? If not, synthesize + validate. 2) Is your model using PEFT or other memory-light tuning to enable many experiments? If not, adopt adapters/LoRA. 3) Are prompts constrained and outputs validated against a schema? Enforce this end-to-end. 4) Do you attach provenance (URL, selector, chunk) to every extracted value? Store it with the record. 5) Are you running fixture-based regression tests on every fine-tune? Fail the pipeline on regressions. 6) Have you set explicit latency targets and a fallback path for slow queries? Make them part of SLIs. 7) Do you surface low-confidence extractions to a human review queue? Make review efficient with compact UI snippets.

Conclusion

Fine-tuning small language models can yield high-precision extraction without the cost and operational complexity of large models — but only if you pair them with grounding, strict schemas, deterministic post-processing, and good data hygiene. In constrained settings like the Universal Scraper pipeline, the combination of PEFT, retrieval augmentation, fixture tests, and human-in-the-loop review produces a practical, auditable system. The remaining hard work is operational: reliable hosting, continuous monitoring, and respectful adherence to provider robots and rate limits.

References

[1] Model Context Protocol — introduction. https://modelcontextprotocol.io/introduction

[2] Anthropic — Model Context Protocol background. https://www.anthropic.com/news/model-context-protocol

[3] arXiv:2005.11401 (relevant fine-tuning / transfer learning literature). https://arxiv.org/abs/2005.11401

[4] arXiv:2307.03172 (methods for calibration/data efficiency). https://arxiv.org/abs/2307.03172

[5] Universal Scraper and extraction pipeline (portfolio). https://github.com/deepanshuvermaa/my-portfolio

[6] ARIL local monorepo (tenant-scoped KB, embeddings, pgvector retrieval). https://github.com/deepanshuvermaa/my-portfolio

Other portfolio links referenced in the write-up:

- Amazon Voice Agent (runtime and audit features). https://github.com/deepanshuvermaa/my-portfolio - Listenly (local-first meeting copilot, citation assembly). https://github.com/deepanshuvermaa/my-portfolio