Designing Separate Blob and Metadata Planes for Scalable Video Platforms
16th August, 2026
I once stared at a support ticket: a creator uploaded a 2 GB tutorial, the CDN served the MP4 fine, but searching for the video by topic returned nothing. The thumbnail existed, transcripts were missing, and the recommendation engine kept surfacing unrelated content. That quiet bug exposed a tension every video platform faces: media blobs live in cheap object stores, but the metadata that makes videos discoverable, safe, and usable needs fast, consistent, searchable systems—and the two scale in very different ways.
In this post I explain a simple mental model for that split, give a concrete architecture and data-flow example, and walk through operational failure modes and mitigations. I’ll also describe what I’ve implemented in my projects and what remains a deployment or operational gate.
A compact mental model
Think of a video platform as two orthogonal systems:
- The blob plane: immutable, large binary objects stored in object storage (S3-compatible), often served via CDN. These scale by throughput and capacity and benefit from tiering and lifecycle policies. - The metadata plane: small, structured records (title, tags, transcripts, renditions, thumbnails, embeddings, moderation state, usage metrics). This needs low-latency read/write, indexes for search, and often a vector-index for similarity. It scales by query volume and index size, not by TBs.
Keeping this separation explicit avoids two common anti-patterns: storing blobs in the database (costly, slow backups) and embedding heavy metadata inside the object storage (search and consistency nightmares).
Architecture and data-flow example
Below is a typical ingestion and surface pipeline I use in designs.
Uploader -> Ingest Service -> Transcoder / Derivative Worker -> Object Store (S3) + CDN | | v v Metadata Service <-----> Metadata DB (Postgres) + Search (Elasticsearch) + Vector (pgvector) | v Moderation & Audit (manual review queue)
Explanation, step-by-step:
1. Uploader: client uploads directly to pre-signed S3 URLs. The ingest service records a lightweight Upload record (uploader id, file size, expected renditions) in the metadata DB and returns an upload token. 2. On upload complete S3 triggers an event to the Transcoder/Derivative Worker (serverless or fleet). Worker generates HLS/DASH manifests, thumbnails, a transcript (speech->text), and lower-bitrate renditions. Each derivative is written back to object storage. 3. The worker atomically updates the Metadata Service with rendition objects, a canonical player URL, thumbnails, transcript and a job status. Small structured fields live in Postgres; search-able fields are mirrored into an index (Elasticsearch/OpenSearch), and embeddings are stored in pgvector or a specialized vector DB for similarity/recommendations. 4. CDN serves manifests and blobs. The player fetches metadata via an API that reads from the metadata DB (fast reads, cacheable) and search/index systems. 5. Moderation & audit: automated safety checks run on transcripts, thumbnails, and video frames. Failing items create review tasks; audit events are emitted and stored for compliance.
Why this layout works
- Object store is optimized for large binary throughput and cheap long-term retention. It’s cheap to scale capacity, but expensive for small-file operations and searching content. - Metadata DB handles fast lookups and transactions (e.g., atomically marking an upload complete and publishing). Search and vector indexes provide the query/semantic surfaces. - Separation reduces blast radius: migrating DB schema or scaling search is independent of petabytes of blob data.
What I built around this idea
My recent projects implemented several pieces of this pattern:
- ARIL local monorepo: tenant-scoped knowledge bases, document/chunk storage, embeddings and pgvector retrieval are implemented. That’s a tested pattern for storing small textual artifacts and running vector searches similar to metadata embeddings; a real production Postgres migration is still an environment gate [1]. - Listenly: local-first meeting copilot with grounded context assembly and citation handling shows how small context objects and citations can be assembled and indexed for retrieval; durable cloud sessions remain a gate [2]. - Google WhatsApp Scraper & Amazon Voice Agent: both implemented consent gates, audit events, approval workflows, and encrypted provider keys—patterns that map to the moderation and audit step in the video pipeline. Live streaming and provider acceptance are still operational gates [3][4].
Implemented vs gates (explicit)
- Implemented: tenant-scoped metadata storage with embeddings (pgvector), transcript storage, audit event trails, offline review queues, and pre-signed upload flow in local stacks. - Gates: migrating the metadata DB to a production Postgres cluster (HA, backups), full live streaming ingestion, CDN and multi-region replication, and automated global moderation with provider acceptance are operational gates.
Where this breaks (failure modes + mitigations)
1. Broken transcript / derived metadata pipeline: transcripts fail to arrive or are incorrect. - Mitigation: store a job-level state machine, exponential retry, fall back to cheaper ASR for degraded UX, and enqueue manual review with graceful degradation in UI. 2. Inconsistent metadata vs blob (orphaned blobs): blobs exist but no metadata record (or vice versa). - Mitigation: use S3 event-driven reconciliation jobs, periodic object/DB integrity sweeps, and implement two-phase commit patterns for critical flows (e.g., create DB record first, then finalize after upload confirmation). 3. Search or vector index becomes stale or overloaded. - Mitigation: append-only change logs for replay, versioned index builds, and rate-limit index updates with a near-real-time window; have a cold/async path for heavy re-indexing. 4. Cost runaway due to small files or high egress. - Mitigation: aggregate small files into packs where possible (e.g., thumbnails in combined manifests), use lifecycle tiering (S3 Infrequent/Glacier), and instrument egress per origin with quotas and alerts. 5. Moderation latency causing bad content to surface. - Mitigation: fail-closed publication, progressive enhancement (publish to private or limited audiences), and fast-path heuristics for safe/known creators.
A practical checklist (5–8 checks)
- Do you separate blobs from metadata? (S3 for blobs; Postgres/Search/Vector for metadata) - Are uploads atomic from the user’s perspective? (upload token + state machine) - Is your index (search/vector) replayable from a change log? (yes/no) - Do you have lifecycle policies for cold data and small-file aggregation strategies? - Are moderation and audit events stored and queryable? (immutable event trail) - Can you reconcile S3 vs metadata DB mismatches with scheduled sweeps? - Do you instrument cost per asset and set egress/storage alerts?
Conclusion
Scaling video platforms is mostly about keeping two orthogonal problems separable: cheap, durable blob storage and fast, searchable metadata. Make the boundaries explicit, design for eventual consistency across them, and instrument reconciliation and auditability. Small, structured metadata (including embeddings) is what makes content discoverable and safe—treat it as first-class data, even if the blobs live on a separate tier.
References
[1] ARIL local monorepo (pgvector retrieval, tenant-scoped KB): https://github.com/deepanshuvermaa/my-portfolio [2] Listenly (local-first meeting copilot): https://github.com/deepanshuvermaa/my-portfolio [3] Google WhatsApp Scraper (consent, review packs): https://github.com/deepanshuvermaa/my-portfolio [4] Amazon Voice Agent (audit events, encrypted keys): https://github.com/deepanshuvermaa/my-portfolio [5] Model Context Protocol (context & metadata patterns): https://modelcontextprotocol.io/introduction [6] "Attention Is All You Need" (relevance for embeddings & vector search): https://arxiv.org/abs/1706.03762 [7] "The Illustrated Transformer" (concepts applicable to embeddings and similarity): https://arxiv.org/abs/2005.11401