Designing a Data Curation System Around a Vector Database

Your model fails at 2 a.m. on a rainy street. You have the clip. You have the logs. You even have the exact frame where confidence collapsed.

Now what?

For most teams, the answer is a week of Slack threads, a half-remembered S3 prefix, and someone scrubbing through video files by hand. The training infrastructure is fine — you can kick off a job in minutes. The bottleneck is upstream: out of the petabytes you've collected, which three thousand samples actually matter for this failure?

That question is what a data curation system exists to answer. This post walks through a design for one: first the user flow, then the architecture that makes it executable, then the six things that will bite you if you skip them.

The setup assumed throughout is unstructured data — images, video, sensor streams, text — collected continuously from production, embedded into a vector space, and queried by similarity when something goes wrong. The skeleton is identical for autonomous driving, visual inspection, speech, and LLM fine-tuning. Only the encoder and the trigger definitions change.

The shape of the problem

Two things happen on very different timescales, and conflating them is the most common design mistake.

Continuous, unattended: data arrives, gets embedded, gets indexed. No human is involved. This runs forever and has to be cheap.

Episodic, interactive: an event fires, an engineer sits down, and within one working session they need to go from "here's the failure" to "here's a training job running on a versioned dataset." This is rare, bursty, and has to be fast — not in throughput, but in iteration latency.

The system is really two systems glued together at the vector index. Design them with different priorities.

Part 1 — The user flow

The primary user is an ML engineer. The design goal is that steps 2 through 4 can be repeated ten times in twenty minutes, because that's what curation actually looks like.

Two details in this flow carry more weight than they look like they do.

Step 2 should accept both modalities. Query-by-example handles "find me more of this." Text query — via a CLIP-style joint embedding space — handles "find me the thing I suspect exists but haven't seen yet." Teams that build only the former end up unable to probe for gaps proactively.

Step 4 should record rejections. When an engineer reviews 500 candidates and picks 80, the 420 they passed over are a labeled signal about what "relevant" means. Log them. Later, that log is what lets you train the curation policy itself instead of hand-tuning heuristics forever.

Part 2 — The system architecture

The record schema

Getting this right is roughly half the battle, and it's very hard to change later.

{
  "id": "clip_9f3a#frame_1204",
  "embedding": [/* 768 dims */],
  "embedding_version": "dinov2-vitl14@v3",   // ← partition key
  "raw_uri": "s3://raw/2026/08/10/clip_9f3a.mp4#f=1204",
  "captured_at": "2026-08-10T21:14:02Z",
  "source": { "device": "unit-7731", "sensor": "cam-front" },
  "content_hash": "phash:a91c...",            // near-duplicate detection
  "label_state": "unlabeled | auto | human_verified",
  "model_signals": { "conf": 0.31, "entropy": 2.7, "disagree": 0.44 },
  "retention": { "policy": "P2Y", "consent_id": "..." }
}

Never put raw data in the vector database. Store vectors, metadata, and a pointer. Keep originals in object storage. Teams that blur this line discover it when they need to rebuild the index and find that "rebuild" now means moving petabytes.

Six things that will bite you

1. Embedding versions are a first-class concern

The moment you swap encoders, every vector in your index becomes meaningless — not wrong, just incomparable. Queries silently return garbage.

Make embedding_version a partition key from day one. Roll out new encoders by dual-writing to a separate index and migrating readers gradually. Retrofitting this later means taking the system down, and re-embedding a billion samples is not a weekend project.

2. Top-K returns the same thing a thousand times

Similarity search over video is brutal about this. Your top 1,000 results are frequently one three-second clip. The engineer scrolls, sees the same intersection over and over, and concludes the search is broken.

Two mandatory post-processing stages:

• Deduplication — enforce a minimum time gap within a source clip; use perceptual hashing to collapse near-identical frames.
• Diversity sampling — from K retrieved, select M that actually differ. k-center greedy or MMR both work well and are cheap.

Without these, retrieval quality looks terrible even when the index is perfect.

3. Training only on hard cases will wreck your baseline

Curated data is biased by construction — that's the point. But feed a model nothing but its own failures and you get catastrophic forgetting, dressed up as a mysterious regression on cases that used to work.

Make the mix ratio an explicit, tracked parameter (base : curated = 9 : 1 is a reasonable starting point) and keep a fixed replay set in every run. This is the default failure mode of the whole pipeline; treat it as a known hazard rather than a surprise.

4. Vector similarity alone is not enough

"Last winter, at night, in this specific region" is not something embeddings encode reliably. Real curation queries are almost always hybrid: structured filters and semantic similarity.

Design for metadata pre-filtering ahead of the ANN search, and include a brute-force fallback path for when the filter is selective enough that approximate search stops making sense. Check that your chosen vector database does filtered search well — implementations vary a lot, and some degrade badly at high filter selectivity.

5. It's easy to forget the new data in evaluation

The curated slice goes into training. Everyone remembers that part. What gets skipped is holding a portion out for evaluation.

Skip it and you permanently lose the ability to answer "did we actually fix it?" — you're left judging by vibes and spot checks. Always carve a holdout from the newly curated scenario, and always run the full regression suite alongside it. Both are hard gates for promotion.

6. Lineage is a compliance requirement, not a nicety

You need to trace model_version → dataset_version → sample_id → raw_uri in both directions. When a deletion request arrives, you walk that chain backward — and it has to reach all the way into the vector index, not just the data lake.

Manifests must be immutable. A dataset version that can change out from under you isn't a version.

What to build first

Don't build all nine layers before shipping anything. This order produces something useful at every step:

1. Ingestion + metadata catalog. Not losing data is the only thing that's irreversible. Start here.
2. Offline batch embedding + vector DB + a search API. Even as a CLI, "find me things like this" changes how the team works.
3. The curation UI. This is what determines whether anyone actually adopts the system. Under-invest here and the rest becomes shelfware.
4. Dataset registry wired into training. This is the moment the loop closes and the flywheel starts turning.
5. Automated triggers + active-learning scoring. Now the system proposes what to look at instead of waiting for a human to notice.

Steps 1 and 2 are infrastructure. Step 3 is product. Step 4 is where the compounding starts. Step 5 is the part everyone wants to build first and should build last.

The point of all this

The value isn't the vector database. It's the loop.

A model that fails, gets fixed, and ships in a week is on a fundamentally different trajectory from one that fails and waits a quarter for the next data collection campaign. Every rotation of the loop makes the next failure easier to find, because the index is denser and the triggers are better calibrated.

Build the loop first, even if every component in it is embarrassingly simple. A crude flywheel that turns beats a sophisticated one that doesn't.