Extracting Context from Live Video: The Architecture Got Simpler When We Stopped Building Features

We started with a straightforward-sounding problem: take live video streams delivered over HLS and DASH, and extract meaningful context in near real time. Map each moment to a standard content taxonomy so that ad decisioning has something useful to work with, and flag brand-safety risk while we're at it.

The ML architecture came together in an afternoon. The interesting part came later, when we started asking what each component actually cost us — not in GPU-hours, but in exposure. By the time we finished, roughly a third of the pipeline had been deleted, the GPU budget had gone down, and the accuracy target was still intact.

This is the write-up of that process, in the order we actually went through it.

Part 1: The pipeline

Ingest

You're consuming a manifest, not producing one. That single fact drives most of the latency budget.

• HLS: poll the m3u8. Low-Latency HLS adds EXT-X-PART, blocking playlist reload, and preload hints.
• DASH: parse the MPD, walk SegmentTemplate with $Number$ or $Time$. LL-DASH uses CMAF chunked transfer.
• Both are converging on CMAF/fMP4 — one segment format, two manifest dialects.

Before touching a single frame, harvest the free signals: SCTE-35 markers (EXT-X-DATERANGE on the HLS side, emsg on the DASH side), EXT-X-PROGRAM-DATE-TIME, ID3 timed metadata, and whatever EPG you can get. Ad break boundaries come from here, and they turn out to define the real deadline.

DRM is a contract problem, not an engineering problem. Encrypted streams cannot be decoded without a license. If you don't have a partner relationship that provides a clear feed, the project doesn't exist. Plan for this before you plan for anything else.

If you can negotiate access to the contribution feed (SRT, RIST, Zixi) rather than the distribution output, do it. Latency, quality, and DRM all improve at once.

Decode

ffmpeg or GStreamer with NVDEC, ideally through a zero-copy pipeline. When selecting GPUs, check the NVDEC engine count before you check tensor throughput — decode density is the binding constraint far more often than people expect.

Sample frames at keyframes or 1–2 fps. Take audio in full; it's cheap and it's the most valuable signal you have.

Signal extraction

Your ASR choice determines your cluster size. Parakeet-TDT-0.6B-v3 posts an RTFx above 3,300 against Whisper large-v3's 68.56 — roughly 49× the throughput. Whisper large-v3-turbo, which cut the decoder from 32 layers to 4, lands around 216× real time.

But Parakeet's multilingual v3 covers 25 European languages and does not include Korean, Japanese, or Chinese. If your channels are in an unsupported language, you're on Whisper-class throughput, and your GPU count triples. One line in a model card, three times the hardware.

Fusion and classification

The IAB Content Taxonomy spans 700+ categories across four hierarchical tiers. The 2.x → 3.0 migration was a breaking change — parent categories were removed — so version management is a live concern, not a one-time import.

Three approaches, and the answer is a hybrid:

• A. Embedding retrieval. Embed the node descriptions, do cosine similarity. With ~1,500 nodes you don't need a vector database; in-memory FAISS is plenty. Absorbs taxonomy updates without retraining.
• B. Fine-tuned hierarchical multi-label classifier. Best precision, needs labeled data.
• C. LLM with constrained decoding to valid node IDs. Best zero-shot behavior, slowest and most expensive.

Use A to narrow to top-k candidates, then B or C to rerank. Tiers 1–2 resolve on A alone; only tiers 3–4 need the rerank. Add temporal smoothing to stop labels from flapping between adjacent windows, plus parent-child consistency constraints and per-tier confidence calibration.

Part 2: Latency — you're managing two different clocks

Clock one: pipeline latency.

Clock two: semantic maturity. This one matters more, and no amount of compute fixes it.

Six seconds of audio will not produce a reliable Tier 3 label no matter how fast your GPUs are.

The reframe that changes the economics: your actual deadline isn't "real time." It's the SCTE-35 CUE-OUT pre-roll, typically 4–8 seconds of advance notice before an ad break. What you need is the last 30–60 seconds of context, retrievable within about two seconds of an ad request. That's a completely different engineering problem, and an order of magnitude cheaper.

If you do emit provisional labels early and revise them later, confirm first that your downstream consumers can accept a revision. Many can't, in which case you emit final labels only.

Part 3: Cluster design

Four principles that matter more than the box diagram:

1. Batch across streams, not within them. A thousand concurrent channels fills a batch of 128 naturally. This is the entire cost story. Use deadline-aware batching so the batch window is derived from your SLA rather than picked arbitrarily.
2. Backpressure means dropping, not queueing. Live streams cannot be slowed down. Write an explicit drop policy: OCR first, then frames, then — never — audio.
3. Demand-driven activation. Don't analyze channels nobody is requesting ads against. When 90% of your channel count is long tail, this is the single largest cost lever available.
4. Media sequence number is your natural idempotency key.
5.  

Sizing, 1,000 concurrent 1080p channels

Plus 200–400 vCPU and about 5 Gbps sustained ingest — roughly 1.6 PB per month.

That works out to $0.02–0.07 per stream-hour in GPU cost. If that number exceeds your incremental CPM lift, nothing else in this document matters.

Part 4: Cloud or your own hardware

Live stream analysis is a sustained 24/7 load, which is the least favorable possible profile for cloud economics. It also has enormous event-driven spikes, which is the most favorable profile. Both are true of the same workload.

Provider notes. AWS has by far the deepest media stack — if your video workflow already lives there, it's the default. GCP has the best L4 price-performance and the least painful managed Kubernetes. Azure is strongest for enterprise compliance and sovereign regions but thinner on native video tooling.

Recommendation: own the floor, rent the peak. Baseline load on owned hardware in a colo, peered directly with your content sources at an internet exchange. Event spikes, new regions, and training in cloud. Below about 32 GPUs, though, the operational overhead of running your own cluster eats the savings — stay in cloud, or use a specialist GPU provider at 40–60% off hyperscaler rates.

Part 5: The turn — "we're only analyzing" is not a legal position

This is where the project stopped being an ML problem.

The instinct is that passive analysis is safe: we're not redistributing, not publishing, not acting on anything. That instinct is wrong in three specific ways.

1. Copying is the infringing act. Downloading a segment is a reproduction. Decoding to frames is another. The EU's temporary-copy exemption requires that the copy have no independent economic significance — and analysis performed to increase ad revenue plainly has independent economic significance. The exemption doesn't reach you.

TDM exceptions are narrower than people assume. The commercially usable one under the EU DSM Directive requires lawful access and the absence of a rightsholder opt-out, which effectively makes lawful access a two-step test.

2. Collection is the violating act, for biometrics. Illinois BIPA provides a private right of action with liquidated damages of $1,000 per negligent violation and $5,000 per intentional one, and plaintiffs need not prove any individual harm — failure to obtain prior written consent is itself the violation. "We didn't do anything with the data" is not a defense.

In May 2026, seven broadcast journalists, podcasters, audiobook narrators, and voice actors filed nine coordinated class actions alleging that voiceprints were extracted from publicly available recordings without consent. The source material was broadcast audio. That's our exact fact pattern.

And the theory is expanding: a live case argues that even where a product isn't marketed as biometric, functionality that distinguishes speakers by vocal characteristics may constitute voiceprint collection. Speaker diarization sits squarely inside that.

3. Detection creates knowledge, and knowledge creates duties. If your pipeline detects illegal content, you now know. Willful blindness is gone, and you may be holding the material.

There's also an underappreciated one: the EU AI Act's high-risk obligations were deferred — Regulation (EU) 2026/1744 entered into force on 27 July 2026, pushing standalone Annex III systems to 2 December 2027. But the Article 5 prohibited practices took effect on 2 February 2025 and were not deferred. Emotion recognition and biometric categorization inferring race, political opinion, religion, or sexual orientation are prohibited outright, with penalties up to 7% of global turnover. It is remarkably easy to design a taxonomy node that walks into this by accident.

Part 6: What ownership actually fixes

We own the channels. Does that solve it?

Yes, for copyright — completely. Reproduction is a technical fact; infringement is unauthorized reproduction. A copyright holder copying their own work isn't infringing, and doesn't need an exception to rely on. There's no residual exposure. It's binary, not a gray zone.

But two traps:

Channel ownership ≠ content ownership. A single stream is a stack of rights layers — in-house production, licensed film and series, sports feeds, wire agency footage, background music, inserted ad creative, UGC clips, and the performers themselves. A single in-house news broadcast contains agency footage, music, and social clips. Inventory rights by layer, not by channel.

"Authorized to service" ≠ authorized to analyze. Copyright is a bundle. A distribution agreement typically grants the public transmission right, which carries incidental copies necessary for transmission. Analysis copies aren't necessary for transmission. Different right, separate grant needed.

That's a contract amendment, not a structural obstacle — and it's an easy negotiation, because it costs the rightsholder nothing and better ad performance often improves their revenue share. Language that gets approved usually pairs the grant with three reassurances: no redistribution, no reconstruction, short retention.

Licensee may reproduce, decode, and process the Content solely for automated analysis, metadata extraction, contextual classification, and content indexing, provided that (i) no portion of the Content is publicly displayed or made available to any third party; (ii) derived data does not permit reconstruction of the Content in whole or substantial part; and (iii) working copies are deleted within twenty-four hours of processing.

Negotiate model training rights separately. Attaching them to the same sentence sharply increases resistance. And add a derived-data ownership clause — if it's silent on who owns the transcripts, embeddings, and labels, that becomes a dispute later.

What ownership does not fix: the people in the footage never consented to anything. Biometric law, portrait rights, GDPR purpose limitation, AI Act Article 5, and performer agreements are orthogonal to copyright. Owning the content buys you nothing there.

Part 7: The traffic light

Three questions per feature:

1. Is this signal attributed to a specific individual?
2. Is it persisted?
3. Does it leave the system?

All three no → green. Any yes → yellow. Yes to #1 and it generates a biometric template → red.

🟢 Green — just build it

ASR. OCR. Shot boundaries. Scene and object classification. Audio event tagging. Brand and logo detection. Location at country/city/landmark granularity. Language ID. SCTE-35, EPG, and as-run logs.

Output: taxonomy labels and brand-safety grades.

Two preconditions: in-house content, and transcripts, frames, and embeddings are ephemeral — only labels persist.

This group alone delivers Tier 1–3 taxonomy and most of brand safety. Roughly 85–90% of the original goal. It needs essentially no legal review: you're classifying your own content, storing nothing about individuals, and shipping nothing outside.

🟡 Yellow — one gate each, independently

These are independent. You don't have to solve all of them to open one.

🔴 Red — don't build it

Face recognition and face embeddings. Speaker identification and speaker embeddings. Age, gender, race, or emotion inference. Cross-stream person matching. Searchable person/brand/location indexes. Exporting keyframe images. Circumventing any access control.

Red items aren't negotiable — they're design constants. Fixing them here makes everything else easy.

What you give up: less than you'd think

Contextual classification asks what a moment is about, not who is in it. Person identification was always a nice-to-have. Removing it also removed GPUs from the bill.

Two things worth flagging

Embeddings are invertible. CLIP/SigLIP frame embeddings implicitly encode faces. Storing years of them while claiming you don't do face recognition is a weak position. Persist labels, not embeddings — which is cheaper anyway.

Brands and locations are fine legally, and awkward contractually. Trademark isn't an issue; "there's a Nike logo in this frame" isn't trademark use. The problem is that conquesting — the most commercially attractive application — collides with your own sponsorship and product-placement agreements. Ship brand avoidance first; gate conquesting behind an as-run join.

For location, precision is the risk dial. Country and city are content attributes. Building-level, real-time location alongside identifiable people is something else. And keep in mind that "this footage was shot in a church" is a content attribute, while "this person was in a church" is special-category personal data — the distinction is entirely about attribution, which is exactly what the green design already prevents.

Store locations as standard geocodes and render display strings per locale. Place names are politically contested in ways that will otherwise force a full re-labeling pass.

Part 8: Regional regulation — there is no approval to obtain

There is no license, permit, or filing for contextual video classification anywhere. It isn't broadcasting, banking, or medical devices. What exists instead:

Korea. The AI Framework Act took effect 22 January 2026, imposing transparency, safety, and high-impact-operator duties. High-impact AI means systems materially affecting life, physical safety, or fundamental rights — medical diagnosis, credit scoring, hiring, autonomous driving. Ad classification isn't in scope, and a classifier isn't generative AI, so the labeling duties don't apply either. Action required: log it in your AI system register with a documented "not high-impact" determination.

EU. With biometrics removed, there's no Annex III entry path, so the system isn't high-risk. Article 5 is avoided by design. Article 50 transparency targets generative and deepfake systems. And ePrivacy doesn't apply at all — no terminal access means no consent banners, no consent strings, no TCF. Action required: a DPIA, an Article 30 record, and an Article 27 representative if you're established outside the EU.

United States. No federal statute; 20-plus state comprehensive privacy laws and growing. BIPA, CUBI, and Washington's health-data law are all avoided by the no-biometrics design. State AI statutes target consequential decisions — employment, lending, housing, insurance — not advertising. The one to check separately is COPPA, if you own children's channels.

Elsewhere. Japan's Article 30-4 is unusually permissive and actually works in your favor. The UK is principles-based with no separate AI statute. China is the one jurisdiction with a genuine algorithm filing requirement — irrelevant unless you're entering that market.

The accreditation that actually gates you is industry, not government. MRC accreditation is effectively mandatory to sell brand-safety measurement; TAG certification clears advertiser RFPs. Both are global and single-instance, not per-region.

Right shape: one global architecture, one document per region. Not regional forks of the system.

The thing that surprised us

Contextual targeting sits in an unusually favorable regulatory position, and it's not an accident — it's the reason the category grew as third-party cookies receded.

• No terminal access → ePrivacy doesn't apply
• No individual profiling → GDPR Article 22 doesn't apply
• Not generative → AI labeling duties don't apply
• No biometrics → the highest-risk tier is avoided entirely
• No Annex III path → not high-risk under the AI Act

The green-group design keeps all of that.

We went into this expecting compliance to be a tax on the architecture. It turned out to be the opposite. Every constraint we accepted removed a component: no face recognition, no speaker embeddings, no persistent frame storage. The pipeline shipped lighter than the original design, with fewer GPUs, less storage, a smaller attack surface, and a legal posture that fits in a paragraph:

The system analyzes first-party content, generates no biometric identifiers, does not persist source media or transcripts, and outputs only taxonomy labels and brand-safety grades. Information about individuals may be transiently processed during classification but is not reflected in storage or output.

Making that paragraph true — across logs, Kafka retention, debug dumps, training corpora, and backups — is the remaining engineering work. In practice it's where these systems most often fail their own audit.

If we'd started from the compliance boundary and designed inward, we'd have arrived at the same architecture in a fraction of the time. That's the recommendation: draw the red list first. It's not a constraint on the design. It is the design.

This reflects a specific design process and is not legal advice. Rights positions, biometric statutes, and AI regulation vary considerably by jurisdiction and change quickly — validate against current counsel for your own footprint.