The Two Clocks of Ad Server Architecture

Every ad platform I have seen starts life as the same diagram. Someone opens a whiteboard and writes seven boxes in a row:

Client → Preprocessing → Data Ingestion → Orchestration → Worker → Governance & Validation → Persistence

It is a good diagram. The layers are real, the separation of concerns is honest, and every box earns its place. You can hand it to a new engineer and they will understand the shape of the system in about ninety seconds.

Then you try to serve an ad in thirty milliseconds, and it falls apart.

The problem is not that any layer is wrong. The problem is the arrows. Read literally, that diagram says an ad request must pass through Kafka before the auction runs, and must clear an async validation stage before a response goes back to the client. Both of those are fatal. Kafka's produce-and-consume round trip alone will eat your entire latency budget, and a governance stage that reconciles spend and re-scores invalid traffic is doing work that takes minutes, not milliseconds.

So the first design decision is not about technology. It is about reading your own diagram correctly.

The system has two clocks

An ad serving system runs on two clocks at once, and they differ by four orders of magnitude.

The serve clock ticks in milliseconds. A user opened a surface, there is a slot to fill, and you have a hard deadline before the slot either renders empty or renders late. Everything on this clock is synchronous, in-process where possible, and single-hop where not.

The learn clock ticks in seconds to hours. Impressions get counted, segments get recomputed, models get retrained, budgets get reconciled, invalid traffic gets clawed back. Nothing here is urgent in the millisecond sense, and almost all of it is batch or stream processing.

The seven layers are correct. What is wrong is assuming a single traversal through them. The system traverses the same layers twice, on two different clocks, and the two paths meet in exactly one place: the hot store.

Ingestion is the one layer that cannot sit inline. On the serve path you tee to Kafka and never wait for the ack. Everything else stays a genuine sequential stage, which means the original diagram was closer to right than it looked. It just needed to be drawn twice.

The two stores in the middle are where the clocks meet, and they are the only place they do. The fast path reads them and never writes durably. The slow path writes them and never blocks. Every design decision in the rest of this post is downstream of that arrangement.

Retrieval = Candidate Funnel (aka 후보추출단계). Frequency Cap is a part of Retrieval.

Inverted Index = Targeting Condition

Pacing Policy = Compaign/Budget-based Control

IVT = InValid Traffic

DSAR = Data Subject Access Right covering Data Privacy Rights like European Union's GDPR and California's CCPA/CPRA.

Functional vs. Non-Functional in the Context of Ad Architecture

1. Functional Requirements (FRs)

• Preprocessing: Normalize request identifiers, extract consent context.

• Candidate Retrieval & Ranking: Perform inverted-index search, score candidates with linear models, and rank via deep learning.

• Auction & Pacing: Run a generalized second-price or first-price auction subject to campaign budget caps and pacing constraints.

• Data Logging: Asynchronously tee request, impression, and click events to event streams (Kafka).

2. Non-Functional Requirements (NFRs)

• Latency SLA: Serve path target of p99≤30 end-to-end latency.

• Zero Network Hops: Execute ranking models and pacing policies directly from in-process memory rather than making external RPC calls.

• Decoupled Clocks: Isolate synchronous fast-path execution (milliseconds) from asynchronous continuous learning (seconds to hours).

• Data Freshness / Staleness Bounds: Materialize user record updates within seconds and refresh configuration cache every 30–60 seconds.

What "real-time bidding" means when you own both sides

If your system is a closed loop — you own the surface, the users, and the campaigns, and nothing crosses an organizational boundary — then real-time bidding stops meaning what it means in the open programmatic world. There is no fan-out to external bidders and no waiting on the slowest partner. Nobody sends you a bid. You compute every bid yourself.

This deletes the single largest line item in a typical latency budget. In an open exchange, waiting on external bidders is sixty to seventy of your hundred milliseconds. Remove it and a realistic budget looks like this:

Because you also control the caller, you get to set that deadline yourself. Resist the temptation to let it drift toward a hundred milliseconds simply because nothing forces you to keep it low. Ad latency shows up directly in surface engagement, and once you have spent that time you never get it back.

The auction itself becomes a ranking problem. For each eligible campaign you compute an expected value:

eCPM = bid_price × p(action) × pacing_multiplier × quality_score

Rank by eCPM, then apply slot-fill rules for competitive separation, creative dedup, and ad-load ceilings. For the pricing rule, first-price is simplest to reason about but invites bid shading and unstable prices. A generalized second-price rule, where the winner pays the minimum bid that would have held their position, costs more complexity but stops campaign owners from constantly re-tuning bids against your own system. Keep it pluggable per surface either way.

There is one thing a closed-loop system can do that an open exchange structurally cannot: blend a non-revenue term into the ranking. User experience, organic content balance, long-term retention. Make room for that term in the scoring function on day one, even if its weight starts at zero. Retrofitting it later means renegotiating every campaign owner's expectations at once.

The candidate funnel is the real Worker layer

With no external bidders, retrieval and ranking become the central engineering problem rather than a supporting one. The standard three-stage funnel, all in-process:

One rule matters more than the rest: retrieval applies the frequency-cap and budget filters before ranking, never after. Filtering afterward means you burned CPU scoring candidates that were never eligible, and worse, you can end up with an empty slot after the filter runs.

The models are trained on the learn clock and shipped as versioned artifacts loaded into the ad server process. The serve path never calls a model service over the network. That is another hop, and you do not have the budget for it.

Frequency capping, and the single-record trick

The naive implementation reads one counter per candidate campaign. At a thousand candidates that is a thousand round trips, which is obviously untenable, so people reach for batching. Batching helps but it is treating the symptom.

The better move is to co-locate everything keyed by user into a single record, so one read serves capping, segmentation, and consent simultaneously:

Key: u:{user_id}
Bins:
  seg      → roaring bitmap of segment IDs
  fc       → map<campaign_id, packed_counter>   // [h1|h6|d1|lifetime]
  fc_ts    → last-write epoch, for bucket rotation
  consent  → personalization purpose flags
  ctx      → recency and session features

Packing per-bucket counts into a fixed-width integer gives you a sliding window without storing timestamps. Rotate the buckets on read using fc_ts. Fixed windows are cheaper but produce boundary bursts, where a user hits their daily cap at 23:59 and again two minutes later. Hourly buckets are usually the right compromise.

Three things get materially easier in a closed loop.

Identity is solved. Logged-in user IDs are durable and cross-device. There is no cookie churn, no ATT opt-out cliff, no probabilistic household matching. Caps actually mean what they claim to mean, and lifetime caps become enforceable rather than approximate. In the open web you need Bloom filters and count-min sketches to hold lifetime state at population scale; here you can just store the number.

The impression signal is fast and trustworthy. Your own client confirms the render. The window in which a cap can be breached shrinks from seconds to a few hundred milliseconds. Sticky routing by user ID plus an in-process pre-increment is enough to close it. Over-delivery mostly stops being a real category of problem.

You can cap on richer keys. Not only campaign, but creative, ad group, category, and surface. Ad-load capping, meaning a maximum number of ads per session, becomes just another counter in the same record. That is a genuine user-experience lever and it lives naturally in the layer that was already doing this work.

Segmentation, in two speeds

Two classes of segment, with different freshness SLAs, worth keeping physically separate on the write side and identical on the read side.

Batch segments come from Spark over the lake: behavioral cohorts, model-scored audiences, lifecycle stage. Freshness in hours. This is most of your targeting volume.

Streaming segments come from Flink over the event topics with keyed state: in-session intent, recency, sequence signals. Freshness in seconds.

Both write into the same roaring bitmap, so the serve path cannot tell them apart and does not need to.

Matching at serve time runs backward from how people usually build it. Do not iterate over campaigns testing each one's targeting rules. Build an inverted index in memory on every ad server node, mapping segment ID to the set of campaign IDs that want it. Campaign metadata is small enough to hold resident even at hundreds of thousands of campaigns, refreshed from Postgres every thirty to sixty seconds by version-stamped delta pull. Candidate selection then becomes a bitmap intersection over the user's segment set: microseconds, no network. Boolean targeting expressions like (A OR B) AND NOT C compile directly to bitmap operations on the same structure.

Cold start is a much smaller branch here than on the open web, because you have first-party history on most users. It still needs to exist for new accounts and logged-out sessions, so keep a contextual fallback using surface, placement, geo, and daypart as a first-class path rather than an error case.

One rule to enforce before anyone builds targeting on top of the system: minimum segment size. Small segments are the mechanism by which a targeting system accidentally becomes a system for singling out individuals. That floor is nearly impossible to introduce once campaign owners have built campaigns that depend on tiny audiences.

Governance belongs in two places

The original diagram puts Governance after Worker and before Persistence. On the learn clock that is exactly right: you validate computed data before it lands. On the serve clock it cannot be right, because consent scope and creative approval cannot be checked after the impression has already rendered.

So governance has one policy source and two enforcement points.

The inline half is itself split in two, for the reason given earlier: anything that determines eligibility has to run at retrieval, before ranking spends CPU on candidates that were never going to serve. Campaign paused, budget exhausted, creative not approved, and consent scope for personalization are all eligibility checks, and they belong in the retrieval filter alongside frequency caps. What remains for the render gate after the auction is the small set of checks that depend on the winner being known: surface content policy for the specific creative, competitive separation against the other slots on the page, and final creative integrity. That is roughly two milliseconds of work.

The async stage does spend reconciliation against served impressions, invalid traffic scoring, ad quality review queues, and user deletion requests propagating through the hot store and the lake.

It also does the thing most teams skip and later regret: decision logging. Record which campaign won which impression and why — the candidate set, the scores, the filters that fired. When someone asks why a particular user saw a particular ad, you want a real answer, and reconstructing it after the fact ranges from expensive to impossible.

The risk in splitting enforcement is policy drift between the two sides. Mitigate it by making the inline gate a compiled artifact: the async governance layer publishes a versioned policy bundle, ad servers load it, and neither side implements rules independently.

Storage, and the cadences between the stores

The first diagram compressed persistence into two boxes to keep the two-clock idea legible. That compression hides something worth seeing, so here is the full picture.

The property worth noticing is that the serve path touches only the top two rows. Postgres, ClickHouse, and the lake never appear on the fast path at all. They hydrate in-process memory on background refresh cycles, and that indirection is most of what makes a 30ms budget achievable. A design where the auction queries Postgres for campaign state, or ClickHouse for remaining budget, will work fine in staging and fall over the first time either one has a slow afternoon.

It also means there are not one but four feedback loops closing back onto the fast path, each on its own clock. User records materialize in seconds. Pacing allocation rebalances every ten to thirty seconds off actual spend in the OLAP store. Campaign config pulls every thirty to sixty seconds from Postgres. Models and policy bundles ship on publish, measured in hours. Every one of these is a place where the serve path is running on information that is slightly stale, and the staleness window is a real parameter you should choose deliberately rather than inherit from whatever the refresh job happened to be set to.

ClickHouse deserves specific mention because it is easy to file mentally as a reporting database and therefore as something that does not matter operationally. It is not. It is a read dependency of pacing, which means it is a read dependency of what the auction is allowed to serve. If OLAP ingestion falls behind, pacing allocates against stale spend numbers and campaigns over-deliver. Monitor its ingestion lag as a serving metric, not an analytics one.

The trade-offs nobody puts on the diagram

Ranking quality is entirely your problem now. In an open exchange, external bidders perform price discovery on your behalf. In a closed loop, if your action-prediction model is bad, revenue is bad, and no market corrects it. Budget real engineering effort for the model, and instrument counterfactual evaluation with a random-serve holdout from the beginning. Without it you will never be able to tell whether a model change actually helped.

The revenue-versus-experience weight is a permanent political question. Because you own both sides, every ranking change trades surface quality against revenue. Decide who owns that weight and how changes to it get reviewed before the system ships. Afterward it becomes a recurring argument with no designated referee.

Two of your constraints fight each other. Holding campaign metadata in process caps you at roughly a million campaigns per node. Past that you shard campaigns across node groups, which breaks the sticky-routing-by-user assumption that made frequency capping cheap. Know which one you will sacrifice first, because you will eventually have to choose.

Pacing leaks at window boundaries. Per-node budget allocation with periodic rebalancing avoids a shared-state bottleneck on the serve path, but it over-delivers a little at the edges. A hard global kill switch checked at coarser granularity catches the tail.

Conclusion

The seven-layer diagram was never wrong. It was underspecified in one particular way, and that single ambiguity is what makes these systems hard to build from a whiteboard sketch.

Layers describe responsibility. Arrows describe traversal. A layered diagram quietly implies there is only one traversal, and in ad serving there are two: a synchronous path measured in milliseconds and an asynchronous path measured in hours. They pass through the same seven boxes in the same order and meet only in the persistence layer, where the slow path writes what the fast path reads.

That meeting point is not a single moment. It is four refresh cycles running at four different speeds, which is another way of saying the fast path is always serving on information that is slightly out of date. The engineering question is never how to eliminate that staleness. It is how much of it each loop is allowed to accumulate.

Once you internalize that, most of the specific design decisions stop feeling like clever tricks and start feeling forced. Frequency caps and segments live in one record because the fast path can only afford one read. The inverted index is resident in memory because the fast path cannot afford a network hop. Governance splits in two because consent must be checked before rendering while reconciliation can only happen after. Ingestion is a tee rather than a stage because the fast path cannot wait on a broker.

None of that is visible in the boxes. All of it is visible the moment you draw the arrows twice.

If you are building one of these, my suggestion is to start there. Draw the two paths explicitly, put a millisecond budget on every stage of the fast one, and make the meeting point between them an intentional design object rather than an accident. Almost everything else follows.