Exactly-Once Delivery: The Ad Tech Blueprint for Accurate Billing

Exactly-once delivery is the guarantee that each impression or click event is recorded exactly once in the data pipeline, without duplication or loss. In ad tech, this is critical for accurate billing, revenue reconciliation, and reporting. When an advertiser pays per impression or per click, duplicate events lead to overbilling; lost events lead to underbilling and broken trust. The challenge compounds at scale: with 10–100 billion impressions per day across multiple geographic regions, the probability of network failures, consumer crashes, and producer retries becomes near certain.

The core problem is that at-least-once delivery (the industry default for high-throughput systems like Amazon MSK) can result in retried events appearing multiple times if a producer doesn't know whether its previous send succeeded. Similarly, if a consumer crashes after processing an event but before committing its position, the event is replayed. Naive deduplication downstream adds latency and storage costs. Yet full exactly-once semantics (distributed transactions across producer, broker, and sink) is expensive and slow.

Ad tech platforms handle 100s of millions of QPS (queries per second) in real time, with expected SLAs of sub-100ms for auction results and sub-second latency for impression logging. Any solution must not introduce queuing delays or synchronous blocking operations. Furthermore, events are immutable: an impression that happened cannot be "un-happened," so the system must be append-only and idempotent.

Key Requirements

Functional Requirements

• No duplicate events in billing store: each impression_id, click_id, or conversion_id is recorded exactly once
• No lost events: 99.99%+ of events from publisher SDKs must reach the data lake
• Idempotent event IDs: every event must carry a unique, immutable ID (e.g., UUID or snowflake ID)
• Audit trail: all events must be traceable from source through sink, with timestamps and lineage
• Transactional producer-consumer pairs: producer knows if its message was committed before retrying
• Support for high cardinality: 1000s of campaigns, advertisers, publishers running simultaneously

Non-Functional Requirements

• End-to-end latency: <100ms from impression to MSK (p99), <5s from MSK to data lake commit
• Throughput: support 10–100B impressions/day (115k–1.15M QPS), with burst capacity 2x average
• Durability: 99.999% of committed events must be recoverable even after data center failure
• Storage cost: <$0.01 per 1000 events in durable store (dedup index adds overhead)
• Operational complexity: on-call time <2h/week, no manual reconciliation >1x/month
• Consistency SLA: billing-approved events available in warehouse within 24h

Simplified Architecture

Architecture Proposals

Option 1: Amazon MSK Exactly-Once Semantics (EOS) with Transactional Producer & Read-Committed Consumer

This approach leverages Amazon MSK's native transactional guarantees. The producer wraps each event in a transaction: writes the impression to an MSK topic, and only commits the producer transaction after receiving acknowledgment from the broker. The consumer (or stream processor like Kafka Streams) uses isolation.level=read_committed to consume only committed messages, ensuring it sees exactly the events the producer intended.

The flow is:

1. Producer (SDKs, ECS Fargate loggers) batches events → writes to MSK with idempotent flag enabled
2. MSK brokers replicate across 3 Availability Zones (us-east-1a, us-east-1b, us-east-1c); min.insync.replicas=2 for durability
3. Kafka Streams app or consumer (in ECS Fargate) reads with read_committed isolation
4. Consumer writes to idempotent sink (e.g., S3 with dedup key, or Aurora with unique constraint)
5. On failure (e.g., network partition), producer retries with same event ID; consumer deduplicates via idempotency

The key challenge is performance: transactional writes in MSK introduce ~30–50ms latency per batch due to coordination overhead. Batching mitigates this (group 1000 events per transaction), but then latency spikes on low-traffic campaigns.

Pros

• Native guarantee from Kafka; no custom dedup logic needed at application layer
• Single source of truth: Kafka broker is the authority on what was committed
• No external dependencies beyond Kafka cluster (no Redis, no extra databases)
• Well-tested in production at large scale (LinkedIn, Uber)

Cons

• ~30–50ms overhead per batch; not suitable for sub-10ms latency requirements
• Requires coordinated shutdown of producers and consumers (kills operational agility)
• Kafka controller becomes a bottleneck if many transactional producers compete
• Rebalancing or broker failures trigger consumer lag (can spike to minutes)
• Difficult to debug: transaction state is hidden in Kafka internals

Failure Modes & Mitigations

• Producer crash mid-transaction: Kafka automatically aborts; next retry is clean. Risk: if clock skews, producer ID reuse can cause orphaned transactions. Mitigation: NTP sync + producer.id.expire.ms tuned conservatively (7 days).
• Consumer lag during producer surge: Buffered in MSK (default 7 days retention). Risk: if producer rate exceeds consumer capacity, lag grows, billing reports delay. Mitigation: over-provision consumer threads in ECS; monitor lag via CloudWatch.
• Broker failure mid-replication: If min.insync.replicas=2, one broker failure is tolerated. Risk: if 2+ brokers fail simultaneously, unacknowledged data is lost. Mitigation: use 3-zone replication; enable AWS Backup for durability.

When to Choose Choose this option if your team is already deeply invested in Kafka and can tolerate 30–50ms latency per batch. Ideal for mobile SDKs or server-side loggers where 100–200ms end-to-end latency is acceptable. Avoid if you need sub-5ms producer latency or if your team lacks Kafka operations expertise.

Option 2: At-Least-Once Delivery + Downstream Deduplication Store (ElastiCache Redis)

This is the pragmatic "good enough" approach used by most ad tech platforms. MSK is configured for at-least-once (acks=all, min.insync.replicas=2), meaning the producer retries until acknowledged, but no transactional guarantee. Events may be replayed. The deduplication happens downstream: a fast dedup store (ElastiCache for Redis) holds a set of seen event_ids with a TTL (e.g., 24 hours).

The flow is:

1. Producer sends event with unique event_id (UUID or snowflake) to MSK
2. MSK durably logs the event across 3 AZs; consumer reads (any isolation level, no coordination overhead)
3. Consumer dedup check: SETNX event_id in ElastiCache (atomic compare-and-set). If key already exists, skip; else process and set TTL to 24h
4. Consumer writes to sink (S3, Aurora) only if dedup check passed
5. After 24h, TTL expires; event_id key is forgotten (space-efficient)

This avoids MSK's transactional coordination and is extremely fast: ElastiCache SETNX is sub-1ms.

Pros

• Minimal latency: at-least-once has no producer coordination; Redis check is <1ms
• Linear scalability: add consumer instances independently; no rebalancing coordination
• Operational simplicity: just Kafka + Redis (no Kafka Streams, no complex state stores)
• Easy to debug: dedup logic is explicit and testable
• Cost-efficient: Redis cluster is cheaper than Kafka transactional overhead at scale

Cons

• Two sources of truth (Kafka + Redis); potential inconsistency if Redis is lost
• Redis failure means: (a) all dedup checks fail (write everything twice), or (b) dedup stops. Need HA Redis (cluster or Sentinel)
• Event_id collision risk: if not using true UUIDs, collisions cause false positives. Mitigation: always use 128-bit IDs or snowflake + counter
• TTL mistuning: if TTL is too short, events replayed after expiry are duplicated; if too long, Redis memory bloats
• Eventual consistency: deduplicated events may not appear in data lake for hours; billing reports lag

Failure Modes & Mitigations

• ElastiCache replica lag: Primary writes event_id; replica hasn't synced yet; primary crashes. Risk: new consumer reads from replica, sees missing key, writes duplicate. Mitigation: use ElastiCache Cluster Mode (not just Replication Group) with automatic failover; accept 1–2ms additional latency.
• Consumer crash before writing to S3: Event is deduped in Redis, but never reaches sink. Risk: event is lost. Mitigation: consumer commits to Kafka AFTER writing to S3, not before. Use a write-ahead log pattern.
• Clock skew on event_id (snowflake): Two SDKs with skewed clocks generate same snowflake ID. Risk: false dedup. Mitigation: use centralized ID service (like Twitter Snowflake) or rely on UUID v4 (collision rate ~1 in 5 billion per year).

When to Choose Choose this if you need fast, scalable, operationally simple deduplication. Works well for most ad tech platforms at scale. Requires Redis expertise and HA discipline. Good for 10B+ impressions/day with <50ms latency budget.

Option 3: Outbox Pattern with Change Data Capture (CDC) via AWS DMS or Debezium

This approach treats MSK as the immutable source of truth, but avoids MSK's transactional coordination overhead. Instead, events are written first to a durable transactional database (Amazon Aurora PostgreSQL) in an "outbox" table, then a CDC tool (AWS Database Migration Service or Debezium on MSK Connect) polls the outbox and publishes changes to MSK. The key insight is: the database transaction is local and fast; replication to MSK is asynchronous.

The flow is:

1. Producer writes event directly to Aurora: INSERT INTO outbox (event_id, payload, status, created_at) VALUES (...)
2. Aurora transaction is ACID-durable; unique constraint on event_id prevents duplicates at source
3. AWS DMS (or Debezium CDC on MSK Connect) captures changes from Aurora WAL; publishes to MSK Topic
4. Downstream consumer reads MSK (no dedup needed; source is already deduplicated)
5. Consumer writes to S3 or data lake; can also use AWS Lambda or Kinesis Data Firehose for exactly-once delivery into final store

This inverts the usual flow: database is primary, MSK is secondary.

Pros

• Database unique constraint is the dedup guarantee; no secondary Redis needed
• Producer latency is just DB insert (~5ms for Aurora with fsync); no MSK coordination overhead
• Full auditability: outbox table is an immutable log; can replay from any checkpoint
• Exactly-once delivery is enforceable end-to-end if using AWS Lambda + DynamoDB idempotency keys
• Works across multiple data centers (Aurora Global Database for cross-region replicas)

Cons

• Adds complexity: requires CDC tool (AWS DMS or Debezium) and its operational burden
• CDC lag: DMS or Debezium may lag 1–10s; events are not immediately visible in MSK. Not suitable for sub-second reporting.
• Aurora bottleneck: all events funneled through one Aurora cluster (unless sharded). Max write throughput ~100k QPS per cluster.
• Two systems to monitor: Aurora + MSK + DMS. More failure modes.
• DMS cost: AWS DMS replication instances are not cheap (~$1–2k/month for production)

Failure Modes & Mitigations

• DMS captures duplicate: if DMS crashes mid-snapshot, it must resume from WAL position; may re-emit some events. Risk: downstream duplicates. Mitigation: ensure consumer is idempotent (which it should be anyway).
• Aurora replica lag / failover: Primary fails; replica hasn't caught up. Risk: events on primary are lost. Mitigation: use Aurora Cluster (synchronous replication) ensures replica has data before ack; adds ~10ms latency per write.
• MSK and Aurora diverge: Aurora has events MSK missed (e.g., DMS was down). Risk: billing discrepancy. Mitigation: daily reconciliation job using AWS Glue that scans Aurora outbox and compares with MSK offset.

When to Choose Choose this if your team is strong in database operations (Aurora PostgreSQL) and wants to avoid MSK complexity. Ideal for smaller scale (1B–10B events/day) or if you already have a mission-critical Aurora cluster. Good if you need strong auditing and regulatory compliance (financial records). Avoid if you need sub-100ms global distribution or if your infrastructure team is MSK-native.

Comparison Table

Recommendation

For most ad tech platforms at scale, Option 2 (At-Least-Once + ElastiCache Dedup) is the best starting point. It balances latency, throughput, and operational simplicity. ElastiCache is a battle-tested technology that your team likely already operates; adding an MSK consumer layer is straightforward. The dedup logic is explicit and easy to test, unlike MSK's hidden transactional state.

If you are already running a high-scale MSK infrastructure (1B+ events/day) and your team is skilled in Kafka Streams, Option 1 can work, but budget significant on-call overhead for transactional state debugging.

If your organization prioritizes auditing and regulatory compliance, and your Aurora team is strong, Option 3 (Outbox + DMS CDC) provides the best auditability and data governance guarantees. It is particularly valuable if you must reconcile with billing systems frequently.

Implementation advice:

• Start with Option 2. Generate event_ids using UUID v4 or a snowflake service.
• Use ElastiCache Cluster Mode with automatic failover in a 16-shard configuration for HA.
• Monitor ElastiCache memory and TTL eviction rate; alert if eviction exceeds 1% of keys via CloudWatch.
• Implement daily reconciliation using AWS Glue: compare MSK message count vs. deduplicated events written to data lake.
• Log all duplicate detections to a separate MSK topic for analysis and debugging.

Capacity & Performance Notes

Sizing for 10B impressions/day:

• QPS: 10B / (86400s) ≈ 115k QPS average; 230k QPS at 2x peak.
• MSK cluster: 3 brokers (kafka.m5.large), each with 8 core, 32 GB RAM, 2 TB EBS. Retention 7 days → ~29 TB total storage.
• ElastiCache dedup store: Assume 1 KB per event_id (128-bit UUID + metadata). 10B events/day * 7 days retention (before TTL eviction) = 70 GB. Use ElastiCache Cluster Mode with 16 shards, each 8 GB, on cache.r6g.xlarge instances.
• Consumer fleet: 50 ECS Fargate tasks (2 vCPU, 4 GB RAM each) with 50 threads per task = 2500 threads total, each handling ~50 events/s.
• Disk I/O: S3 writes via Firehose at ~500 Mbps sustained; use multipart upload to tolerate 1s–5s network blips.

Latency breakdown (Option 2):

• SDK → MSK: 20ms (network) + 5ms (batch buffer) = 25ms p99
• MSK → Consumer dedup check: 10ms (poll) + 1ms (ElastiCache) = 11ms p99
• Consumer → S3: 30ms (multipart) + 5ms (ACK) = 35ms p99
• Total: ~70ms p99