Ad Tech Idempotency: Engineering for Reliability and Scale

Idempotency is the property that applying the same operation multiple times produces the same result as applying it once. In ad tech, this is critical for handling duplicate bid requests, billing signals, and API calls without side effects. A network timeout or SDK retry may cause the same bid request to arrive twice; an idempotent system must return the same bid (same creative, same price) both times, and charge the advertiser only once.

The fundamental challenge is that most systems are not naturally idempotent. A naive implementation might accept two identical bid requests and return two different bids (due to randomness in audience targeting), or charge twice for a single impression. At scale (100M+ requests per day), network failures and retries are near-certain; without idempotency, these compound into billing errors and lost revenue.

Idempotency requires three components:

(1) a unique request ID that the client generates and sends with every request,

(2) a store that remembers previous results keyed by request ID,

(3) logic to check if the request ID exists, return cached result, or process and cache new result.

The challenge is that this store must be extremely fast (adding <5ms latency), highly available (failures must not break the entire system), and eventually consistent (not all duplicates must be caught in real time, but none should escape billing).

Unlike exactly-once delivery (which focuses on the entire pipeline), idempotency is about individual operation safety. A bid request is idempotent if the same requestId always yields the same result. This is especially critical for expensive operations like real-time auctions (which involve external calls to demand partners, creative rendering, etc.) and financial transactions (billing, revenue attribution).

Key Requirements

Functional Requirements

• Unique request IDs: every bid request, billing event, or API call carries a client-generated UUID or nonce
• Result caching: the system remembers results of previous executions of the same request ID
• Duplicate detection: fast comparison to detect when a requestId has been seen before
• Consistent responses: identical request ID always returns identical response (same bid, same price, same creative)
• Timeout tolerance: if a client doesn't receive a response, retrying with same requestId should not change the outcome
• Cross-partition support: request IDs must be recognizable across all server instances (no local-only state)

Non-Functional Requirements

• Latency overhead: <5ms added to per-request time (median should be <1ms for cached responses)
• Throughput: support 50M–100M requests/day (579–1157 QPS) with burst to 10x peak
• Cache hit ratio: >95% for legitimate retries (duplicates within 24 hours)
• False positive rate: <0.01% (must not mistake different requests as duplicates)
• TTL: idempotency key expires after 24 hours (storage cost vs. compliance risk)
• Availability: 99.99% of requests must proceed even if idempotency store is slow or partially down
• Cost: <$0.001 per request for idempotency infrastructure

Simplified Architecture

Architecture Proposals

Option 1: Idempotency Key in ElastiCache Redis with TTL

The simplest approach: store (requestId → response_hash) pairs in ElastiCache for Redis with a 24-hour TTL. The flow is:

1. Client sends requestId with every bid request (e.g., bid_request_id: "550e8400-e29b-41d4-a716-446655440000")
2. Bid server (in ECS Fargate behind ALB) checks ElastiCache: GET requestId
3. If hit: return cached response immediately (sub-1ms)
4. If miss: execute bid logic (auction, creative selection, pricing), hash the response, SET requestId → response_hash with EX 86400 (24h)
5. Return response

The response is either the full response object (small, <1KB) or just a hash for dedup verification.

Pros

• Ultra-simple to implement: ElastiCache SETNX and GET are single operations
• Minimal latency: cache hit is <1ms; miss adds only auction latency (which exists anyway)
• Horizontal scalability: ElastiCache Cluster spreads load; add shards as needed
• Operational simplicity: just ElastiCache; no databases, no complex state machines
• Cost-effective: ElastiCache is ~10x cheaper than database transactions per operation
• Easy to debug: keys are human-readable (requestId); inspect with redis-cli

Cons

• Two sources of truth: ElastiCache + application state. If ElastiCache is lost, idempotency guarantee is lost (subsequent retries are non-idempotent).
• ElastiCache failure modes: (a) cache miss → duplicate processing, (b) slow ElastiCache → retries exceed timeout, causing cascading failures. Need HA ElastiCache (Cluster Mode).
• TTL tuning: if 24h is too short, old retries become non-idempotent; if too long, memory bloats. Requires monitoring and tuning.
• Response size: storing full response in ElastiCache can be expensive if responses are large (>10KB). May need compression or summary hashing.
• Clock skew: if requestId has timestamp component (e.g., snowflake ID), clock skew on client can break dedup.
• Cascade risk: if ElastiCache is slow, all requests slow down (no fallback). Need circuit breaker.

Failure Modes & Mitigations

• ElastiCache primary failure, replica didn't sync: Request is processed twice (once on old replica, once on new primary). Risk: duplicate bid, double billing. Mitigation: use ElastiCache Cluster Mode with synchronous replication; accept 1–2ms additional latency.
• ElastiCache timeout (>100ms): Client retries; bid server retries auction. Risk: double auction, wrong creative. Mitigation: set ElastiCache timeout to 50ms; if timeout, treat as cache miss (process again but log for manual review).
• Cache poisoning: if response is corrupted/wrong, wrong response is returned for all retries. Risk: serving wrong creative or price. Mitigation: validate response hash before returning; reject if hash mismatches.

When to Choose Choose this if you have a strong ElastiCache operations team and can live with occasional idempotency violations (due to cache loss). Ideal for 10M–50M requests/day where ElastiCache TTL overhead is acceptable. Works well if responses are small (<1KB) and clients are well-behaved (not generating duplicate IDs randomly).

Option 2: Database-Backed Idempotency with Unique Constraint (Amazon DynamoDB or Aurora)

Store idempotency keys in a primary data store (Amazon DynamoDB or Aurora PostgreSQL) with a unique constraint on requestId. The flow is:

1. Client sends requestId + request payload
2. Bid server attempts: INSERT INTO idempotency_log (request_id, payload, response, created_at) VALUES (...)
3. If INSERT succeeds: execute auction, update response column, commit
4. If INSERT fails (unique constraint violated): SELECT * WHERE request_id = ... and return cached response

This uses the database as the source of truth; no external cache needed.

This approach uses transactional semantics: either the INSERT succeeds (row is created, then updated with response) or it fails (row exists, response is returned).

Pros

• Database is single source of truth; idempotency is guaranteed even after failures
• Strong consistency: ACID transactions mean no races or double-processing
• Audit trail: full request + response is logged; useful for compliance and debugging
• Works across data center boundaries: Aurora Global Database allows replicas in different regions; reads are eventually consistent
• No additional infrastructure: if you have Aurora for core app, idempotency is "free"
• Scalable: if requests are sharded by advertiser_id, each shard has its own idempotency table

Cons

• Higher latency: INSERT + potential UPDATE is ~10–20ms (not <1ms like ElastiCache)
• Database becomes bottleneck: all requests funneled through one Aurora cluster. Max write throughput ~50k–100k QPS per cluster.
• Schema migration risk: adding idempotency table requires downtime or complex rolling deploy.
• Cost: Aurora transactions are expensive; SSD storage for 30-day retention of 100M requests/day = ~300GB, plus index overhead.
• Replication lag: if primary fails and replica hasn't caught up, some recent idempotency keys are lost.
• Constraint violation handling is tricky: application must catch INSERT failure, then SELECT (potential race if concurrent deletes).

Failure Modes & Mitigations

• Primary fails before UPDATE completes: Row is inserted (requestId is locked), but response is NULL. Risk: next retry sees NULL response. Mitigation: in application, treat NULL response as cache miss; re-execute and try UPDATE again (same INSERT will fail, next SELECT returns non-NULL).
• Aurora replica lag on failover: new primary doesn't have recent requests. Risk: duplicate execution. Mitigation: use Aurora cluster with synchronous replication (aurora_parallel_query=ON); slower but ensures standby has data.
• Concurrent requests with same ID: both INSERT simultaneously. Risk: one succeeds, other fails; but both attempt auction. Mitigation: use DynamoDB conditional PutItem (attribute_not_exists) or application-level locking.

When to Choose Choose this if you already have a mission-critical Aurora or DynamoDB infrastructure and your requestId volume is modest (10M–30M/day). Good for use cases requiring strong auditability (compliance, billing disputes, fraud investigation). Avoid if you need <10ms latency or if your DBA team prefers eventual consistency.

Option 3: Event-Sourced Idempotency (Deterministic Request Projection via Kafka Streams)

Rather than explicitly storing idempotency keys, use event sourcing: the entire request (including requestId) is treated as an immutable event on Amazon MSK. All operations (auction, creative selection, pricing) are purely deterministic functions of the request. Thus, executing the same request twice with same seed always produces same result.

The flow is:

1. Client sends requestId + request payload
2. Bid server writes event to MSK: {requestId, payload, timestamp}
3. Kafka Streams processor (running in ECS Fargate) consumes event and deterministically projects to bid output
4. If same event arrives (same requestId), stream processor recognizes it as duplicate (dedup by requestId), returns same projection
5. Bid output (creative, price) is stored in DynamoDB, keyed by requestId

This is idempotent by construction: the function f(requestId, payload) → bid is pure and deterministic. No cache needed; idempotency is implicit.

The key insight: idempotency comes from determinism. If the auction logic is free of randomness (all decisions are derived from request attributes, feature vectors, etc.), the same request always produces the same bid.

Pros

• Idempotency by design: no explicit cache or storage; inherent in function determinism
• Horizontally scalable: each Kafka Streams instance processes independently
• Full auditability: all events are immutable; can replay to recompute any historical bid
• Fault-tolerant: if Streams instance crashes, another instance picks up; same event produces same result
• Composable: multiple systems can consume bid-results topic; all get consistent output
• No external dependencies: MSK + DynamoDB + application logic; no ElastiCache needed

Cons

• Requires deterministic auction logic: no randomization, no time-dependent decisions (no TTL-based targeting, no "show this creative for next 1 hour")
• Latency: events flow through MSK (~100ms) before result is materialized. Not suitable for real-time response to client.
• Complexity: requires significant refactoring of auction logic; all external calls must be eliminated or cached/precomputed
• State management: if stream processor needs state (e.g., user profile), that state must be versioned and immutable
• Client communication: client doesn't get response until event is processed; adds latency and requires callback mechanism
• Testing burden: must validate determinism across all inputs; non-determinism bugs can be subtle

Failure Modes & Mitigations

• Non-deterministic logic sneaks in: e.g., random creative selection, or time-dependent logic (current time). Risk: same requestId produces different bids. Mitigation: strict code review; unit tests that verify f(x) = f(x) for all x; use deterministic ID generation instead of random.
• State divergence: Streams instance crashes; another instance resumes from different state. Risk: different projection. Mitigation: all state must be sourced from MSK; no local state. Use Kafka Streams state stores with changelog topics for recovery.
• Event loss in MSK: event is processed, but MSK partition is lost before replication. Risk: bid result is lost. Mitigation: use 3-zone replication; enable log compaction on bid-results topic (MSK retains latest value per requestId forever).

When to Choose Choose this if your organization can commit to deterministic auction logic and can afford the refactoring effort. Ideal for high-volume campaigns (1B+ requests/day) where replay-ability and auditability are critical. Works well if you already use Kafka Streams for other pipelines. Avoid if your auctions require real-time external data (live user segments, dynamic creative selection based on user session).

Comparison Table

Recommendation

For most ad tech platforms, Option 1 (ElastiCache Redis Idempotency) is the best starting point. It is simple, fast, and handles 99% of idempotency needs. The trade-off is accepting occasional violations (if ElastiCache cluster is lost), which is acceptable for retries within 24 hours.

Use Option 2 (Database-Backed) if:

• You have strong regulatory/compliance requirements (financial services, healthcare ads)
• Your request volume is modest (< 10M/day) and latency budget is loose (>50ms)
• Your Aurora infrastructure is already mission-critical and heavily monitored
• You need full audit trails for billing disputes

Use Option 3 (Event-Sourced) if:

• Your auction logic is deterministic and you can refactor to eliminate randomness
• You need to replay historical auctions for debugging or regulatory inquiries
• Your organization is deeply invested in event streaming and can afford the complexity
• You have 500M+ requests/day and need extreme horizontal scalability

Implementation guide for Option 1 (recommended):

1. Generate requestId on client side (UUID v4 or snowflake).
2. Use ElastiCache Cluster Mode with 16 shards, each with HA (automatic failover).
3. Store (requestId → response_json) with EX 86400 (24h).
4. On cache miss: execute auction, serialize response, SET in ElastiCache.
5. Monitor ElastiCache memory usage via CloudWatch; alert if TTL eviction exceeds 5% of keys.
6. Weekly: reconcile bid server logs vs. ElastiCache cache hits (for debugging).

Option1+Option2 Combination

Capacity & Performance Notes

Sizing for 50M requests/day (Option 1):

• QPS: 50M / 86400s ≈ 579 QPS average; 5,790 QPS at 10x peak.
• ElastiCache storage: assume 500 bytes per response (request metadata + bid details). 50M requests * 500B = 25 GB per day. With 7-day retention before TTL: ~175 GB. Use ElastiCache Cluster Mode with 16 shards, each 16 GB (cache.r6g.xlarge), total 256 GB capacity.
• Cache hit ratio: ~95% (first request misses; subsequent retries within 24h hit).
• Latency breakdown: cache hit ~0.5ms + network RTT 10ms + serialization 0.5ms = 11ms p50, 20ms p99. Cache miss adds ~200ms for auction execution.

Option 2 sizing (50M requests/day):

• Table size: 50M requests/day * 30 days = 1.5B rows. With columns (request_id, payload, response, timestamp), each ~1KB. Total: 1.5 TB on disk.
• Index on request_id (unique): adds 50% overhead (750 GB).
• Aurora cluster: primary on r6g.4xlarge (16 core, 128 GB RAM, 1 TB NVMe SSD); 2 replicas for HA.
• Write latency: 15–20ms per INSERT + response UPDATE.
• Retention policy: delete rows older than 30 days (nightly batch); keep recent for disputes.

Option 3 sizing (50M requests/day):

• MSK topic (bid-events): 50M events/day, each ~500B = 25 GB/day. Retention 30 days = 750 GB. Use 8 partitions for parallelism.
• Kafka Streams app: 4 ECS Fargate tasks, each with 2 vCPU 4GB RAM, handling ~145 QPS.
• Latency: event → published 100ms (MSK queue) + processing time 50ms + materialization to DynamoDB 100ms ≈ 250ms total.