AWS Ad Exchange Disaster Recovery: Active-Active, Active-Passive, and Cell Strategies - Disaster Recovery

An ad exchange spanning multiple AWS regions (us-east-1, eu-west-1, ap-southeast-1) is resilient against server failures but vulnerable to regional outages: a data center power failure, network cut, or AWS regional incident affecting an entire zone can knock offline all ECS Fargate tasks, Aurora databases, and load balancers in that region. A typical region generates 30–40% of global revenue; a 1-hour outage costs millions in lost impressions and broken advertiser relationships.

Disaster recovery (DR) strategies ensure that when a region fails, traffic is rerouted to healthy regions within seconds, and critical state (campaigns, pacing, user data) is preserved via database replication. Three approaches scale DR: Active-Passive with warm standby using Aurora Global Database (complex, RTO of minutes), Active-Active multi-region with Route 53 automatic failover (operationally simple, stateless design required), and Cell-based architecture (sub-regional isolation, shed capacity gracefully without losing the entire region).

Key Requirements

Functional Requirements

• Automatic detection of regional failure: Detect when a region is down (all ECS Fargate, Aurora, or ALB offline) within 5–10 seconds, without human intervention.
• Traffic reroute to healthy region: Within 15 seconds of failure, all traffic is rerouted to healthy regions; users experience timeout but requests eventually reach a working ad server.
• Session state preservation: If a user had an in-flight auction or session state, that state is preserved via DynamoDB Global Tables or ElastiCache Global Datastore; users don't see divergent ads or pricing.
• Graceful fallback: If all regions are degraded (not fully down), accept reduced capacity rather than failing completely.
• Failback on recovery: When failed region recovers, automatically migrate traffic back within 5 minutes.

Non-Functional Requirements

• Recovery Time Objective (RTO): Return to normal capacity within 1 hour (industry standard: 4 hours, but ad tech demands faster).
• Recovery Point Objective (RPO): Lose < 5 minutes of state (acceptable for ad tech; some impressions may not be charged).
• Failover latency: Failover traffic reroute completes within 15 seconds; Route 53 health checks + DNS propagation account for the delay.
• Cost efficiency: DR standby infrastructure costs < 30% of normal operating costs (avoid paying for 2x capacity).
• Operational ease: Failover and failback should be automatic; operators should not be paged to manually trigger switches.

Simplified Architecture

Architecture Proposals

Option 1: Active-Passive with Warm Standby (Aurora Global Database)

Description

One region runs active (serving all traffic): us-east-1. Other regions (eu-west-1, ap-southeast-1, us-west-2) run passive standby: they have ECS Fargate tasks ready, Aurora databases warm with replicated data from the primary, but receive no production traffic. All databases are asynchronously replicated from the active region via Aurora Global Database (RTO typically 5 minutes for replication lag to become critical).

On regional failure: Route 53 health check detects us-east-1 is down (via health check endpoint that's multi-layered: ALB, ECS tasks, Aurora database). Route 53 fails over to eu-west-1 (now active). EU Aurora database becomes writable, catches up on replication lag (< 5 minutes), then starts accepting production traffic. ECS Fargate tasks in EU region (previously dormant) start receiving traffic.

Pros

• Simple failover logic: Route 53 health check is a single point of control; failover is just a DNS switch + Aurora promotion.
• Cost-controlled: Passive regions are kept warm but don't burn compute; only storage/replication cost (10–15% of active cost).
• RPO moderate: Replication lag is 5 minutes (configurable); lose last 5 minutes of state on failover.
• Proven technology: Aurora Global Database and Route 53 failover are well-understood; many enterprises use this pattern.

Cons

• RTO 5–15 minutes: Passive region must promote Aurora (convert read replica to writable), catch up on replication lag, and warm up ECS tasks before accepting full traffic at full capacity. Impressions are served with stale campaign data until catch-up.
• Cascading failure on failover: If EU region (the standby) then fails while handling traffic (before replication catches up), no tertiary fallback exists; all traffic is lost.
• Data consistency risk: If active region (us-east-1) fails partway through a database transaction, passive region has incomplete state; must manually reconcile or accept data loss.
• Failback complexity: When us-east-1 recovers, failing back (making it active again) is risky; must resync Aurora databases, re-promote to primary, clear old transactions, validate consistency before switching traffic back.

Failure Modes & Mitigations

• Replication lag accumulates: If EU Aurora falls behind, failover loses more state. Mitigate by monitoring replication lag continuously via CloudWatch; if lag > 60 sec, alert and trigger manual inspection.
• Health check false positive: If Route 53 health check flakes (temporary network issue), failover is triggered unnecessarily. Mitigate by using multi-layered health checks (HTTP + TCP + database query); require 2 of 3 checks to fail before failover.
• Cascading failure on failover: EU is under-provisioned (was a warm standby); suddenly receiving 100% global traffic causes overload. ECS tasks CPU/memory saturate, Aurora read replicas promoted too slowly. Mitigate by provisioning passive regions to handle 60–70% global traffic (warm but not scaled).

When to Choose

Choose this option if you have time to accept 5–15 minute RTO, your team can manage Aurora Global Database replication, and you want to save operational complexity compared to Active-Active. Typical in smaller ad exchanges or those prioritizing cost over latency.

Option 2: Active-Active Multi-Region (Route 53 Latency-Based Routing + DynamoDB Global Tables)

Description

All regions (us-east-1, eu-west-1, ap-southeast-1) run active simultaneously, each handling traffic routed to them by Route 53 latency-based routing (returns closest region's load balancer IP). If a region fails, Route 53 detects it via health check and reroutes to the next-closest region. All regions have full campaign data (read-only Aurora replicas synced every 1 minute via AWS Glue / Lambda), session state is distributed (no single point of failure).

Ad servers are stateless; session state lives in DynamoDB Global Tables (eventual consistency, all regions writable) and/or ElastiCache Global Datastore (Redis with multi-region replication). Auctions are short-lived; if a region fails mid-auction, the request is retried upstream (subscriber infrastructure) and lands in a healthy region.

Sequence for region failure and reroute

Pros

• Instant failover: No passive standby warming up; traffic switches to nearest healthy region in seconds (Route 53 health check cycle + DNS TTL).
• Automatic and transparent: No manual failover trigger; Route 53 handles everything automatically.
• True high availability: All regions active; capacity loss is proportional to failure (if us-east-1 fails, global capacity drops by ~35%).
• Cost-efficient: No standby regions burning money; pay for what you use.
• Latency-optimized: Users always route to nearest region (latency-based routing); best user-facing latency.

Cons

• Stateless design required: All servers must be stateless; state lives in DynamoDB (5–10 ms latency per round-trip) and ElastiCache Global Datastore. This is 5–10 ms slower than local Redis.
• Eventual consistency: DynamoDB uses eventual consistency by default; if us-east-1 writes a pacing counter and immediately fails, eu-west-1 might see stale value. Advertisers might over-spend slightly.
• Campaign data sync complexity: Every region needs synchronized campaign data via read-only Aurora replicas or Glue Jobs. If central campaign DB is in us-east-1 and fails, eu-west-1 has stale rules (until local cache expires).
• Operator complexity: All regions run live; harder to test changes safely (must deploy to all regions or use canary).

Failure Modes & Mitigations

• Route 53 health check false negatives: Region is healthy but health check fails due to temporary network glitch. ECS tasks are running but health endpoint is slow. Mitigate by multi-layered health checks (ALB + ECS task health) and requiring multiple failures before failover.
• DynamoDB global table replication lag: If us-east-1 writes pacing counter and fails immediately, eu-west-1 sees stale data. Mitigate by using strongly consistent reads for critical data (at cost of latency) or accepting slight overspend.
• Cascading failure: If two regions fail simultaneously (us-east-1 and eu-west-1 down, only ap-southeast-1 left), capacity drops to 25% of normal. Mitigate by provisioning each region for 70% global traffic (so one region failure = 30% capacity drop, acceptable).

When to Choose

Choose this option if you have global users and want the best latency + availability trade-off. Standard for large modern ad exchanges. Requires investment in stateless architecture (no local caches, distributed session store via DynamoDB / ElastiCache) but pays dividends in operational simplicity and performance.

Option 3: Cell-Based Architecture

Description

Instead of regional isolation, partition the ad server fleet into cells: independent slices of compute, cache, and database that can fail independently. Each cell handles a subset of global traffic (by advertiser region, customer tier, or geography). If one cell fails, that subset of traffic is shed, but other cells continue serving.

For example: Cell A (US advertiser customers), Cell B (EU advertisers), Cell C (APAC advertisers). Each cell has its own ALB, ECS Fargate cluster, Aurora database (or DynamoDB table), and ElastiCache. Route 53 routes based on customer region (not geography); if Cell A fails, customers in Cell A see no ads, but customers in Cell B and C continue normally.

Cells can be colocated in the same AWS region or spread across regions; the key is isolation at the logical level, not physical.

Pros

• Granular failure isolation: Cell failure only affects that cell's traffic; other cells unaffected. Capacity loss is limited to that cell's size (e.g., 10% if 10 equal cells).
• Simple recovery: Each cell is independent; recover a cell without coordinating with others. AWS Backup can restore cell's Aurora database independently.
• Scalable: Add new cells for new advertiser regions or traffic types; each cell is a self-contained unit.
• Simpler replication: Each cell replicates within itself (Aurora multi-AZ); no need for complex multi-region replication.

Cons

• Advertising complexity: Advertisers must be pre-assigned to cells; if advertiser wants to expand to new region, cell assignment must change (requires offline migration or careful routing logic).
• Capacity skew: If one cell (e.g., US advertisers) has more demand, that cell becomes overloaded while others are idle. Load balancing across cells is limited.
• Cold start on cell failure: If a cell fails, traffic meant for it is lost (no automatic reroute unless you have cross-cell redundancy). Need fallback cells to prevent total loss.
• Operational burden: N cells to monitor, patch, and scale independently; N times the operational work.

Failure Modes & Mitigations

• Cell assignment mismatch: If routing logic assigns advertiser to wrong cell (or advertiser moves between cells without migration), traffic is lost. Mitigate by automated pre-checks: validate advertiser-to-cell mapping at Route 53 configuration time.
• Skewed capacity: One cell overloaded due to advertiser concentration. Mitigate by dynamic cell assignment: periodically rebalance advertisers across cells based on traffic patterns (via Route 53 weighted routing or Lambda authorizer).
• Cascading cell failure: If Cell A fails and fallback Cell D is enabled, but Cell D is also degraded, traffic is lost. Mitigate by maintaining 2 active copies of each cell (A and B); if A fails, traffic automatically reroutes to B via Route 53 health checks.

When to Choose

Choose this option if you have a diverse advertiser base (different regions, account sizes) and want fine-grained failure isolation. Typical in very large ad exchanges (> 10 billion impressions/day) that can afford operational complexity of N cells. Works well when combined with Option 2 (cells distributed across regions for geographic diversity).

Comparison Table

Recommendation

For most ad tech platforms on AWS, choose Option 2 (Active-Active Multi-Region with Route 53) unless your auctions are deeply stateful. Here's why:

1. Operational simplicity: All regions active; no passive standby to maintain or failback complexity.
2. Fastest failover: RTO < 30 seconds (industry-leading); Route 53 detects failure and reroutes automatically.
3. Best latency: Latency-based routing ensures users connect to nearest region; 50–100 ms improvement over single-region.
4. Cost-efficient: No standby overhead; pay only for active capacity.

Make the trade-off of stateless design: Accept DynamoDB eventual consistency and 5–10 ms overhead per request (per read/write to DynamoDB or ElastiCache Global Datastore). For ad auctions (< 100 ms total), this overhead is acceptable (5–10% of latency budget).

Use Option 1 (Active-Passive with Aurora Global Database) only if:

• Your auctions are deeply stateful (long-lived, session persistence required).
• Your team is still building multi-region infrastructure and wants to start simple.
• You can accept 5–15 minute RTO (acceptable for some less-critical ad networks).

Add Option 3 (Cell-Based) as a complementary strategy:

• Implement cells within each region for sub-regional isolation (e.g., us-east-1 has Cell-A for premium advertisers, Cell-B for long-tail).
• If Cell-A fails, Cell-B continues.
• Provides extra granularity of failure isolation without full operational burden of Option 3 as a standalone strategy.

Recommended hybrid approach:

1. Implement Active-Active multi-region (Option 2) as the primary strategy.
2. Make ad servers stateless; session state in DynamoDB Global Tables.
3. Use ElastiCache Global Datastore for high-performance cache (circuit state, user frequency caps).
4. Implement cell-based partitioning within each region for advertiser isolation (reduces operational impact if a cell fails).
5. Target RTO: 30 seconds (Route 53 failover), RPO: < 1 second (DynamoDB eventual consistency).

Capacity & Performance Notes

Assume global traffic: 400k req/s (us-east-1: 35%, eu-west-1: 30%, ap-southeast-1: 20%, other: 15%).

Option 2 (Active-Active) regional breakdown:

• us-east-1: 140k req/s, handled by 70 ECS Fargate tasks (2k req/s each, 2 vCPU, 4 GB RAM).
• eu-west-1: 120k req/s, handled by 60 ECS Fargate tasks.
• ap-southeast-1: 80k req/s, handled by 40 ECS Fargate tasks.
• Total: 170 ECS Fargate tasks running live.

On us-east-1 failure (35% traffic lost):

• Reroute 140k req/s to eu-west-1 and ap-southeast-1.
• eu-west-1 capacity increases from 120k to ~150k req/s; overload margin: 25%.
• ap-southeast-1 capacity increases from 80k to ~110k req/s; overload margin: 37%.
• Degradation expected: 5–10% increase in latency (queue depth increases as tasks near saturation).
• Provision each region to handle 70% global traffic (not just its normal share) to accommodate this reroute. Scale up ECS cluster: us-east-1: 100 tasks; eu-west-1: 100 tasks; ap-southeast-1: 70 tasks. Total: 270 tasks (60% capacity headroom).

DynamoDB Global Table latency:

• us-east-1 to eu-west-1 replication: 100 ms (transatlantic cable latency).
• eu-west-1 to ap-southeast-1 replication: 150 ms (via Middle East / Asia routing).
• Read latency (local region, strongly consistent): < 10 ms.
• Read latency (cross-region, eventual consistency): 150–200 ms.
• Write latency (local): 10 ms; consistency achieved across regions in 100–200 ms.
• For ad auctions, use eventual consistency reads (local < 10 ms); occasional stale pacing counter is acceptable.

ElastiCache Global Datastore latency:

• Read latency (local region): < 1 ms.
• Read latency (cross-region): 100–150 ms.
• Write latency (local): 1 ms; replication to other regions in 50–100 ms.

Campaign data replication:

• Central campaign DB (us-east-1) -> read replicas (eu-west-1, ap-southeast-1) every 1 minute via AWS Glue or Lambda.
• Assume 10,000 active campaigns, each ~5 KB of targeting rules/budget state. Total 50 MB per replica.
• Network bandwidth: 50 MB / 60 sec = 833 KB/s per region. Negligible on AWS inter-region backbone.
• Staleness SLO: < 1 minute; enforced by automated resync every 60 seconds.

Route 53 health check timing:

• Health check interval: 30 seconds (standard FastInterval setting).
• Health check failure detection: 3 consecutive failures = 90 seconds (can be tuned to 10 seconds for FastInterval).
• TTL on DNS answer: 60 seconds (standard for Route 53).
• Worst-case failover: 90 sec (health check detection) + 60 sec (DNS TTL expiry) = 150 seconds = 2.5 minutes.
• Optimize by: reducing health check interval to 10 seconds (FastInterval), reducing TTL to 30 seconds (faster propagation). Target: 30–45 second failover.

Failback on recovery:

• When us-east-1 recovers (health checks pass again), Route 53 automatically directs new traffic to it.
• Old connections (with cached DNS) still point to eu-west-1 / ap-southeast-1; they complete naturally as users refresh.
• Full traffic migration back to us-east-1: 2–5 minutes (DNS propagation, client cache expiry, connection reuse timeouts).

AWS Backup for DR:

• Enable AWS Backup for Aurora databases in each region; automatic daily snapshots.
• Cross-region backup copies (us-east-1 snapshots copied to eu-west-1 / ap-southeast-1) for additional durability.
• RPO via snapshots: 24 hours. For critical campaigns, supplement with continuous replication (Aurora Global Database).