Ad Tech Circuit Breakers: Architectures for Resilient Bidding Engines - Circuit Breaker

The ad exchange engine calls external DSPs, ad networks, and third-party APIs synchronously during the bidding phase. When a downstream service degrades—whether due to latency spikes, temporary outages, or cascading failures—the ad engine can exhaust thread pools, accumulate request backlogs, and ultimately fail to serve ads to any user. Circuit breaker patterns detect these failures fast and fail open by returning a default house ad, preserving availability at the cost of reduced revenue on individual requests.

In ad tech, circuit breakers are critical because a single sluggish DSP (one that responds in 5–10 seconds instead of 100 ms) can trigger a chain reaction: auction timeouts propagate upstream, user-facing requests start timing out, and legitimate traffic stops being monetized. The circuit breaker cuts this Gordian knot by immediately opening the circuit when error rate exceeds a threshold or response time breaches a ceiling, preventing thread pool exhaustion and allowing fast recovery.

Three architectural approaches exist: in-process circuit breakers (Resilience4j, Polly) embedded in ECS Fargate tasks that protect individual downstream calls via NAT Gateway or VPC PrivateLink, service mesh circuit breakers (AWS App Mesh with Envoy) that apply protection transparently across all tasks, and hybrid adaptive approaches that combine circuit breaking with concurrency limits and ElastiCache for cluster-wide state sharing. Each trades observability, cross-service consistency, and operational complexity.

Key Requirements

Functional Requirements

• Automatic failure detection: Identify when a DSP or third-party service is slow or erroring and trip the circuit within 1–5 seconds of sustained degradation.
• Default fallback: When circuit is open, serve house ads or pass on that impression without blocking other auctions.
• Graceful recovery: Periodically test the downstream service (half-open state) to detect when it recovers.
• Per-dependency circuit state: Different DSPs may have different failure modes; circuit state must be granular (e.g., per DSP, per geographic region, per request path).
• Real-time alerting: Notify operators when circuit trips on critical dependencies.

Non-Functional Requirements

• Sub-millisecond circuit check overhead: Checking circuit state must not add latency to the critical path; use in-memory state or mesh-level checks.
• Consistent circuit state across instances: In a fleet of ad servers, circuit state for a given DSP must be shared or synchronized to avoid thundering herd when the DSP recovers.
• Fast recovery time (RTO): Once downstream service is healthy, circuit should move to half-open and gradually accept traffic again within 30–60 seconds.
• No cascading circuit trips: A circuit breaker on DSP-A should not trip a circuit breaker on DSP-B.
• Operational observability: Operators must see which circuits are open, how many requests were shed, and why.

Simplified Architecture

Architecture Proposals

Option 1: In-Process Circuit Breaker (Resilience4j / Polly) in ECS Fargate

Description

Each ECS Fargate task runs a circuit breaker library (Resilience4j for Java, Polly for .NET) for each downstream dependency. When calling a DSP, the library wraps the RPC call and tracks success/failure/latency. If error rate exceeds 50% in a 10-second window or p99 latency exceeds 500 ms, the circuit opens immediately; subsequent calls fail fast with a fallback (house ad). After 30 seconds, the circuit moves to half-open and allows a small number of probe requests; if they succeed, it closes and traffic resumes.

Outbound calls to external DSPs leave the VPC via NAT Gateway (for internet-routed endpoints) or VPC PrivateLink endpoints (for partner APIs). CloudWatch collects circuit state metrics from all tasks.

Pros

• Per-instance circuit state: Each Fargate task maintains its own circuit state, allowing aggressive local tuning.
• Fine-grained observability: Libraries like Resilience4j expose detailed metrics (requests/sec, error rate, latency percentiles) per circuit; shipped to CloudWatch.
• Zero cross-instance coordination: No shared state, no cache-coherency issues, no need for external coordination.
• Sub-millisecond overhead: Circuit check is a memory-local comparison; minimal latency tax.
• Mature ecosystem: Resilience4j, Polly are battle-tested in high-frequency trading, fintech, and streaming platforms.

Cons

• Thundering herd on recovery: If DSP-A recovers, each of the 100 Fargate tasks independently enters half-open, potentially creating a synchronization burst. Need care in tuning probe rates.
• Operational fragmentation: Operators must ship metrics to CloudWatch; no single pane of glass without dashboards. Requires CloudWatch Insights queries to see fleet-wide circuit state.
• Duplicate fallback logic: Each task must implement the same fallback (house ad selection, passthrough logic); risk of inconsistency if logic varies by task.
• Language-specific: Each application language needs its own library; heterogeneous stacks require multiple implementations.

Failure Modes & Mitigations

• False positives (circuit opens when DSP is healthy but slow): Tune thresholds carefully; use adaptive thresholds based on historical percentiles. Implement jitter in probe request timing to avoid synchronized probes.
• Stuck open circuit: If half-open state malfunctions (probe logic breaks), circuit stays open. Implement a maximum-open duration (e.g., 5 minutes, force-close the circuit). Log all circuit state transitions via CloudWatch Logs.
• Cascading circuit trips across dependencies: If DSP-A circuit opens and fallback logic tries DSP-B synchronously, DSP-B may become overloaded. Isolate fallback in its own thread pool and apply circuit breakers to fallback paths too.

When to Choose

Choose this option if you have a small number of distinct downstream services (< 10), you want maximum operational control at the task level, and your team is comfortable shipping metrics to CloudWatch. Common in organizations where services are owned by different teams and each team wants its own circuit-breaking parameters.

Option 2: Service Mesh Circuit Breaker (AWS App Mesh with Envoy)

Description

Deploy AWS App Mesh on ECS Fargate or EKS. App Mesh configures Envoy sidecar proxies on each task. All outbound RPC calls route through Envoy. Envoy's outlier detection engine monitors each upstream service; when a host (DSP server) exceeds a configurable failure threshold (e.g., 5 consecutive 5xx responses), Envoy ejects it from the load-balancing pool. Traffic automatically routes to healthy hosts. After a cooldown period (30 seconds), the host is re-added. App Mesh control plane (managed by AWS) coordinates policy across all tasks; no operator-managed Istio control plane.

Pros

• Single pane of glass: All circuit state lives in the App Mesh control plane; operators query AWS APIs to see all circuits.
• Cross-service consistency: All Fargate tasks use the same circuit-breaking policy; no divergence in fallback logic.
• Per-host ejection: Outlier detection ejects individual backend hosts, not entire services; if a DSP has 10 servers and 1 is slow, traffic is redistributed intelligently without losing the entire service.
• Application-agnostic: No need to modify application code; works with any language or framework.
• Managed infrastructure: AWS App Mesh control plane is fully managed; no operator overhead compared to self-hosted Istio.

Cons

• Additional latency: Every outbound request routes through Envoy sidecar; adds 1–5 ms per request even in the fast path.
• Operational complexity: Requires running Envoy sidecars alongside each Fargate task, configuring App Mesh VirtualNode/VirtualService. Sidecars consume memory (~50 MB).
• Coarser fallback control: If circuit opens, Envoy ejects the backend; application doesn't know circuit is open and can't implement custom fallback logic (must rely on Envoy's default retry policy).
• Limited debugging: Envoy internals are opaque to application; harder to troubleshoot circuit state or ejection decisions.

Failure Modes & Mitigations

• Cascading sidecar failures: If Envoy crashes, all application traffic through that sidecar stops. Mitigate with ECS task restart policies and liveness probes.
• Stale circuit state in sidecars: If App Mesh control plane is degraded, sidecars continue with cached configuration. Implement fallback mechanisms in application layer.
• Over-aggressive ejection: Outlier detection thresholds tuned too aggressively can eject hosts prematurely. Use CloudWatch metrics to validate thresholds against production traffic patterns before rolling out.

When to Choose

Choose this option if you're already running App Mesh on ECS/EKS, have hundreds of Fargate tasks, and need a single source of truth for circuit-breaker policies. Typical in large organizations with centralized infrastructure teams.

Option 3: Hybrid Adaptive Concurrency + Circuit Breaker with ElastiCache

Description

Combine in-process circuit breakers with adaptive concurrency limiting (Vegas algorithm or similar). Each outbound dependency is assigned a max-concurrency quota; as request latency increases, the quota shrinks, reducing load on the slow service. When latency exceeds a ceiling or error rate spikes, the circuit opens.

Concurrency state is stored in ElastiCache (Redis) so all Fargate tasks see the same limit; when a task detects degradation, it updates ElastiCache, and all other tasks immediately reduce their concurrency. This provides cluster-wide awareness without explicit coordination.

Sequence diagram for failure recovery

Pros

• Graceful degradation: Concurrency limiting causes natural backpressure; slow services are deprioritized rather than hard-rejected, preserving some revenue.
• Adaptive to load: Vegas algorithm adjusts concurrency limit based on observed latency, making it resilient to transient slowness without manual tuning.
• Cluster-wide awareness: ElastiCache Redis shared state means all Fargate tasks coordinate automatically; no thundering herd on recovery.
• Fewer false positives: By reducing concurrency, the service has breathing room; circuit opening becomes a last resort for catastrophic failures.
• Fast recovery: As latency improves, concurrency limit ramps back up; no half-open probing needed.

Cons

• Complex tuning: Vegas algorithm parameters (queue depth threshold, alpha, beta) require careful tuning; wrong settings lead to limit thrashing or too-aggressive reduction.
• Harder to reason about: Binary circuit state is simple; adaptive limits introduce state space complexity. Operators must understand queue depth vs. latency tradeoffs.
• ElastiCache dependency: All tasks depend on Redis; if ElastiCache fails, tasks fall back to local limits (less effective). Requires ElastiCache Multi-AZ for HA.
• Implementation effort: Requires custom code; not available in standard libraries out of the box.

Failure Modes & Mitigations

• Limit oscillation: If Vegas parameters are poorly tuned, concurrency limit can thrash (up-down-up-down cycle). Mitigate with larger alpha/beta dampening factors.
• Slow leak: If a service becomes degraded gradually (latency creeping up from 100 ms to 800 ms), adaptive limits might not react fast enough, leading to connection exhaustion. Add an emergency circuit breaker as a backstop.
• ElastiCache unavailable: If Redis fails, each task reverts to local limits; cluster coordination is lost. Mitigate with ElastiCache Multi-AZ failover and fallback to hardcoded safe limits.

When to Choose

Choose this option if you need high availability and want to minimize false positives (hard rejections), you're willing to invest in tuning Vegas parameters, and you have ElastiCache Multi-AZ for HA. Ideal for environments where DSP latency is naturally variable and you want to ride out degradation rather than fail fast.

Comparison Table

Recommendation

For most ad tech deployments on AWS, choose Option 1 (In-Process Circuit Breaker with Resilience4j or Polly) unless you already operate on App Mesh. Here's why:

1. Operational simplicity: No additional infrastructure; circuit breaking is co-located with the Fargate task. Easier to debug because state is local.
2. Sub-millisecond overhead: Ad auctions are latency-sensitive; every millisecond counts. In-process checking beats sidecar routing.
3. Mature ecosystem: Resilience4j is production-grade in thousands of companies; fewer surprises.
4. Customizable fallback logic: Each DSP may require different fallback strategies (some serve house ads, others pass silently); in-process breakers give you this flexibility.

Use Option 2 (App Mesh) only if you're already running App Mesh on ECS/EKS at scale and want centralized policy management. The 1–5 ms per-request latency tax is acceptable if you have many low-value impressions but not for premium auctions.

Adopt Option 3 (Adaptive + ElastiCache) if you need to squeeze the last 5% of revenue from degraded services, your team has expertise in control theory, and you're willing to operate ElastiCache Multi-AZ. Cluster-wide coordination via Redis is powerful but adds operational burden.

Hybrid recommendation: Start with Option 1, ship detailed metrics to CloudWatch, and layer on Option 3 (adaptive limits + ElastiCache) if circuit-breaker false-open rate exceeds 1% per month.

Capacity & Performance Notes

Assume an ad server cluster with 200 ECS Fargate tasks (2 vCPU, 4 GB RAM each), each handling 2,000 req/s, totaling 400k req/s platform-wide. Typical DSP call distribution:

• DSP-A: 40% of requests, 200 ms p99 latency, 1 ms network RTT. Concurrent connections: 200 tasks × 50 threads × 25% utilization = 2,500 active connections.
• DSP-B: 30% of requests, 150 ms p99, 5 ms network RTT. Concurrent connections: 2,000.
• House ads: 30% of requests, local in-memory lookup, < 1 ms.

In-Process Circuit Breaker overhead: Each call checks circuit state (memory comparison): ~0.01 ms. At 400k req/s, this is ~4 ms of CPU per core on a 2-vCPU task; negligible.

Half-open probing: When circuit is open, every 30 seconds send a single probe request to each DSP (200 tasks × 2 DSPs = 400 probe requests/min = 6.7 req/s). Network bandwidth: negligible.

NAT Gateway: 200 Fargate tasks making DSP calls through single NAT Gateway in subnet. NAT Gateway throughput: 45 Gbps per AZ. Outbound traffic: assume 5% of 400k req/s = 20k DSP calls/sec, each ~1 KB response = 20 MB/s. NAT Gateway is not a bottleneck.

Fallback latency: Serving house ads from local cache should be < 0.5 ms. If fallback is a network call (e.g., to a house-ad Lambda), ensure it's cached and has its own circuit breaker.