Architecting Ad Tech's Terabyte Log Pipeline - Log Aggregation

A single ad exchange processing 100k requests per second generates terabytes of logs per day: impression logs (one per auction), error logs (DSP failures, timeouts), financial logs (pacing, budget, spend), and audit logs (campaign changes, user access). This torrent of data is too large for traditional centralized logging (Splunk, ELK) and too critical to lose (impressions are revenue, audit logs are legally required).

Log aggregation in ad tech requires three simultaneous guarantees: lossless ingestion (every impression counted), queryability within seconds (real-time dashboards), and long-term archival (7-year compliance retention). Batch processing (Spark, EMR) and streaming (Amazon MSK, Kinesis) must coexist; streaming provides real-time queries, batch provides durability and cost-effective storage.

Three strategies scale log handling: lambda-style (Amazon MSK + Flink on ECS + Parquet on S3, streaming + batch hybrid), serverless (Kinesis Firehose direct to S3 with format conversion, minimal ops), and modern open table formats (Apache Iceberg / Delta Lake on S3 with Glue catalog, supporting schema evolution and time-travel queries).

Key Requirements

Functional Requirements

• Lossless ingestion: Every log entry that arrives at the platform must be persisted; no silent drops. Acknowledgment to sender only after write is durable (committed to S3 or Kinesis).
• Real-time queryability: Operators and analytics teams must query logs within 1 second of an event; dashboards refresh every 10 seconds.
• Multi-tenant isolation: Logs from different ad networks or customers must be segregated; no cross-customer data leaks.
• Schema evolution: Campaign log schema evolves over time (new fields added, old fields removed); old and new events must be queryable together.
• Long-term retention: Impressions and financial logs retained for 7 years (compliance); delete old logs on schedule without re-ingesting.

Non-Functional Requirements

• Throughput scalability: Handle 100k logs/sec today, 1M logs/sec in 3 years, without rebuilding infrastructure.
• Latency: End-to-end latency from log generation to queryable state < 5 seconds for 95% of events.
• Cost efficiency: Log storage + processing < 20% of platform revenue. Compression and tiered storage are essential.
• Operational simplicity: Operators should not need to manage hundreds of Kafka brokers or Spark clusters; delegate to managed services where possible.
• Disaster recovery: If the log platform fails, recover within 1 hour; no data loss during recovery.

Simplified Architecture

Architecture Proposals

Option 1: Amazon MSK + Flink → Parquet on S3 (Lambda Architecture)

Description

Ad servers emit logs (Avro or JSON) to Amazon Managed Streaming for Apache Kafka (MSK). A Flink streaming job (running on ECS Fargate or EMR) consumes logs in real-time, performs light processing (deduplication, filtering), and writes windowed batches (e.g., every 5 minutes) to S3 as Parquet files (partitioned by date/campaign). A separate Spark batch job (EMR on-demand) runs hourly, reading Parquet files from S3, aggregating, and writing summary tables (impressions by campaign, CPM by geography, etc.) back to S3. Athena or Trino queries both streaming (Parquet files) and batch (aggregated tables) outputs.

Pros

• Separation of concerns: Real-time Flink handles streaming, batch Spark handles aggregation; easy to tune independently.
• Cost-effective storage: Parquet is highly compressible (100:1 for structured logs); S3 is cheap for long-term retention (use S3 Lifecycle to tier to Glacier).
• Query flexibility: Athena can query raw Parquet files (detailed impressions) or aggregated tables (fast summaries).
• Proven technology: MSK, Flink, Spark, S3 are battle-tested in enterprises; ecosystem is mature.
• Scalable: Flink and Spark both scale horizontally; MSK can handle 100k+ msg/sec (shard autoscaling).

Cons

• Operational complexity: Three separate systems (MSK cluster, Flink job on ECS, Spark jobs on EMR) to deploy, monitor, and patch.
• Latency between layers: Flink writes every 5 minutes, Spark reads 30 minutes later; aggregated data is always 30+ minutes stale.
• Debugging is hard: A missing impression might be due to MSK failure, Flink failure, or Spark failure; tracing across systems is tedious.
• Double storage: Both Parquet files and summary tables stored on S3; effective cost per TB is higher.
• Schema management overhead: Avro/Parquet schema changes require coordinating with producers, Flink job, and Spark job.

Failure Modes & Mitigations

• MSK retention exceeded before Flink catches up: If Flink job is down for > 7 days, oldest logs are deleted from MSK before Flink can consume them. Mitigate by replicating MSK to S3 immediately (MirrorMaker stream to S3) as a cold backup.
• Duplicate impressions from Flink reprocessing: If Flink crashes and restarts from an earlier checkpoint, some logs are reprocessed and written to S3 twice. Mitigate by using Flink's exactly-once semantics (Kafka transactions, idempotent S3 writes) or deduplication in Spark.
• Spark job failure silently drops aggregates: If Spark job fails, no summary tables are written, but raw Parquet still exists; queries might be inconsistent (some data aggregated, some not). Mitigate by emitting completion markers; if marker missing, re-run Spark job via Step Functions.

When to Choose

Choose this option if you have existing MSK infrastructure, your team has Flink/Spark expertise, and you can tolerate 30–60 minute staleness in aggregated data. Common in large tech companies (Google, Netflix, LinkedIn) that built log infrastructure before managed services were available.

Option 2: Kinesis Firehose → S3 → Athena (Serverless)

Description

Ad servers emit logs directly to Amazon Kinesis Data Firehose (fully managed, no cluster management). Firehose buffers logs in memory (configurable window: 60 seconds or 128 MB), batches them, optionally compresses (gzip), and writes to S3 as Parquet files. Enable Firehose data transformation via Lambda; lightweight processing (e.g., filtering sensitive fields, enriching geolocation) happens inside Firehose. Athena queries the S3 Parquet files directly; no need for separate batch jobs.

Pros

• Minimal operations: Firehose is fully managed; no MSK brokers, Flink jobs, Spark clusters to run.
• Simple pipeline: One path from source (ad server) to sink (S3); less to debug.
• Automatic format conversion: JSON → Parquet conversion built in; no custom Flink job needed.
• Cost-effective: Pay for Firehose data transfer (~$0.029 per GB) + S3 storage; no compute layer overhead.
• Built-in reliability: Firehose retries on S3 write failure, implements exactly-once delivery semantics, supports extended retention in redrive S3 bucket.

Cons

• Limited customization: Firehose transformations via Lambda are simple (field redaction, timestamp parsing); complex logic (deduplication, windowing, aggregation) not possible.
• Query latency: Data is written to S3 every 60 seconds or 128 MB (whichever first); Athena query of fresh data is 1–2 second latency after event (acceptable for some use cases, not for real-time dashboards).
• No real-time aggregates: Firehose doesn't pre-aggregate; Athena queries raw data every time (slow for large datasets). Must create separate materialized views manually (via Athena, Glue).
• Parquet partitioning overhead: Firehose can partition by date/campaign, but configuration is limited. Athena queries may scan more files than necessary.

Failure Modes & Mitigations

• Firehose Lambda transform fails: If Lambda throws an error, Firehose retries; if all retries fail, record goes to redrive S3 bucket (configurable). Mitigate by implementing comprehensive error handling in Lambda (fallback to default values, log error to CloudWatch Logs).
• S3 write failure: Firehose retries; if S3 is unavailable for > 24 hours, logs go to redrive bucket or are dropped. Mitigate by using S3 cross-region replication and setting up CloudWatch alarms.

When to Choose

Choose this option if you're AWS-native, your team is small (no dedicated data engineering), and you can tolerate 1–2 second latency to queryable state. Suitable for startups or small-to-medium ad networks.

Option 3: Vector + Amazon MSK + Apache Iceberg / Delta Lake (Modern Open Format)

Description

Use Vector (lightweight, open-source log shipper) to collect logs from ad servers and emit to Amazon MSK. A Flink job consumes MSK, performs transformations, and writes to S3 using Apache Iceberg or Delta Lake format (modern open table formats with ACID semantics, schema evolution, and time-travel queries). Store Iceberg metadata in AWS Glue catalog (shared, managed). Query via Athena, Spark SQL (EMR), or Trino.

Iceberg/Delta advantages: schema changes don't require data rewriting (stored as metadata); time-travel queries (query data as of a specific timestamp); ACID transactions prevent duplicates; Z-order/data skipping speeds up queries.

Sequence for schema evolution and time-travel query

Pros

• Schema evolution without rewriting: Add new fields to logs; Iceberg auto-evolves schema; old partitions still queryable.
• Time-travel queries: Query data as of any timestamp; useful for auditing, debugging, or replaying old configs.
• ACID transactions: Multiple Flink jobs can write to the same table without duplicates or conflicts.
• Open standard: Iceberg and Delta are not vendor-locked; works with Athena, Spark, Trino, and others.
• Query optimization: Iceberg's data skipping, Z-order, and metadata caching make queries faster without manual partitioning.

Cons

• Newer technology: Iceberg and Delta Lake are less battle-tested than MSK/Spark (though adoption is growing rapidly).
• Operational complexity: Vector + MSK + Flink + Iceberg is a 4-layer stack; requires expertise in each.
• Cost: Flink cluster + Iceberg metadata overhead slightly higher than Firehose. Cost is still reasonable (< 30% higher than Option 2).
• Debugging: Time-travel and schema evolution are powerful but harder to reason about; errors can be subtle.

Failure Modes & Mitigations

• Schema collision: If two Flink jobs evolve schema simultaneously (one adds field A, another adds field B), Iceberg transaction may conflict. Mitigate by serializing schema changes through a schema registry or using Flink's distributed transaction coordinator.
• Iceberg metadata bloat: Over time, Iceberg metadata (snapshots, manifests) accumulates. Mitigate by running Spark maintenance jobs (expire_snapshots, remove_orphan_files) daily via Step Functions.

When to Choose

Choose this option if you expect schema changes to be frequent, need time-travel queries for debugging, and your team is comfortable with modern data tooling. Typical in companies with dedicated data engineering teams (data platforms, data lakes). Future-proof option if your data lake is long-term.

Comparison Table

Recommendation

For most ad tech platforms, choose Option 2 (Kinesis Firehose → S3 → Athena) unless you have complex requirements or existing infrastructure. Here's why:

1. Operational simplicity: Fully managed Firehose eliminates operational burden of MSK/Flink/Spark clusters.
2. Cost-efficiency: Cheapest option (pure managed services); no compute overhead.
3. Fast iteration: Easy to modify Firehose configuration (buffer size, compression) without redeploying code.
4. Acceptable latency: 1–2 second latency is fine for most real-time dashboards and ad operations.

Use Option 1 (MSK → Flink → Parquet) if:

• You have existing MSK/Flink infrastructure and team expertise.
• You need sub-second real-time aggregates (very rare in ad tech; most dashboards refresh every 10 seconds anyway).
• You operate on non-AWS cloud and Firehose is not available.

Adopt Option 3 (Iceberg) if:

• Your data schema evolves frequently (> 1 change/week).
• You need time-travel queries for debugging or compliance audits.
• Your logs are critical long-term assets (7-year retention) and query performance matters.

Recommended hybrid approach:

1. Start with Option 2 (Firehose → S3).
2. At 500k logs/sec (when Firehose costs become significant), migrate to Option 3 (Iceberg).
3. Use Firehose for non-critical logs (debug logs), Iceberg for critical logs (impressions, financials).

Capacity & Performance Notes

Assume 100k logs/sec, 24/7 (8.64 billion logs/day).

Per-log size: Impression log (JSON): campaign_id (4B), user_id (32B), impression_id (36B), timestamp (8B), price (8B), metadata (50B) = ~150 B per log. Compressed (gzip): ~50 B.

Daily volume: 8.64B logs × 50 B (compressed) = 432 GB/day. Monthly: ~13 TB. Yearly: 158 TB.

Option 2 (Firehose) cost:

• Ingestion: 432 GB/day × $0.029/GB = $12.5/day.
• S3 storage (S3 Standard): 432 GB/day × 365 days = ~158 TB/year. S3 Standard: ~$4.50 per TB/month. Cost: 158 TB × 4.5 = $711/month. After 1 year, tier to S3 Glacier: ~$1 per TB/month. Average (assuming 7-year retention): ~$3.5k/month.
• Athena query cost: Assume 10 queries/day scanning 1 TB each. Cost: 10 × 1 TB × $5/TB = $50/day = $1.5k/month.
• Total: ~$12.50/day ingestion + ~$3.5k/month storage + $1.5k/month queries = ~$5k/month.

Option 1 (MSK → Flink → Parquet) cost:

• MSK cluster (3 brokers, 8-core, 64GB RAM each; autoscaling): ~$3k/month.
• Flink cluster on ECS (2 task managers, 8-core, 32GB RAM each; or use Kinesis Managed Service for Flink): ~$1.5k/month.
• Spark cluster (EMR on-demand, 10 hours/day for batch jobs): ~$500/month.
• S3 storage (same tiering as Option 2): ~$3.5k/month.
• Total: ~$8.5k/month (70% higher than Option 2).

Option 3 (Iceberg) cost:

• Vector + MSK + Flink (optimized): ~$5k/month.
• S3 storage (same as Option 2): ~$3.5k/month.
• Iceberg metadata overhead: ~5% additional storage, +$175/month.
• Total: ~$8.7k/month (similar to Option 1).

Latency SLO: 95th percentile latency from log emission to Athena queryable < 5 seconds (Option 2) or < 10 seconds (Option 1/3). Achieved by: Firehose buffer (60 sec max) + S3 write (< 5 sec) + Athena metadata refresh (< 10 sec). For MSK-based options: MSK commit (< 100 ms) + Flink windowing (5 min) = 5+ minutes.