Ad Tech Multi-Tenancy: A Spectrum of Isolation Architectures on AWS

Ad tech platforms serve hundreds to thousands of customers simultaneously: advertisers, agencies, publishers, networks. Each customer (tenant) expects isolated data, isolated compute resources, and isolated audit trails. Multi-tenancy is the architectural choice to share infrastructure (databases, servers, message queues) efficiently while maintaining logical and physical isolation. The challenge is preventing noisy neighbor problems—one tenant's expensive query or burst of traffic must not degrade another's performance—and enforcing fine-grained access control.

Three approaches span a spectrum: shared cluster with tenant tagging and row-level security (RLS) at the database layer, per-tenant logical schemas or databases within shared infrastructure, and per-tenant cells (complete physical isolation: dedicated cluster, database, message queues). The choice depends on customer base size, SLA strictness, and cost tolerance. A startup serving 100 agencies uses shared cluster with RLS; a mature platform with 10,000 SMB advertisers might use per-schema isolation; a mega-platform with demanding enterprise customers operates per-tenant cells.

This case explores isolation strategies, noisy-neighbor mitigation, tenant-aware billing/metering, and audit isolation using AWS services (Aurora PostgreSQL, DynamoDB, ECS Fargate, ElastiCache, AWS Cloud Map, IAM).

Key Requirements

Functional Requirements

• Logical isolation: Tenant A cannot see Tenant B's data (campaigns, conversions, reports) via any query or API.
• Rate limiting: Tenant A's workload (QPS, concurrent connections, storage growth) does not impact Tenant B's SLO.
• Multi-tenancy abstraction: Queries automatically filtered by tenant_id; developers should not need to remember "WHERE tenant_id = ?" in every query.
• Tenant lifecycle: onboard new tenant (create schema/DB, assign ID, configure quotas), offboard (archive data, verify deletion).
• Billing/metering: track usage (API calls, storage, compute) per tenant for chargeback or revenue recognition.
• Audit trail: tenant-specific logs; audit access to any tenant's data.
• Fair resource share: under contention, quota-based allocation prevents starvation.

Non-Functional Requirements

• Tenant count: support 100–100,000+ tenants without linear cost growth.
• Noisy neighbor resilience: one tenant's runaway query must not block other tenants' requests. p99 latency impact <10%.
• Query isolation: tenant-specific queries, indexes, and query plans do not interfere.
• Storage growth: one tenant maxing out storage quota should not affect others' provisioning.
• Failover / disaster recovery: tenant data must not be lost or leaked during failover.
• Compliance: GDPR right-to-forget, CCPA deletion, SOC 2; tenant data fully erasable without affecting other tenants.

Simplified Architecture

Architecture Proposals

Option 1: Shared Cluster with Tenant_ID Tagging + RLS + Rate Limiting

All tenants share one Aurora PostgreSQL cluster, one ElastiCache Redis cluster, and one ECS Fargate service pool. Each row in a multi-tenant table includes a tenant_id column. Aurora Row-Level Security (RLS) enforces tenant filtering at the database layer: role-based policies automatically filter queries. Application-level middleware (running on ECS Fargate) injects tenant_id into request context and JWT claims. Rate limiting (Redis-backed, per tenant) enforces quota on API calls, concurrent connections, and query resource consumption.

Pros:

• Cost efficient: single Aurora cluster, ElastiCache, ECS Fargate fleet; minimal operational overhead.
• Simple scale-out: add tenants without provisioning new infrastructure.
• Uniform deployment: updates, patches applied once to benefit all tenants.
• Resource sharing: idle tenant does not reserve resources; spare capacity used by active tenants.
• IAM-driven access: JWT claims map to IAM roles; tenant isolation via IAM permission boundaries.

Cons:

• Noisy neighbor risk: expensive query from Tenant A consumes CPU/disk I/O, slowing Tenant B.
• Query plan interference: one tenant's large JOIN query can cause PostgreSQL planner to choose suboptimal plans for other tenants' queries.
• Storage contention: if Tenant A's storage quota not enforced at DB level, could fill cluster disk, breaking all tenants.
• Debugging complexity: logs, metrics for all tenants mixed; must tag and filter.
• Shared resource limits: one tenant hitting connection limit (Aurora max_connections configurable, typical 5,000) blocks others.

Failure Modes & Mitigations:

• Tenant A's expensive query: implement query timeout (e.g., 30 seconds) at application layer; kill runaway queries via Lambda. Per-tenant query budget (CPU cost units/hour) tracked in CloudWatch.
• Connection pool exhaustion: Tenant A opens 50 connections, leaving limited pool for others. Mitigate: enforce max_connections per tenant in application (e.g., Tenant A max 25, others 5 each) via PgBouncer on ECS.
• RLS bypass: developer forgets to apply RLS policy to new table. Mitigate: policy-as-code in Terraform; default-deny RLS on all tables; deny SELECT by default, require explicit grant.
• Storage disk full: Tenant A's audit log grows unbounded. Mitigate: enforce storage quota per tenant (10 GB default) via S3 archival Lambda trigger; alert at 80%, block writes at 100%.

When to Choose:

• Tenant count <500 and relatively homogeneous (similar query patterns, storage size).
• SLA tolerant of occasional latency spikes (p99 <500 ms acceptable).
• Operations team small; cannot manage per-tenant infrastructure.
• Cost per tenant must be minimal (startup mode).

Option 2: Per-Tenant Schemas or Databases (Logical Isolation)

Shared Aurora PostgreSQL cluster; each tenant gets dedicated schema (or separate database). Data is logically isolated at the schema level. Tenant_id no longer needed in every table; isolation enforced by schema-level permissions. Rate limiting still shared cluster-wide via ElastiCache, but each tenant's queries run in separate schema with dedicated indexes and statistics. Aurora supports up to 1000+ schemas per cluster.

Pros:

• Query plan isolation: Tenant A's queries do not interfere with planner for Tenant B; each schema has independent statistics.
• Cleaner access control: Postgres roles per tenant; GRANT only on schema_42 to tenant_42_role.
• Easier debugging: tenant A's tables, indexes isolated; monitoring more transparent.
• Index customization: Tenant A can add custom indexes without affecting others.
• Data migration: moving tenant to different infrastructure simpler (dump schema, restore elsewhere).

Cons:

• Storage overhead: per-tenant schema overhead (system tables, constraints). 1000 tenants × 1 MB schema metadata ≈ 1 GB wasted.
• Connection contention: shared pool; still vulnerable to noisy neighbor stealing connections.
• Maintenance complexity: schema migration (e.g., adding column) requires batching across tenants; cannot be instant.
• Shared query cache: Aurora query cache (prepared statements) shared; one tenant's queries can evict another's from cache.
• Still single database failure: outage affects all tenants; no isolation at cluster level.

Failure Modes & Mitigations:

• Tenant A maxes storage quota: Aurora disk fills, all tenants blocked. Mitigate: per-schema tablespace (separate EBS volume), enforce quota at tablespace level, alert before hit.
• Tenant A's DDL locks schema: ALTER TABLE on Tenant A's campaigns blocks schema operations. Mitigate: use CONCURRENTLY option; schedule DDL during low-traffic windows; use EventBridge for coordinated maintenance.
• Connection pool starvation: Tenant A opens 30 connections; others starve. Mitigate: PgBouncer connection pooling on ECS, per-tenant connection limits.

When to Choose:

• Tenant count 500–5,000 with moderate isolation requirements.
• Can manage per-schema backups, migrations.
• Operations team 2–5 people; can automate schema provisioning via Terraform.
• Need query plan isolation but not full physical isolation.

Option 3: Per-Tenant Cell (Full Physical Isolation)

Each tenant (or cohort of <10 top tenants) gets dedicated infrastructure: Aurora PostgreSQL instance, ElastiCache Redis cluster, ECS Fargate service, Kinesis Data Streams topic. Tenants in different cells never share compute or storage. Only metadata (AWS Cloud Map service registry), billing systems (DynamoDB), and control plane (Route 53) are shared. This is the most operationally complex but provides complete isolation.

Pros:

• Perfect isolation: Tenant A's hardware failure, DoS, or runaway query zero impact on Tenant B.
• Custom SLA: each cell can have different Aurora backup frequency, failover policy, Kinesis shard count.
• Scaling knob: Tenant A's growth does not affect capacity planning for others.
• Debugging clarity: all metrics, logs, queries isolated; easy to correlate events to tenant.
• Compliance: Tenant A in EU (eu-west-1), Tenant B in US (us-east-1); deploy cells in correct region; KMS keys per tenant.
• Data residency: each tenant's data stays in specified region; satisfies regulatory requirements.

Cons:

• Operational burden: managing 100+ Aurora clusters, monitoring, patching, backups per cell.
• Cost: each cell 3–5x cost of shared cluster; provisioning N cells costs linear in N.
• Idle waste: small tenant in dedicated cell wastes capacity; must enforce minimum tenant tier.
• Data transfer: cross-cell queries, reporting impossible without moving data; eventual consistency model.
• Deployment complexity: infrastructure-as-code (Terraform, CloudFormation) with multi-region failover must be automated.

Failure Modes & Mitigations:

• Cell provisioning takes hours: slow onboarding. Mitigate: template infrastructure via Terraform modules; 15-min provisioning SLA.
• Data replication lag across cells: batch reporting queries hit stale data. Mitigate: eventual consistency acceptance; separate read-only Aurora replica in reporting region with hourly sync.
• Cost explosion for small tenants: 1000 micro-tenants × $5,000/mo per cell = $5M/mo unviable. Mitigate: multi-tenant shared cluster for low-tier; cells only for enterprise.

When to Choose:

• Tenant count <100 but each is enterprise (high SLA, compliance, custom integration).
• Geographic distribution required (data residency, latency).
• Tenant isolation for competitive or security reasons critical.
• Operations team >10 people; infrastructure-as-code maturity high.

Comparison Table

Recommendation

Hybrid tiering strategy using AWS services:

Tier 1 (Free / Micro, <$100/mo): Shared Cluster with RLS

• Single Aurora PostgreSQL cluster (shared across all Tier 1 tenants).
• Tenant_id + RLS + ElastiCache rate limiting.
• Storage quota: 1 GB, API quota: 10k calls/day.
• SLA: best-effort, no uptime guarantee.
• Scaling limit: ~5,000 tenants before p99 latency unacceptable.
• Monitoring: CloudWatch metrics per tenant_id; SNS alerts on quota breaches.

Tier 2 (Pro / Standard, $100–2,000/mo): Per-Schema or Per-Database

• Dedicated schema within shared Aurora cluster (or separate Aurora instance per 10 tenants).
• Storage quota: 100 GB, API quota: 1M calls/day.
• SLA: 99.5% uptime commitment.
• Scaling limit: ~2,000 tenants on single cluster; beyond, shard into multiple clusters (geographic or cohort-based) via Route 53.
• Monitoring: dedicated CloudWatch dashboards per schema; RDS Performance Insights for query optimization.

Tier 3 (Enterprise, >$2,000/mo): Per-Cell

• Dedicated Aurora PostgreSQL instance, ElastiCache, ECS Fargate service.
• Custom SLA, data residency (specify region), HA/DR policies.
• Scaling limit: each cell scales to 1B+ impressions/day independently.
• Onboarding: 2–4 weeks (compliance review, security audit).
• Provisioning: AWS Cloud Map service registry + Route 53 health checks + automatic failover.

Transition logic:

• Free tier automatically moved to Pro schema at 1GB storage or 100k daily API calls.
• Pro tenants requesting >99.9% SLA or data residency constraints automatically provisioned to Enterprise cell.

Capacity & Performance Notes

Shared Cluster (RLS, Option 1):

• Aurora PostgreSQL instance: 32 vCPU, 128 GB RAM, 2 TB EBS gp3.
• ElastiCache Redis: single-node cache.db.r6g.xlarge (4 vCPU, 26 GB RAM).
• ECS Fargate: 5–10 tasks (2 vCPU, 4 GB RAM each).
• Density: 5,000 tenants with 100 MB avg storage each = 500 GB data (50% utilized capacity).
• Per-tenant rate limits: 100 concurrent connections (PgBouncer: 10–20 per tenant), 10,000 API calls/day (100 req/sec per tenant).
• Query latency: p50 20 ms, p99 100–200 ms under quota enforcement.
• Cost: ~$10,000/month Aurora + ElastiCache + ECS + operations = ~$2/tenant/month.

Per-Schema (Shared Cluster, Multiple):

• Cluster pool: 5 Aurora clusters × 32 vCPU each.
• ElastiCache: 5 Redis clusters (one per Aurora cluster for affinity).
• ECS Fargate: 20 tasks (4 vCPU, 8 GB RAM each).
• Density: 1,000 tenants per cluster; 5 clusters = 5,000 tenants total.
• Per-tenant rate limits: 50 concurrent connections, 100,000 API calls/day.
• Query latency: p50 15 ms, p99 50–100 ms (less RLS filtering overhead).
• Cost: ~$50,000/month clusters + ops = ~$10/tenant/month.

Per-Cell (Enterprise, Option 3):

• Single cell: 1 Aurora instance (16 vCPU, 64 GB RAM), 1 ElastiCache (cache.db.r6g.large), 4 ECS Fargate tasks.
• Cost: ~$5,000–15,000/month per cell depending on size.
• Single cell can handle 1–10 enterprise tenants or 1 very large tenant.
• AWS Cloud Map: $0.50/service + $0.10 per million requests (negligible cost).

Billing/Metering:

• Storage: billed per GB; metered daily from Aurora table size, exported to DynamoDB for chargeback.
• API calls: metered via ECS Fargate middleware counter; reset daily/monthly per tenant, logged to CloudWatch.
• Compute: not directly billed (included in tier cost).
• Example: Pro tier $500/month includes 100 GB storage + 1M API calls; overage $0.01/GB + $0.001 per additional 1000 calls.