Messaging Platform with Limited TPS based on the vendor
CK 3788 Created at Updated at
19400
This highly available, horizontally scalable notification platform decouples message ingestion from delivery using an asynchronous, event-driven architecture to route messages across multiple channels and third-party vendors while strictly respecting dynamic transaction-per-second limits.
The ingress layer utilizes an API Gateway and a stateless Ingestion Service to persist and queue messages in under 50 milliseconds, leveraging isolated, partitioned message brokers to prevent head-of-line blocking between priority-tiered queues.
A distributed worker pool dispatches messages by verifying rate-limit permits against a clustered Redis store using atomic Lua scripts, employing a weighted fair queuing algorithm to prioritize critical transactional messages like one-time passwords over bulk campaigns.
To guarantee 99.99% uptime and durable, at-least-once delivery, the system is deployed across multiple availability zones, tracks end-to-end message lifecycles, enforces backpressure controls at the gateway, and utilizes circuit breakers to automatically route traffic to backup vendors during primary service outages.
Functional Requirements
Multi-Channel & Multi-Vendor Routing: The platform must support sending messages via different channels (SMS, Email, WhatsApp, Push Notifications) and route them to various third-party vendors (e.g., Twilio, SendGrid, MessageBird).
Strict Vendor Rate Limiting (TPS Cap): The core engine must strictly respect individual vendor-imposed Transactions Per Second (TPS) limits to prevent HTTP 429 Too Many Requests responses or account suspensions.
Dynamic Rate-Limit Configuration: Administrators must be able to update vendor TPS allocations dynamically without restarting the system.
Priority-Based Queuing: Critical messages (e.g., One-Time Passwords (OTP), transaction alerts) must bypass non-critical bulk messages (e.g., marketing campaigns) while staying within the vendor’s TPS limits.
Delivery Status Tracking: The system must track message lifecycles (Ingested, Queued, Processing, Sent, Delivered, Failed) and support vendor callbacks (webhooks).
Automatic Retries & Failover: Failed deliveries must be retried with exponential backoff. If a primary vendor suffers an outage, traffic must automatically fail over to a backup vendor.
Non-Functional Requirements
High Availability (HA): The system must guarantee 99.99% uptime with no single point of failure (SPOF).
High Scalability (Ingress): Ingestion must scale horizontally to handle sudden traffic spikes (e.g., millions of API requests during peak hours), regardless of downstream vendor throughput limitations.
Durability (At-Least-Once Delivery): Messages must not be lost once acknowledged by the ingestion API. They must be persisted to disk/database before delivery attempts.
Isolation (No Head-of-Line Blocking): Slowdown or failure of one vendor must not affect message delivery to other vendors.
Low Latency (Ingress): The client ingestion API must respond in under 50ms.
System Architecture
The architecture decouples message ingestion from message delivery using an asynchronous, event-driven pattern with message queues and rate-limited worker pools.
API Gateway: Validates authentication tokens, enforces client-level rate limits, and routes requests to the Ingestion Service.
Ingestion Service: A lightweight, stateless microservice. It assigns a unique Message ID, persists the message state as INGESTED in the database, and publishes the message payload to the Message Broker.
2) Message Broker (Distributed & Partitioned)
Uses technology like Apache Kafka or RabbitMQ.
Topic/Queue Isolation: Designed with dedicated queues per vendor and priority level (e.g., vendor_twilio_otp, vendor_twilio_marketing). This layout prevents bulk campaigns from delaying transactional messages (Head-of-Line blocking).
3) Dynamic Rate Limiter (Redis Cluster)
Maintains real-time sliding window counters or token buckets using an in-memory database (Redis).
Stores global rate-limit state accessible by all instances of the Worker Service.
4) Distributed Worker Pool (Dispatchers)
Stateless consumer group.
Each worker pulls messages from assigned queues, checks the rate limiter for a permit, and dispatches the message to the vendor API upon permit approval.
5) Datastores
Primary DB (PostgreSQL / Cassandra): Holds the source of truth for message delivery records, audit logs, and delivery receipts.
Config Store (Consul / ZooKeeper): Holds metadata configurations for vendors (endpoints, active/passive status, TPS limits).
Rate Limiting & Queue Management Strategy
To enforce strict vendor TPS limits without dropping messages, the system separates queue storage from traffic-shaping token buckets.
1) Token Bucket Algorithm with Redis
We implement the Token Bucket algorithm over Redis using Lua scripting for atomicity:
Bucket Key: Each vendor profile has a key rate:limit:vendor_id.
Capacity & Refill: If Vendor A has a limit of 100 TPS, the bucket capacity is set to 100, refilling with 100 tokens every second.
Atomic Retrieval: Workers execute a Lua script to check if tokens are available. If tokens > 0, the script decrements the token count and returns true.
-- Redis Lua Script for Rate Limiting
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local current = tonumber(redis.call('get', key) or "0")
if current < limit then
redis.call("INCR", key)
if current == 0 then
redis.call("EXPIRE", key, 1) -- 1-second sliding window
end
return 1 -- Permit Granted
else
return 0 -- Rate limit reached
end
2) Priority Queue Scheduling
To ensure High-Priority (OTP) messages are not starved by Low-Priority (Bulk) messages:
Workers pull messages from the high-priority queue first.
A weighted fair queueing (WFQ) algorithm is applied: if both queues have messages, the dispatcher consumes in an 8:2 ratio (80% OTP, 20% Bulk) to avoid completely starving bulk queues while prioritizing critical traffic.
Scalability and High Availability (HA)
1) Horizontal Scalability
a) Stateless Microservices
Both the Ingestion Service and Worker Service store no local state. They can be auto-scaled using Kubernetes Horizontal Pod Autoscaler (HPA) based on CPU usage or Queue Depth metrics.
b) Queue Partitioning
Using Apache Kafka, topics are partitioned. Multiple worker instances can consume from different partitions of the same topic in parallel, ensuring high throughput up to the rate limit constraint.
c) Redis Clustering
Redis is configured in a clustered mode with sharding. Since keys are vendor-specific (rate:limit:vendor_a), they distribute evenly across the Redis shards using hash tags (e.g., {vendor_a}:rate_limit).
2) High Availability & Fault Tolerance
3) Multi-Availability Zone Deployment
All services are deployed across three Availability Zones (AZs). If an entire AZ goes offline:
The load balancers route traffic to surviving zones.
The Message Broker (Kafka) keeps running because replication factors are set to 3 (min.insync.replicas=2).
4) Circuit Breaker Pattern for Vendor Failures
To handle vendor outages or latency spikes, the system uses a Circuit Breaker (e.g., Resilience4j patterns):
Closed State: Normal operations. Messages route to Primary Vendor.
Open State: If the failure rate of the Primary Vendor exceeds 50% (e.g., HTTP 5xx errors or timeouts) within a sliding window, the circuit breaker trips.
Failover Action: The worker pool automatically switches routing configurations to point to the designated Backup Vendor (utilizing the backup vendor's independent TPS queue).
Half-Open State: After a configured cooldown period, a small percentage of traffic is routed back to the Primary Vendor to test for recovery.
5) Backpressure Control
If downstreams are completely broken and queues fill up beyond storage thresholds:
The Ingestion Service receives pushback alerts.
The API Gateway begins rate-limiting ingress traffic, returning HTTP 429 to bulk clients to protect system stability.
Message Lifecycle Sequence
The diagram below illustrates the end-to-end flow of a message as it passes through the system, gets throttled according to vendor TPS limits, and completes delivery.
