Enterprise-Scale 100k+ TPS Order Processing System Architecture

This architecture is designed to handle ultra-high concurrency (100,000+ Transactions Per Second) during peak events (e.g., flash sales, limited-edition drops) while ensuring zero data loss, strict transactional integrity, and sub-second response times.

1. High-Level Architecture Overview

To process 100k+ TPS without overloading the persistent storage layer, the architecture decouples synchronous user interactions from asynchronous database writes.

2. Ingestion & Buffering Layer (Throughput Distribution)

Directly hitting a Relational Database Management System (RDBMS) with 100k TPS will cause instant lock contention and system failure. This layer absorbs sudden traffic spikes and levels the load.

API Gateway & Rate Limiting

• Edge Protection: Deploy an API Gateway (e.g., Kong, AWS API Gateway) to filter out duplicate requests, perform token bucket rate limiting, and block malicious bot traffic.
• Decoupled Order Intake: The Order Edge Service acts as a lightweight ingestion engine. Its sole responsibility is validation, checking/deducting memory-based inventory, publishing an "Order Created" event, and returning a 202 Accepted status to the client.

Message Queue Buffering (Apache Kafka)

• Asynchronous Processing: Instead of synchronously waiting for database writes, order requests are serialized into event payloads and pushed to Kafka.
• Partitioning Strategy: Kafka topics are partitioned using a consistent hashing key (e.g., user_id or order_id). This guarantees:
  • Horizontal scalability by distributing partitions across multiple brokers.
  • In-order processing of messages for any single user.
• Backpressure Control: Consumers pull messages from Kafka in batches (e.g., 500 records per batch). If the downstream database slows down, consumers naturally throttle their ingestion rate, protecting the DB from crashes.

3. High-Speed Concurrency & Inventory Control

Preventing "overselling" (double-spending of inventory) at 100k TPS requires extremely fast, atomic operations before the data ever reaches the physical database.

In-Memory Inventory Reservation (Redis Cluster)

• Why Redis?: Single-threaded execution of commands ensures absolute serializability of operations at sub-millisecond latencies.

Optimization: Lua Scripting vs. Distributed Locks

• The Problem with Distributed Locks (Redisson): Acquiring, holding, and releasing locks across network boundaries introduces latency and overhead that degrades performance at 100k TPS.
• The Solution (Lua Scripts): Rather than locking, run atomic Lua scripts directly inside Redis. Because Redis executes scripts atomically, the check-and-deduct sequence happens in a single step without thread context switching.

-- Atomic Inventory Deduction Lua Script
local inventoryKey = KEYS[1]
local requestedQuantity = tonumber(ARGV[1])

local currentStock = tonumber(redis.call('get', inventoryKey) or "0")

if currentStock >= requestedQuantity then
    redis.call('decrby', inventoryKey, requestedQuantity)
    return 1 -- Success
else
    return 0 -- Insufficient Stock
end

Read-Through Cache & Reconciliation

• Redis as Source of Truth for Stock: During peak sales, Redis acts as the temporary source of truth for stock levels.
• Reconciliation: The asynchronous consumer updates the persistent database. If a database transaction fails (e.g., due to payment failure), a compensating event is published to return the stock back to Redis.

4. Database Scale-Out & Data Integrity

To handle 100k writes per second, the storage layer must scale horizontally, and reads must be separated from write operations.

Database Sharding (Horizontal Partitioning)

• Sharding Key: Shard the relational database (e.g., PostgreSQL or MySQL) using user_id (or tenant_id for B2B) as the shard key.
• Uniform Distribution: Consistent hashing ensures that write operations are evenly distributed across $N$ database instances (e.g., 10 shards handling 10k TPS each, which is well within RDBMS limits).
• Local Transactions: Keeping all data related to a single user's order within the same physical shard avoids slow, complex distributed transactions (2PC - Two-Phase Commit).

CQRS (Command Query Responsibility Segregation)

• Write Store (Command): Relational databases optimized for rapid single-row inserts (INSERT INTO orders ...). These databases do not handle complex search queries or read operations.
• Read Store (Query): Read-optimized data stores (e.g., Elasticsearch for order history search, Redis for order status caching).
• Data Sync (CDC): Use Change Data Capture (CDC) tools like Debezium to read transaction logs from the Sharded DB in real-time and stream updates to the Read Store via Kafka. This ensures low-latency eventual consistency without impacting write performance.

5. Transactional Integrity & Resilience

Operating asynchronously requires robust patterns to ensure data consistency across distributed systems.

Transactional Outbox Pattern

• To prevent inconsistency where an order is saved in the database but the event fail to publish to Kafka:

1. The Order Service writes both the Order record and an OutboxEvent record to the sharded database within the same local RDBMS transaction.
2. A CDC engine (Debezium) polls the Outbox table and safely publishes the events to Kafka.
3. This guarantees At-Least-Once delivery of events.

Saga Pattern for Distributed Transactions

• When an order spans multiple microservices (Order, Payment, Inventory, Shipping):
  • Use a Choreography-based Saga Pattern managed via Kafka events.
  • If the Payment Service fails, it emits a PaymentFailed event.
  • The Inventory Service listens to this event and executes a compensating transaction to restore stock in both Redis and the RDBMS.

Graceful Degradation & Failover

• Circuit Breakers: Implement Resilience4j on API calls to third-party payment gateways. If latency spikes, fail fast and queue the request for retry.
• Polite Notification: If the consumer lag on Kafka increases, the UI transitions to a polling state, notifying users: "Your order is in queue. We will notify you via push notification shortly." This maintains a seamless user experience under extreme loads.