Architecture Design - Database Partitioning vs. Sharding

CK CK 3788 
Created at  
150 0 0

Before drawing, you must establish the distinction between the two. When presenting to stakeholders or interviewers, define them as:

  • Database Partitioning (Horizontal/Vertical): Splitting a large table into smaller, more manageable pieces within a single database instance.
  • Data Sharding: A type of horizontal partitioning where data is distributed across multiple physical database instances (nodes).

 

 

How to Draw Database Partitioning (Single Instance)

This diagram focuses on how a single database engine logically divides data to improve query performance (e.g., partition pruning) and maintenance.

Architecture Design - Database Partitioning vs. Sharding

Key Elements to Draw & Explain:

  • Single DB Border: Draw a giant box representing one database server.
  • The Partition Key: Explicitly label the column used to partition (e.g., created_at for Range, region for List, or id for Hash).
  • Storage Files: Draw separate physical storage files (tablespaces) pointing to each partition under the hood.
  • Query Pruning: Draw an arrow from the query planner directly to only one partition to show that the system ignores the other partitions during a search.

 

 

How to Draw Data Sharding (Multi-Instance)

This diagram focuses on horizontal scalability (Scale-Out). It must include a routing layer that decides which physical database node holds the requested data.

Architecture Design - Database Partitioning vs. Sharding

Key Elements to Draw & Explain:

  • The Routing Layer (Routing Proxy): This is the most critical element. You must draw a component (like a load balancer, proxy, or application-level router) that hashes or maps the Shard Key (e.g., user_id) to the correct destination.
  • Independent Nodes: Draw distinct boxes representing completely separate physical servers or cloud VM instances. They do not share memory or disks (Shared-Nothing Architecture).
  • Shard Key Routing Logic: Draw how a request with user_id = 15500 bypasses Node 1 and Node 3 to hit Node 2 directly.

 

 

The Advanced Hybrid Architecture (Sharding + Partitioning)

In enterprise-level designs, Sharding and Partitioning are used together. First, data is sharded across multiple nodes, and then each node partitions its allocated data locally.

Architecture Design - Database Partitioning vs. Sharding

 

 

Technical Checklist to Discuss While Drawing

Architecture Design - Database Partitioning vs. Sharding

For Partitioning:

  • Index Strategy: Explain whether you will use Local Indexes (indexed per partition) or Global Indexes (one index spanning all partitions).
  • Data Lifecycle: Mention how easy it is to drop old data by simply dropping a partition (e.g., dropping Partition_2020) instead of running heavy DELETE queries.

For Sharding:

  • The Join Problem: Explain that cross-shard joins are highly inefficient. Thus, data that needs to be joined must share the same Shard Key so they reside on the same physical node (Co-location).
  • Rebalancing (Resharding): Explain how the architecture handles a shard becoming full (e.g., using consistent hashing to minimize data movement when adding Shard Node 4).
  • High Availability (HA): Draw Primary-Replica replication lines for each Shard Node to prove that a single shard failure won't cause total system downtime.
Tags Data Architecture Data Sharding Database Architecture Database Partitioning Database Scaling Horizontal Scaling Infrastructure Design Partitioning Scalable Systems Sharding System Architecture System Design Facebook X
Comments 0
Similar posts
  1. How to Design Redis Caching in Infrastructure Architecture
    154
  2. Architecting Dynamic Scale-Out and Scale-In with Docker Containers
    131
  3. Enterprise-Scale 100k+ TPS Order Processing System Architecture
    174
  4. Enterprise-Scale 100k+ TPS Order Tracking System Architecture
    198
  5. Designing Resilient Infrastructure to Prevent Order Data Loss During Kafka Failures
    151
  6. Stateful Databases: A Deep Dive with Cassandra & TiDB
    186
  7. System Design: High-Traffic Time-Deal Promotion System
    171
  8. Optimize the performance of a search results page that experiences high traffic
    139
  9. Ad Tech Idempotency: Engineering for Reliability and Scale
    357
  10. Balancing Real-Time Ad Stats and Billing Accuracy - Eventual vs Strong Consistency
    642
  11. Flink with Python: Core Concepts, Kafka Integration, and Architectural Trade-offs
    184
  12. Arm Kernel DNNs: Powering Efficient AI at the Edge
    221
  13. The Blueprint of Success: Why System Design and Architecture Matter?
    395
  14. Bridging the Gap: How Machine Learning Integrates with Infrastructure
    263
  15. System Requirements and Architecture: Intelligent Priority-Based Scheduler
    161
  16. Beyond Static Pipelines: Architecting LLM Multi-Agent Systems for Autonomous Workflow Orchestration
    576
  17. Designing a Skill Registry Architecture: Minimize Token Usage and Maximize AI Performance
    224
  18. Collecting Proof-of-Play from 100,000 Screens
    141
  19. The Two Clocks of Ad Server Architecture
    63
  1. Software Design Patterns Cheat Sheet
    3,315
  2. Various naming for the Software Architecture - I can't find Component-based Architecture, Functional Architecture
    3,325
  3. Builder Pattern - offering pre-defined Interfaces to set Individual Parts
    1,429
  4. Abstract Factory Pattern - Commonly Used for UI Themes or Objects
    1,465
  5. Software Design Patterns
    1,430
  6. Observer Pattern
    2,389
  7. Factory Method Pattern allows overriding to return different concrete object types
    1,524
  8. Optimize the performance of a search results page that experiences high traffic
    139
  9. System Design: High-Traffic Time-Deal Promotion System
    171
  10. Designing Resilient Infrastructure to Prevent Order Data Loss During Kafka Failures
    151
  11. Real-Time Inventory & WMS System Design
    240
  12. Enterprise-Scale 100k+ TPS Order Tracking System Architecture
    198
  13. Enterprise-Scale 100k+ TPS Order Processing System Architecture
    174
  14. Architecting Dynamic Scale-Out and Scale-In with Docker Containers
    131
  15. How to Design Redis Caching in Infrastructure Architecture
    154
Recently updated
  1. Why 100% SEO Guarantees Better User Engagement and How to Achieve It in Your Service
    27
  2. Why SPF and DKIM Are Essential for Linux Email Servers?
    24
  3. How do I create animated background for Google Meet?
    19,731
  4. MaxMind GeoLite2 vs. DB-IP Lite vs. ip-api.com: Choosing the Right Free IP Geolocation Solution
    217
  5. Meet Valkey: Why Redis Changed Its License, Why Valkey is Better, and How to Use It as a Drop-in Replacement
    86
  6. How Rogue Web Scrapers Are Stealing My Content and Destroying My Server
    153
  7. Array of arguments passed to script in PHP
    85
  8. Invalid Auction Storms in Ad Tech
    93
  9. Building MRC-Compliant Instrumentation in Ad Tech: Implementation Steps and Key Challenges
    198
  10. The Two Clocks of Ad Server Architecture
    63
  11. Frequency Cap Architecture Overview
    509
  12. Collecting Proof-of-Play from 100,000 Screens
    141
  13. Vector Search in Relational Databases (RDBs): Trends & Usage Environments
    280
  14. What is RedisVL?
    105
  15. Designing a Skill Registry Architecture: Minimize Token Usage and Maximize AI Performance
    224
  16. TPU & GPU Cluster Management Tools
    234
  17. The Concept of Scaffolding in AI Agents
    222
  18. Beyond Static Pipelines: Architecting LLM Multi-Agent Systems for Autonomous Workflow Orchestration
    576
  19. AI-DLC vs. AI-SDLC: Model-Centric vs. System-Centric AI Development
    253
  20. Fine-Tuning Open-Weight LLMs: Training, Licensing, and Deploying Custom Models with Ollama
    328
  21. Inside the Next-Generation Streaming Architecture: Ultra-Low Latency, SGAI, and AI Codecs
    197
  22. Machine Learning: Training vs. Ranking
    213
  23. The Continuous Learning Machine: Designing a Loop That Actually Turns
    217
  24. The Next-Gen Low-Latency Standards: LL-HLS & MoQ
    189
  25. Designing a Data Curation System Around a Vector Database
    215
  26. Extracting Context from Live Video: The Architecture Got Simpler When We Stopped Building Features
    229
  27. The New Ad Stack Behind Live Sports Streaming: Inside SGAI, EABN, and the Standards Race to Fix Live Monetization
    649
  28. Designing MLOps Architectures to Handle Seasonal Data Drift in EV Fleet Models
    185
  29. From 15 to 40+ FPS: Accelerating PyTorch Object Detection on Automotive ECUs
    207
  30. Understanding Data Lineage
    252