Capacity Planning for Sharded Clusters

In this lesson, you will learn how to mathematically forecast storage and throughput requirements to determine the optimal number of nodes for your sharded cluster.

Capacity planning is the process of predicting the future resource demands of your database before you hit performance bottlenecks. Because sharded systems distribute data across multiple nodes, you must calculate requirements not just for the total dataset, but for individual shards to ensure no single node becomes a point of failure or a performance bottleneck.

The Planning Formula

To determine the number of nodes ($N$), you must calculate requirements based on your peak anticipated load for both storage and throughput.

1. Storage Capacity: Calculate the total projected data size ($S$) including indexes and replication overhead. Define the maximum storage capacity per node ($C_s$).
   • Minimum nodes for storage: $N_s = \lceil S / C_s \rceil$
2. Throughput Capacity: Determine the peak Operations Per Second ($OPS$) for writes and reads. Define the maximum $OPS$ a single node can handle without exceeding latency SLAs ($C_t$).
   • Minimum nodes for throughput: $N_t = \lceil OPS / C_t \rceil$

The final node count is $N = \max(N_s, N_t)$, plus a "headroom" factor (typically 20–30%) to handle unexpected traffic spikes or background operations like rebalancing and resharding.

Example Scenario

Imagine you expect your database to grow to 10TB of data within 18 months, with a peak write throughput of 50,000 writes per second.

• Your chosen hardware supports 2TB of effective storage per node and can reliably handle 10,000 writes per second.

-- Storage nodes required: 10TB / 2TB = 5 nodes
-- Throughput nodes required: 50,000 OPS / 10,000 OPS per node = 5 nodes
-- Total needed: 5 nodes + 30% headroom = 6.5 nodes -> 7 nodes

When you define your shard_key, remember that poor distribution (hotspots) will effectively reduce your per-node capacity ($C_t$), forcing you to over-provision nodes to compensate for the imbalance. Always plan for the worst-case shard density, not the average.

Key Takeaway: Effective capacity planning requires balancing storage volume and peak throughput against individual node limits, then adding a strategic buffer to accommodate hotspots and future scaling operations.