Elasticsearch Sizing Calculator

Worked example: the reference scenario

Reference scenario the calculator's default inputs: 250 GB/day of raw data, 90 days retention split 7 hot / 23 warm / 60 cold (cold as regular indices), 1 replica, index/raw factor 1.0, 25% growth, and an 85% planning disk utilization target. Open this scenario.

Headroom is applied by division: to keep usage at or below 85%, provisioned disk is usable ÷ 0.85, which adds about 17.6%, not 15%. With every tier regular, factor 1.0 and one replica, the result equals the generic log storage estimate divided by 0.85.

Data tier comparison

Cold regular is not cold searchable snapshot. A regular cold index keeps primaries and replicas on local disk like hot and warm. A fully mounted searchable snapshot keeps one local copy and relies on the snapshot repository for resilience. In the reference scenario, switching the 60 cold days to searchable snapshots lowers provisioned disk from 66.2 TB to 44.1 TB and adds 18.8 TB in the repository. Searchable snapshots require an Enterprise subscription on self-managed clusters.

Frozen is not a full local dataset. Partially mounted indices keep only a shared cache locally; every search that misses the cache reads from the repository. The 10% cache ratio ObservKit uses is a planning assumption, not an Elastic recommendation — size it from the share of frozen data you actually search. The data tier sizing guide covers when each tier is worth it.

Reference scenarios

Reference scenario Each row is computed by the engine above and opens with its inputs filled in; they are comparison points, not recommended layouts. One replica on regular tiers, factor 1.0, 25% growth, 85% planning utilization.

Keeping a year with 335 days frozen needs 33.7 TB of local disk and 104.7 TB in the repository. The same year on regular cold indices with one replica needs 268.4 TB of local disk.

Shard planning example

Elastic's shard sizing guidance is to aim for shards between 10 GB and 50 GB and to keep each shard below 200 million documents; for time-series data it suggests rollover with max_primary_shard_size at 50 GB. The planner divides the indexed primary data in each rollover window by a target shard size and rounds up.

Each replica adds the same number of replica shards. Elasticsearch limits non-frozen shards to 1,000 per node by default, so shard count, not only disk, can set a minimum node count. Open the 30 GB target · Open the 50 GB target.

Common mistakes

1. Adding replicas to searchable snapshots. Fully mounted cold indices have no replicas by default. Counting a replica for the 60 cold days gives 44.1 TB of cold disk instead of 22.1 TB.
2. Adding replicas, or full copies, to frozen. Treating 335 frozen days as regular cold data with a replica asks for 246.3 TB of local disk; partially mounted frozen data needs a 11.6 TB local cache in this model.
3. Planning to 100% disk. The reference scenario needs 56.3 TB of data with growth. Provisioning exactly that leaves every node above the 85% low watermark, where Elasticsearch stops allocating shards to it (primaries of newly created indices excepted); at 90% (high) it tries to move shards away, and at 95% (flood stage) indices with a shard on that node are made read-only. Planning at 85% gives 66.2 TB.
4. Reading the 85% target as allocator math. Watermarks are evaluated per node, not as a cluster average: one full node blocks allocation even when the cluster is at 70%. On large disks the default low.max_headroom of 200 GB means the low watermark is reached when 200 GB remain free — about 98% used on a 10 TB disk.
5. Assuming index/raw factor = 1. Mappings, multi-fields, _source, codecs and LogsDB change on-disk size. Hypothetical example With a factor of 1.3 the reference scenario needs 86 TB; with 0.7, 46.3 TB. Measure it as pri.store.size of rolled-over indices divided by the raw bytes shipped for them.
6. Too many small shards. Illustrative example At 2 GB/day with daily rollover and 90 days retention, each data stream accumulates 90 primary shards of about 2 GB. Rolling over every 15 days gives 6 shards of about 30 GB for the same data. Open the 15-day rollover scenario.
7. Budgeting the snapshot repository as local disk. In the frozen scenario 104.7 TB lives in object storage and 33.7 TB on nodes. Buying node disk for the repository total overbuys; forgetting the repository underbuys object storage.

How Elastic sizing is calculated

indexed/day   = raw GB/day × index/raw factor
tier dataset  = indexed/day × tier days
regular tier  local = dataset × (1 + replicas)
cold snapshot local = dataset × 1          repository = dataset
frozen        local = dataset × cache %    repository = dataset
usable        = local × (1 + growth)
provisioned   = usable ÷ planning utilization (85%)      hot, warm, cold
              = usable ÷ 0.90 shared-cache share        frozen
shards/rollover = ceil(indexed/day × rollover days ÷ target shard size)

• Planning utilization (85%): an aggregate planning assumption inspired by, not equal to, the per-node disk watermarks.
• Index/raw factor: defaults to 1.0 because there is no universal ratio; set it from measurement.
• Replicas: applied to regular indices only.
• Searchable snapshots: no replicas; the repository holds the full dataset.
• Frozen cache: 10% of the frozen dataset by default (ObservKit assumption); frozen nodes dedicate 90% of disk to the shared cache by default.
• Shards: a count derived from size and rollover, an orientation rather than a production recommendation.
• Not modeled: node counts and RAM:disk ratios, regular snapshot backups of hot/warm data, OS and filesystem overhead, merge and translog peaks.

The common model behind all calculators is described in the methodology.