Capacity planning for observability data
Work from what you know to a storage plan in five steps. You can skip any step: each tool starts from sensible defaults and shows its math.
Capacity planning workflow
Measure
Know your events per second and average event size, or your daily GB? Convert to a daily ingest. EPS × average event bytes × 86,400 gives GB/day.
Retain
Decide how many days stay searchable, how many can sit in cheaper storage and how many only need to be archived. Log Storage applies one retention window; the platform tools split it into tiers.
Choose a platform
Each tool starts from the daily ingest and retention you already have, and uses its own documented defaults for everything else.
Refine
Replication, tiering, shards, object storage, cache, growth and headroom. Expert mode in each tool exposes the ratios; shard planning is in Elasticsearch and OpenSearch.
Compareoptional
Compare two scenarios, save them in your browser, open a printable report or share a link. None of this is required to get a result.
How the inputs relate
Every model starts from the same base: daily ingest × retention days is the volume you must keep. Three things then change it:
- On-disk factor: data rarely lands on disk at its raw size. Compression shrinks it; indexing structures and mappings can grow it.
- Copies: replication factors, search factors and replicas multiply some or all of that volume for resilience and search availability.
- Headroom: a growth buffer for next year's ingest, plus free disk the platform needs to keep operating.
Shards are a count, not a size: they split the same data into units the cluster can place and recover, so they shape node layout more than total disk.
Why each platform needs its own model
Generic log storage
Use it for a platform-neutral estimate when you only know GB/day: raw volume × replication × growth, with no compression assumed.
Splunk
Use it for Splunk Enterprise indexer clusters: rawdata is kept on every copy (× RF) but index files only on searchable copies (× SF), and SmartStore moves warm data to object storage.
Elasticsearch
Use it when data moves through hot, warm, cold and frozen tiers: replicas apply per tier, searchable snapshots change what stays on local disk, and disk is provisioned with a utilization target.
Grafana Loki
Use it when logs live as chunks in object storage: one stored copy, a replication factor that affects ingesters and their WAL, and a published compute reference instead of a derived node count.
OpenSearch
Use it for Amazon OpenSearch Service (AWS sizing-guide overhead, UltraWarm and cold in S3) or self-managed OpenSearch. It is not the Elasticsearch model with a different name.
All five are planning estimates calculated in your browser. Each tool shows its formulas, assumptions and sources so you can check the result against your own data.
The capacity pipeline, with numbers
Every model walks the same six steps. What differs is the factor each platform applies at each step. Here they are for 250 GB/day kept 90 days, using each calculator's defaults.
Reference scenario inputs chosen for comparison, not a recommended size.
- Ingest
- Retention
- On-disk factor
- Copies / replicas
- Growth / headroom
- Provisioned capacity
| Step | Generic | Splunk RF2/SF2 | Elasticsearch, 1 replica |
|---|---|---|---|
| Ingest × retention | 22.5 TB | 22.5 TB | 22.5 TB |
| On-disk factor (one copy) | × 1.0 → 22.5 TB | × 0.5 (0.15 + 0.35) → 11.3 TB | × 1.0 → 22.5 TB |
| Copies | × 2 → 45 TB | rawdata × RF2 + index × SF2 → 22.5 TB | primary + 1 replica → 45 TB |
| Growth 25% | 56.3 TB | 28.1 TB | 56.3 TB |
| Disk headroom | not modeled | not modeled | ÷ 85% → 66.2 TB |
| Provisioned capacity | 56.3 TB | 28.1 TB | 66.2 TB |
Ingest is the daily volume you must keep, measured on real traffic. Retention multiplies it into the dataset. The on-disk factor turns raw bytes into stored bytes. Copies add resilience and searchability. Growth reserves room for next year's ingest, and headroom keeps disks below the level where the platform stops working well.
Same input, different architectures
Reference scenario 250 GB/day for 90 days with two copies and a 25% growth buffer gives different answers, and none of them is wrong. They are answers to different storage models, and not every number measures the same thing.
Raw volume × 2 copies × 25% growth. No compression assumed.
About 15% rawdata × RF plus 35% index files × SF: half the raw size per full copy.
Factor 1.0 × 2 copies × 25% growth, then ÷ 85% planning disk utilization.
Object storage dataset, one copy, 25% growth, at a conservative stored ÷ received planning assumption of 1.0× (measure yours). The replication factor changes ingester WAL, not this number.
Hot storage with one replica, 1.1 index factor, 25% growth, then the AWS 5% + 20% overhead (about ×1.45 of source).
Loki’s figure is object storage and the others are local disk, so it should not be read as “cheaper”. OpenSearch includes service overhead that the generic model does not.
- Splunk is the smallest because its documented ballpark stores about half the raw size per copy, and index files are only kept on searchable copies.
- Elasticsearch is the largest because the default factor assumes no compression and the result is provisioned disk: 56.3 TB of data ÷ 0.85 = 66.2 TB. With every tier regular, factor 1.0 and one replica, that is the generic result divided by 0.85: 18% more.
- The generic model is the neutral baseline: no compression, every copy complete, no disk headroom. Use it when you do not yet know the platform.
Compare the models
| Model | Main multiplier | Replication model | Headroom | Storage location |
|---|---|---|---|---|
| Generic | Raw GB/day × days (factor 1.0) | Whole dataset × replication (1–3) | Growth % after replication | One local pool |
| Splunk | Rawdata 0.15 and index files 0.35 of pre-indexed volume | Rawdata × RF, index files × SF | Growth % after copies | Indexer disk; with SmartStore, cache + hot locally and one remote copy of warm buckets |
| Elasticsearch | Raw GB/day × index/raw factor (default 1.0) | Primaries + replicas on regular tiers; none on searchable snapshots | Growth %, then ÷ planning utilization (85%) | Hot/warm/cold node disk; frozen cache locally; snapshot repository for searchable snapshots |
| Grafana Loki | Raw GB/day × stored ÷ received (conservative planning assumption 1.0×, editable) | One object-store copy; replication factor applies to ingesters and WAL | Growth % on object storage; WAL volume kept under the 90% disk-full threshold | Object storage for chunks and index; local disk for the WAL; compute as a published reference profile |
| OpenSearch | Source GB/day × index factor (default 1.1) | Primaries + replicas on hot; UltraWarm and cold primary-only (Amazon OpenSearch Service) | AWS: ÷ 0.95 ÷ 0.80; self-managed: ÷ planning utilization target | Hot node storage; UltraWarm and cold in S3 (Amazon OpenSearch Service only) |
Before you size
Most sizing errors come from inputs, not formulas. Check these first:
- Real ingest. Measured daily volume, not the license entitlement or a vendor quote. In a hypothetical 500 GB/day license with 320 GB/day of real traffic, sizing from the license oversizes Splunk storage by 56%.
- Peak vs average. Size storage from daily totals over a full week; size indexing throughput from peaks.
- Retention by purpose. Search and alerting windows, investigation windows and compliance retention often differ; each can be a different tier.
- Disk expansion or compression. A measured on-disk factor beats any default. A hypothetical factor of 1.3 instead of 1.0 changes the reference scenario from 66.2 TB to 86 TB.
- Replicas and copies. Which copies hold full data and which do not (Splunk SF, searchable snapshots).
- Recovery objectives. How many node failures you must survive without losing data or search, and how long recovery may take.
- Tiering. Which days need fast local disk and which can move to cheaper nodes or object storage.
- Object store. SmartStore and searchable snapshots move most retained data to object storage, which needs its own capacity budget.
- Expected growth. New sources and teams usually add more volume than organic growth; agree on a buffer per year.
Go deeper
Log retention & storage planning
Vendor-neutral method: daily volume from event rate, measured storage factor, retention tiers.
Splunk SmartStore sizing
Cache vs retention, hot-bucket replicas, object store sizing and when SmartStore pays off.
Elasticsearch data tiers sizing
Hot, warm, cold and frozen: local disk vs repository, replicas and the utilization target.
Grafana Loki sizing
Object storage, WAL, replication, retention and cache; what the published sizing profiles do and do not cover.
OpenSearch sizing
Ingest, shards, replicas, storage tiers and disk planning, and Amazon OpenSearch Service versus self-managed.
How ObservKit builds and verifies these models: methodology and changelog.
Last reviewed: