Log retention & storage planning

The method

daily GB   = events/sec × average event bytes × 86,400 ÷ 10⁹
dataset    = daily GB × retention days
stored     = dataset × storage factor
copies     = stored × replication
capacity   = copies × (1 + growth), then platform headroom

Illustrative example a team ships an average of 5,000 events/s of 600-byte events, keeps them a year, searches the last two weeks often and older data rarely.

1. Daily volume

5,000 × 600 × 86,400 = 259,200,000,000 bytes = 259.2 GB/day. Use the average rate over full days and at least one full week. A hypothetical one-hour peak of 12,000 events/s extrapolated to the day would give 622.1 GB/day — 2.4 times too much storage. Peaks size ingestion throughput; daily totals size storage.

If you already have a platform, its own accounting is better than an estimate: daily license usage in Splunk, or ingested bytes from your shippers.

2. Retention by purpose

One number for "retention" usually hides three: how far back alerts and dashboards look (days), how far back investigations go (weeks to months), and what policy or regulation requires (months to years). Write them down separately; they map to tiers in step 6. In the example: 14 days of frequent search, 365 days total.

Count retention in days. Twelve 30-day months are 360 days, five or six short of a calendar year, which matters at hundreds of GB per day.

3. Storage factor

Stored size is rarely raw size. Measure stored bytes for one copy ÷ raw bytes ingested over at least seven days on data that is no longer being written. Splunk documents a ballpark of about 0.5 (15% rawdata + 35% index files) for syslog; Elasticsearch varies with mappings, multi-fields, _source, codecs and LogsDB, and can be above or below 1.0. Until you measure, the neutral assumption is 1.0, which is what the generic calculator uses.

4. Copies

Copies exist for two different reasons: surviving node loss (data availability) and keeping data searchable during that loss (search availability). Platforms separate them: Splunk with RF and SF, Elasticsearch with replicas on regular tiers and none on searchable snapshots. A single archive copy in object storage relies on the object store's own durability.

With replication 2 kept all year, the example needs 236.5 TB including 25% growth. Open this scenario in the Log Storage Calculator.

5. Growth and headroom

They are different things. Growth is next year's ingest: new services, teams and verbosity; ObservKit applies it as a percentage after copies. Headroom is free disk the platform needs to keep running: Elasticsearch, for example, stops allocating shards to a node above its 85% low watermark, so the Elastic calculator divides by a planning utilization target. Adding 15% is not the same as dividing by 0.85: dividing adds 17.6%.

6. Tiers

Map the retention purposes to storage: 14 days of frequent search on replicated fast disk, the rest as a single archive copy.

The archive line is still uncompressed raw data; object storage with compression would be smaller, and search over it is slower or needs a restore.

Platform models

Once the platform is chosen, its own model replaces the generic factors:

• Splunk SmartStore, same 259.2 GB/day for 365 days, RF2/SF2, 30-day cache: 5.02 TB of local indexer storage and 59 TB in the object store. Open this scenario in Splunk.
• Elasticsearch, 7 hot, 7 warm and 351 frozen days, one replica: 23.3 TB of provisioned node disk and 113.7 TB in the snapshot repository. Open this scenario in Elastic.
• Grafana Loki, same 259.2 GB/day for 365 days at a conservative stored ÷ received planning assumption of 1.0×: 118.3 TB of object storage, one copy. Open this scenario in Loki.
• OpenSearch on Amazon OpenSearch Service, all 365 days hot with one replica: 342.3 TB including AWS overhead. Open this scenario in OpenSearch.

For the trade-offs behind those layouts, see the Loki guide, the OpenSearch guide, the SmartStore guide and the Elasticsearch data tiers guide.

Common mistakes

1. Extrapolating peak event rate to the day: 622.1 instead of 259.2 GB/day in the example.
2. Sizing from license entitlement instead of measured volume.
3. Assuming a compression ratio from another platform or another data source.
4. Replicating the archive when the object store already provides durability.
5. Mixing up growth and headroom, and applying neither.
6. One retention for everything, which puts a year of rarely searched data on the most expensive disk.

What this method does not cover

• Ingestion throughput. Storage follows daily totals; indexers, ingest nodes and pipelines must be sized for peak event rates separately.
• Search load. Concurrency, dashboards and scheduled searches drive CPU and memory, not disk.
• Compression in the archive tier and object storage pricing, request costs and retrieval times.
• Platform overhead: operating system, filesystem reserve, temporary space for merges, recovery and snapshots.
• Data that is not logs: metrics and traces have different size and retention profiles and usually different storage engines.

Each of these can matter more than storage for the final design; the method gives the capacity input to that design, not the design itself.

Last reviewed: October 2026 · Log Storage Calculator · Capacity Planning · Methodology