Log Storage Calculator

Worked example: 500 GB/day for 90 days

Illustrative example A platform-neutral estimate: 500 GB/day, 90 days, replication 2, 25% growth, no compression assumed. Open this scenario.

The formula is daily × retention × replication × (1 + growth). Because no compression is applied, this is an upper bound for platforms that compress (Splunk's documented ballpark keeps about half of the raw volume per copy) and a fair estimate for systems that store data close to raw size.

Reference scenarios

Reference scenario Three sizes to compare against, not recommendations.

Retention and replication multiply each other: the large environment ingests 100 times the small one per day but needs 900 TB instead of 750 GB, because it keeps six times longer and stores two copies.

From events per second to GB/day

If you only know event rate, convert it to daily volume with the average raw event size:

bytes/day = events/sec × average event bytes × 86,400
GB/day    = bytes/day ÷ 1,000,000,000

Illustrative example 5,000 events/s × 600 bytes × 86,400 s = 259,200,000,000 bytes = 259.2 GB/day. Kept 30 days with replication 2 and 25% growth that is 19.4 TB. Open this scenario.

• Use the average rate over a full day, not the peak. Hypothetical example If the same source peaks at 12,000 events/s for an hour, 12,000 × 600 × 86,400 gives 622.1 GB/day — 2.4 times the real volume. Size throughput for peaks; size storage for daily totals.
• Measure event size on the raw wire format, before parsing or enrichment. JSON with many fields and multi-line stack traces can be several times larger than a syslog line.
• Include a weekly cycle. Weekday and weekend volume often differ; measure at least seven days.

Measure your real storage factor

The generic calculator assumes stored bytes equal raw bytes. Your platform's real factor is:

storage factor = stored bytes for one copy ÷ raw bytes ingested for the same data

1. Pick a window of at least seven days so weekly patterns are included.
2. Take raw volume from the platform's own ingest accounting or the shipper's byte counters for that window.
3. Take on-disk size for one copy of the same data once it is no longer being written, after background merges have settled.
4. Divide, and apply replicas separately.

The factor varies widely. In Splunk it is the rawdata plus index-file ratio, which Splunk documents as about 0.5 for syslog and which changes by source (| dbinspect reports bucket sizes). In Elasticsearch mappings, multi-fields, _source, codecs and LogsDB index mode can push it well below or above 1.0 (compare pri.store.size with raw bytes). Highly repetitive text compresses well; high-cardinality structured fields compress less. Use the platform calculators, which accept a measured factor, once you have it.

Retention tiers

Not all retained data needs the same storage. A common split:

• Hot / active: recent data that is searched and alerted on; fast disk, replicated.
• Warm: older data searched occasionally; cheaper disk, often still replicated.
• Archive / object store: kept for compliance or rare investigations; one copy in object storage, slower to search or requiring a restore.

Size each tier as its own retention window. Illustrative example For 500 GB/day: 7 hot days with replication 2 need 8.75 TB (open), and the remaining 83 days as a single archive copy need 51.9 TB (open), compared with 112.5 TB when all 90 days are replicated. Archive compression and object-storage pricing are not modeled here.

Platform tiers change the math further: Splunk separates rawdata and index files and offers SmartStore; Elasticsearch sizes hot, warm, cold and frozen tiers with searchable snapshots. The log retention and storage planning guide walks through the full method.