- Market
- Saudi Arabia
- Written for
- CTO, CIO
- Decision
- Bring log storage spend back into proportion without blinding investigations
- Languages
- EN · AR
The leaks
Four ways the same log line gets paid for more than once
A logging bill rarely has one dramatic cause. It has four quiet ones, and they multiply each other: the noisiest streams land in the priciest tier, kept the longest, in several copies.
| Variable | Effect on price | Why |
|---|---|---|
| Debug verbosity shipped to premium storage | The noisiest level of logging stored at the most expensive tier | Debug output exists so a developer can watch a system think, and it is written accordingly — many lines per request, most of them uninteresting the moment they are emitted. When that level ships to production and flows into the indexed search tier, the estate pays evidence prices for chatter. |
| Retention set once, on the first day | Every stream kept as long as the most sensitive one needs | Retention gets decided at setup — a default, or the longest period any obligation mentions — and applied to everything. Because logs only accumulate, the cost of that single early decision compounds every month, and nobody revisits it because nothing visibly breaks. |
| Three pipelines storing the same line | Agent, platform and SIEM each keep a full copy | The collection agent buffers and stores, the observability platform indexes, and the SIEM keeps its own copy for security. Each pipeline was added for a good reason; nobody ever decided that one line should be billed three times, at three different rates — it is simply what the defaults add up to. |
| Everything indexed as if about to be searched | Hot, searchable storage as the default landing zone | Indexing is the most expensive thing you can do to a log line, and it buys one capability: instant search. Most lines will never be part of any search, yet the pipeline indexes them all, because the default destination is the tool rather than a tier chosen per stream. |
Ask any supplier quoting this
- Which of our log streams has actually been queried in the last quarter, and what does each cost to keep?
- For every copy of a log line we store, which investigation is that copy there to serve?
The principle
Storage tiers should follow investigation value, not tool defaults
The question that prices a log line correctly is: who will read this, and in what situation? Security events and errors get read under pressure, by someone who needs search in seconds — they belong in the hot, indexed tier and nowhere cheaper. Routine application chatter gets read rarely, during a deep investigation that can wait an hour — it belongs in inexpensive object storage, rehydrated on demand. Debug output in production gets read almost never — it belongs in a short-lived window, or switched off until someone is actually watching. Tiering by that question, instead of by whichever tool collected the line, is where most of a runaway logging bill goes to die.
One boundary matters here. Deciding which events must be collected in the first place — what detection and audit require — is design work, and it is a different engagement from cost work. A cost pass must never quietly delete what an investigation depends on, which is why the tiering decisions are made with the people who run detection in the room, not around them. The spend question and the coverage question share a table; they do not share an answer.
The method
Working the log bill down without blinding an investigation
The order protects the investigations: nothing is dropped or sampled until every stream is named, owned and priced, and the people who read logs under pressure have agreed to each change.
Inventory what ships and who reads it
Every stream is mapped from emitter to each destination it lands in, with its volume, its cost per copy, and the last time anyone queried it. Most estates discover streams nobody can name a reader for. Every stream named, priced and owned.
Collapse the duplicate copies
For each stream, one system is chosen as the system of record for each purpose — security in the SIEM, debugging in the platform — and the other pipelines forward without storing, or stop collecting it entirely. One deliberate copy per purpose, not one per tool.
Tier and set retention per stream
Retention stops being one global number: each stream keeps what its investigation value and its actual obligations require, in the tier that matches how it gets read — hot for the urgent, object storage for the archival. Retention someone can defend, stream by stream.
Sample the flood, and write down what is lost
High-volume, low-value streams are sampled — with the loss stated honestly: aggregates and trends survive, the ability to reconstruct any single routine request does not. Errors and security events are never sampled. A sampling policy with its blind spots stated.
Our role
What we retune in the pipeline and what your responders keep
- 01
The bill, decomposed first
We break the observability spend into streams, copies and tiers before proposing a single change, so every saving on the table has a named source and a named consequence.
- 02
Changes agreed with the readers
Every tiering, retention and sampling decision is made with the engineers and security analysts who read these logs under pressure, because they are the ones a wrong cut would blind.
- 03
Guardrails against the drift back
We leave volume budgets and alerts per stream, so a new debug flag or a chatty release announces itself in days rather than surfacing as a surprise on next quarter’s invoice.
What an investigation must be able to see is a standard your responders set; we make the pipeline meet it at the price of what it keeps, not the price of keeping everything.
Common questions
What buyers ask
How can storing logs cost more than running the application that writes them?+
Because the two are priced on different axes. Compute is priced by capacity, which you control directly and see plainly. Logs are priced by volume — ingested, indexed, stored, often in several copies — and volume grows with traffic, verbosity and dependency count rather than with any decision anyone reviews. A single debug flag left on, or one new chatty service, can multiply log volume overnight while the compute line does not move. And because the observability bill is filed under necessary, it escapes the scrutiny the compute line gets. The imbalance is not a paradox; it is what happens when one line item has owners and the other has defaults.
Is sampling logs safe, and what do we actually give up?+
Safe for the right streams, done deliberately. The candidates are high-volume, routine-success streams — the access line that says a request completed normally, repeated millions of times. Sampling those preserves what they are actually used for: rates, trends, aggregates. What you give up is reconstruction — the ability to find the one specific routine request from last Tuesday, because it may be in the discarded share. That is why errors and security-relevant events are never sampled: those are exactly the lines an investigation needs individually. The discipline is to write the trade down — which streams, what rate, what becomes invisible — and have the people who run investigations sign it, so the blind spot is a decision rather than a discovery.
Do the observability platform and the SIEM both need full copies of every log?+
Almost never, and the full duplication is usually a default rather than a design. The two systems answer different questions: the SIEM needs the security-relevant streams — authentication, access changes, network events — shaped and retained for detection and audit; the observability platform needs the application streams engineers debug with. The overlap where both genuinely need the same stream is real but narrow. The productive exercise is to decide, stream by stream, which system is the system of record for which purpose, and to let the other forward or drop. Which streams the SIEM must have is a design-side question with its own discipline — the cost work implements that answer, it does not overrule it.
In this cluster
Review your log spend with us
A log-spend review that names every stream, prices each of its copies, and says which storage tier its investigation value actually justifies.
Start the conversation
