Logging, metrics, and tracing in Kubernetes are priced on volume ingested and duration retained, and both grow with every pod, replica, and debug statement. Left unmanaged the observability bill can reach the same order of magnitude as the cluster compute, because a chatty application at scale emits gigabytes of logs per day and high cardinality metrics multiply with every new label. The way to cut it without going blind is to treat observability data as tiered: keep high value, low volume signals hot and queryable, sample or drop high volume low value noise, and move anything kept for compliance to cheap cold storage. Verbosity left in debug mode is the most common and most expensive single fault.
Here is where the cost comes from across logs, metrics, and traces, and the specific controls that reduce each.
Why does observability scale faster than the workload?
Three multipliers drive it. Log volume scales with request count and log level, so a service left at debug emits many times the lines it needs. Metric cost scales with cardinality, the number of unique label combinations, so adding a high cardinality label such as user id or request id can multiply a single metric into millions of series. Trace cost scales with sampling rate, and tracing every request at full fidelity is rarely necessary. Because all three are billed on ingest and retention, the bill grows with traffic and with every well meaning addition of a new label or log line, often faster than the compute it observes.
How do you cut log cost without losing signal?
- Fix log levels. Default applications to info or warn in production and leave debug for targeted investigation. A pipeline accidentally left in verbose mode is a recurring and avoidable spike.
- Sample high volume, low value logs. Access logs and health check chatter can be sampled heavily; error and audit logs should not be sampled at all.
- Tier retention. Keep recent logs hot and searchable for days, then move older logs to cheap object storage for the weeks or years compliance requires. Retention at full searchable tier is where most log spend hides.
- Drop at the source. Filter out known noise in the collector before it is ingested, so you never pay to store and index data nobody queries.
How do you control metric cardinality and trace volume?
Metrics: the biggest lever is cardinality. Strip unbounded labels such as request id, user id, and full path from metrics; those belong in traces or logs, not in time series. Set a cardinality budget per metric and alert when a new label explodes the series count. Aggregate at the source where you can.
Traces: sample. Head based sampling keeps a fixed fraction of traces; tail based sampling keeps the interesting ones, errors and slow requests, while dropping the routine. For most services a low single digit percent sample rate preserves the diagnostic value at a fraction of the ingest cost. Keep full fidelity only for the few services where every transaction matters.
Does the managed versus self hosted choice change the math?
Yes, and it is worth pricing both. Managed observability on AWS, Azure, GCP, and OCI bills on ingest and retention with little operational overhead, which is convenient but makes verbosity expensive. Self hosting an open collector stack shifts cost to the compute and storage you run, which can be cheaper at high volume but adds engineering load. The decision should follow your volume and team capacity, not habit. Whichever you choose, the sampling and retention discipline above applies, because both models bill on the data you keep.
A worked example
A scaling fintech found its Kubernetes observability stack had grown to nearly the cost of the cluster compute it monitored. Two faults dominated. A core service had shipped to production with debug logging left on, emitting many times the necessary volume, and a dashboard team had added a request id label to a high traffic metric, exploding it into millions of series. We reset log levels to info with error logs unsampled, stripped the unbounded label and set a cardinality budget, and moved compliance logs to cold object storage after seven days hot. Observability ingest fell sharply with no loss of the signals on call engineers actually used. The reclaimed spend fed the wider program that left the company materially lighter on cloud cost. Figures are verified against billing data and anonymised.
Frequently asked questions
Why is observability so expensive in Kubernetes?
How do you cut logging cost without losing signal?
What is metric cardinality and why does it matter?
Talk this through with us
We help platform teams bring Kubernetes logging, metrics, and tracing cost back in line across AWS, Azure, GCP, and OCI while keeping the signal on call teams rely on. We take zero provider commissions and answer only to you. Our guarantee: we reduce your cloud spend or we reimburse our service fee, on either a Fixed Fee scoped up front or a no risk Gainshare basis. Book a strategy call to scope it for your estate, and follow more analysis in The Cloud Spend Navigator.
Put a defensible number on your cloud spend.
No provider in the room, no published price list. Tell us your footprint and we will scope the savings against your billing data — we reduce your cloud spend or we reimburse our service fee.
The Cloud Spend Navigator: what changed in cloud pricing, commitments, and FinOps — no vendor spin.