Insights — Operations
An alert nobody reads protects nobody.
Every platform we take over already has monitoring. What is missing is not another tool, but a sort: what deserves waking somebody up, what can wait until the morning, and what nobody ever looks at.
01 — The problem
The number of alerts always grows faster than the ability to handle them.
A threshold added after every incident, one more probe with every migration: in two years, the same platform goes from twenty alerts a week to two hundred. At that level nobody reads any more, everybody filters — and the alert that matters drowns among those that never announced anything.
Useful monitoring is not measured by the number of metrics collected, but by the number of incidents detected before users notice. That is the only statistic that matters, and it can be managed.
01Why the wall of alerts always appears
An alert is almost always added in a hurry, right after an incident, by the person who has just been woken up. Nobody ever removes anything, because removing a threshold feels like removing protection.
The result is predictable: a system in which the value of an alert is never assessed. We apply the opposite rule — any alert that fires three times without consequence is revised or deleted, and that review is scheduled, not left to goodwill.
02The four questions that sort
Before adding or keeping an alert, four questions are enough. They take two minutes and save two years of noise.
- Does this stop a user from working?
- Can somebody act now, at this hour?
- Does the threshold come from a measurement, or from a hunch?
- Has this same alert already fired without consequence?
03What we actually watch
We watch little, but we watch signals that announce a failure before it produces an incident: disk space and how fast it is filling, backup success and the date of the last restore test, certificate expiry, queue depth, 95th-percentile latency and the server-side error rate.
These signals share one property: they do not depend on the load of the moment. They say where the system is going, not just where it is — which is exactly what a console never shows.
- Disk space and its filling trend
- Backups: success, age, last restore test
- Certificates and domain names: expiry date
- Queues and late jobs
- 95th-percentile latency and error rate
02 — Common questions
What we are asked about monitoring
Do we need to change tools for this?
Rarely. The platforms we deploy — Prometheus, Grafana, OpenTelemetry — are open source and more than sufficient. The useful work is not choosing the tool, but defining the thresholds and reviewing them periodically.
How long does it take to set up useful monitoring?
Two to three weeks to cover the structural signals of an existing platform, with thresholds set on real measurements rather than default values. The alert review itself never stops: it is part of operations.
What happens at night and at weekends?
That is the first question to settle, before the tool. On-call only makes sense if the alerts that trigger it are few and genuinely actionable. Everything else is handled the next day, with a log and a priority order.
Do you want to know what your monitoring does not see?
Monitoring review: we list the signals being watched, the silent alerts, the thresholds never revisited and the uncovered failure points, then hand over a plan ranked by urgency.