The problem
Most monitoring environments do not have a coverage problem. They have a noise problem. When an operations team receives hundreds of alerts a day, the brain stops treating any single alert as urgent. That is alert fatigue, and it is how real outages slip past well-staffed teams. In one enterprise SolarWinds Orion deployment I inherited, on-call engineers were acknowledging alerts faster than they could read them.
Why it happens
Alert fatigue is rarely caused by one bad rule. It accumulates:
- Static thresholds that were correct on day one and wrong six months later.
- Alerts that fire on transient spikes instead of sustained conditions.
- No distinction between a node being down and a single counter blipping.
- Every alert routed to the same inbox at the same severity.
The result is a stream where a failed backup and a 2-second CPU spike look identical.
Best practices that worked
These are the changes that produced the 30% reduction in actionable alerts:
- Tune for duration, not instants. Require a condition to persist (for example, CPU > 90% for 10 minutes) before an alert fires. Most transient spikes self-resolve.
- Use dependency-aware suppression. If a parent node is down, suppress the children. One router outage should not generate fifty alerts.
- Right-size thresholds per role. A database server and a print server should not share a memory threshold. Group nodes and apply role-specific limits.
- Add severity tiers. Page only on customer-impacting conditions; send everything else to a dashboard or daily digest.
- Review and retire. Schedule a monthly review of which alerts fired, which were acted on, and which were ignored. Delete the alerts nobody acts on.
A real-world example
The single biggest win came from dependency mapping. The environment had a recurring pattern where an upstream switch would briefly flap and generate alerts for every downstream node. By defining the network topology in Orion and enabling dependency suppression, a single flap went from ~40 alerts to one: the switch itself. Combined with duration-based thresholds and a severity split, the actionable alert volume dropped roughly 30% within two reporting cycles, and mean time to acknowledge real incidents improved because the queue was finally readable.
Conclusion
Fewer, better alerts beat more alerts every time. Tune for sustained conditions, model dependencies, assign real severities, and review monthly. The goal is not silence. It is signal. When every alert means something, your team starts trusting the queue again, and that trust is what catches the next outage.