If you’re part of a team that’s involved with increasing or monitoring system, component or infrastructure performance, you already know about the “alert flood.” It’s that constant deluge of emails to notify you that the systems you or your team is responsible for has an issue. And thankfully, we’re working on decreasing that flood in our next release of EDM.
The flood part is because there are so many alerts. It’s a problem that we often hear from teams. To solve it, many choose to simply filter their reports, choose to not receive them, delete them or change their monitoring thresholds to receive less alerts.
Yet, those alerts may be part of a company’s SLAs or other agreements. When the important alerts happen, IT staff might simply miss the message. These alerts are important and in cases where there’s an internal review, it might come down to who was alerted and who was required to react. The results of that review might not be good for individuals who were required to react.
One idea to minimize that problem might be to create an Alerts Team to not only manage alerts but also set the rules on what’s monitored, develop new thresholds based on accumulated historical data and use industry or company best practices to minimize the flood to something more akin to a kiddie pool. Plus, as components are added, the Alerts Team can set the rules based on their expectations and not simply rely on the monitoring solution’s default settings.
We’ll agree that alerts can be a necessary evil. Yet, when one user changes the thresholds in order to minimize their alerts — it might be a mandated threshold alert for another staff member. For companies with a wide variety of operating systems, storage solutions and other devices, team members should reflect experts with knowledge of each system. For instance, few would want a Windows expert to set operating thresholds for a Linux server.
The concept is simple, with expertly-set thresholds the Alerts Team can keep alerts to a minimum and deliver the right message to the right person. For larger organizations, the establishment of an Alerts Team can help experts understand their role in the overall infrastructure and limit the amount of alerts going to the teams.