Overwhelmed with False Alarms (Part 5)
This article is part of a 5 part series covering some of ways to deal with the top challenges for IT Operations and how machine learning techniques can be applied to address them.
- Having Trouble Finding The Root Cause
- Stuck In Reactive Firefighting Mode?
- Not Sure What is Important
- Can not See the Forest for the Trees
- Overwhelmed with false alarms?
Part 5 of a 5 part series
- - - -
What can be done about all the false alarms? In many cases, lots of alarms are simply noise. They are repeated alerts, -alerts that are not interesting. They are alerts that are known not be interesting where nothing ever breaks and nobody cares about them. Ideally the system should take into account the operator’s feedback in the same way for how an email inbox deals with spam - once it is marked as spam the system learns and stops showing those emails.
The reinforcement learning approach can be applied to address this issue. Reinforcement learning is built on the premise that there is an agent (the learning system) that introduces an action into an environment (the operator behind the computer screen). This action means to show or dismiss an alert.
When the system decides to show an alert it will look for the feedback from this environment. The operator can decide whether this was actually a good alert or that it was just complete noise.
The learning system takes this feedback information and looks for two items:
- The reward which could be positive or negative. This is basically the feedback given by the operator suppressing or escalating the alert
- The state associated with this reward.
For instance, if the alert is noise, then the operator behind the computer indicates this as noise and it will be reported to the learning system as a negative reward (or penalty). In the state that shows a repeated alert, the system adapt its internal knowledge to realize that the user does not like to see the repeated alerts and decrease the probability of showing these alerts in the future. Operators train the system and actually indicate what is important to them and what they would like to be alerted on.