Automated anomaly detection system comes into play when manual one fails

No wonder, anomaly detection is critical and is gaining importance with every passing day. On the flipside, manual anomaly detection can't scale to a large number of metrics. This is exactly where an automated anomaly detection system comes into play. End of the day, we all know that one of the best ways to monitor metrics is employing automation. Though a large number of metrics can be addressed to, what is worth noting here is – how many metrics to cater to? The reason why this is important because there are a series of other factors as well that follow. For example – compounding costs.

Manual and automated anomaly detection – how are they different?

Let's have a clearer picture of manual and automated anomaly detection from the cost point of view. The best example to consider here would be that of Anodot's automated anomaly detection system. Here, machine learning techniques are deployed to detect anomalies in real-time followed by assigning those detected anomalies a numerical ranking. These ranks are based on their significance. Lastly, it caters to grouping the related anomalies together for concise reporting.

Now consider the case of manual anomaly detection. Everything right from detecting, ranking to grouping is done manually. Just imagine the time it would take. Also, how precise would the whole process eventually turn out to be is to be given importance as well. The manpower required is huge if thinking of working on huge metrics and so will be the cost incurred. The difficulties are not just in the area of detecting but in the field of ranking as well. Some of the most common issues faced are – inconsistency of one person's quantitative ranking over time and the difference in ranking anomalies between people. What poses an area of concern is that these rankings are used for filtering out insignificant anomalies and in that case, inconsistent rankings might result in important anomalies being missed and insignificant anomalies passing the filter.

One might argue saying – what if the organizations are financially strong to afford analysts for detecting anomalies? Well, the problem with manual anomaly detection is not limited to funds alone. Something that's equally important is –the communication between the team members. Suppose one analyst detects an anomaly, he or she will have to get in touch with the other team members to see if any of them has detected an anomaly at the same time. It doesn't end there. Discussing whether and how those detected anomalies are related forms an important aspect as well. A strong line of communication is thus important.

Manual anomaly detection in real-time is very challenging and becomes difficult beyond words as and when you reach a larger number of metrics. Even though each analyst will have to handle a considerably small share of overheads, time consumed by that communication overhead is blocked, meaning that during that time, one cannot spend in detecting or ranking anomalies.

Yet another point to note is that this is just the cost incurred by communication itself and is independent of any practical channel for that communication.

Why automated anomaly detection is the key?

Let's talk about Anodot's automated anomaly detection system. This real-time, large-scale automated anomaly detection system does exactly what it is supposed to – it is automated at each step in the process: be it detection, ranking, or grouping. It's solely because of this complete automation feature that allows Anodot's system to scale. Machine learning serves to be no less than a blessing here. It not only detects anomalies but also provides very specific indicators of what's anomalous while being able to give a more holistic picture of what's going on behind the data.