Monitoring has proven to be one of the more difficult components of IT for many years, perhaps because we make too many assumptions about how monitoring works and what the monitoring tools we use can actually do. We’ve all been mesmerized by demos at trade shows that boldly claim to have finally “cracked the code”! Unfortunately, after we purchase the tool, take it home, and install it, we begin the long, winding road to disappointment when it doesn’t do what we thought it would do. Alas, the tool just showed well on the trade show floor.
Why is it such a struggle to find the holy grail of monitoring systems that can fix all of our problems? It’s simple: One tool alone can’t possibly fully monitor today’s complex IT environments—oh, and monitoring doesn’t work unless we have a full understanding of what’s really going on within our environment.
In this blog post, I will review the monitoring issues around the data presented from our systems and applications and the steps to reaching a state of monitoring nirvana.
One of the main issues with monitoring comes from a lack of understanding about the flow and value of data. First, let’s think about the initial source of the event we wish to monitor. Where does it come from? What does it say? Most tools are betting on the developers of software, hardware, and operating systems being able to provide clear, concise data that describes exactly what is going on with the code when something goes wrong.
Problems arise when we are unable to identify the full definition of the issue as soon as it appears. There is a subliminal miss when developers fail to consider that the error generated by their code is not being read by a fellow developer, but rather a managed services engineer who is keen on finding the needle in the haystack. Developers should therefore ensure they are providing comprehensive and clearly understandable data.
Even if we have perfect data that defines exactly what’s going on, we need to take the next step to remove the noise. In this case, noise is defined as things that are immaterial to the ability to understand what’s going on and what’s important. In other words, what’s important is defining, understanding, and resolving an issue—and if something isn’t contributing to doing any of these things, it’s noise.
In the course of developing code, programmers dedicate a large percentage of their code base to exception handling. Some of the exceptions that are encountered and reported are not truly problems that a monitoring application needs to further act upon. The true art is being able to identify which data points require further action within the context of the environment.
Imagine that we have great data that tells us everything we need to know with no noise. Have we solved all of our issues? Not quite—and the next issue is a doozy.
Monitoring that’s based on waiting for data to be sent, captured, and analyzed accounts for a small percent of actual problems that occur. What I mean by that is that there are many issue types that can’t be caught by the standard method of what we call “traps” in monitoring.
Here are a few examples that can’t be monitored via standard Simple Network Management Protocol (SNMP) tools:
The list keeps going, but you get the point. There is no single easy way to identify that something has failed and determine exactly what should be done about it.
I know what you’re thinking: Great, is there anything we can do to achieve true monitoring success (which we were fooled into thinking was such an easy feat when we witnessed that awesome demo on the trade show floor)? The answer is yes, but to get to that point, you need to think about monitoring differently.
We are all searching for ways to make the complicated easier to understand. You can achieve your monitoring vision—but you first need to understand the reasons why your current monitoring system doesn’t work and identify the steps to move into a mode that allows you to find the issues in your systems quickly and effectively.