One Company's Journey Out of Darkness, Part V: Seeing the Light Also on Thwack!

Lean IT teams often do whatever they can to get by and my customer was no exception. One of the biggest challenges they had in approaching their network management strategy was to understand what they currently had. We had to work through the "day in the life" of a number of individuals to identify the core tools used, but were constantly surprised by new tools that would appear or were used so infrequently that the team would forget about them until a specific use case arose.

The team found open source tools to be of tremendous use, especially Netman and MRTG. These provided much needed visibility and the price was right given the lack of investment in monitoring tools at the time of deployment. The relatively complex nature of deployment of these tools did limit adoption and we found that often these tools lagged behind from a configuration standpoint. New equipment would be deployed without necessarily being integrated into the tool, likewise old equipment when replaced was not always removed from the tools. Lack of policy and discipline in a busy IT shop had effectively limited the effectiveness of these tools. This was further compounded by only a small subset of the team having access. Additionally, as an outside resource, I had no idea what "normal" was when looking at the tool (e.g. is that router down or has it been removed?).

These tools are something most are familiar with products like Cisco's Prime Infrastructure, Aruba's Airwave, VMWare's vSphere Operations Management (VSOM), etc. Each of these tools had been deployed widely and would tend to be used by those who's job responsibility primarily covered the area managed by the tool, however others that could benefit from this tool very rarely used it if at all. These tools tend to be fairly expensive and offer many features that are typically not leveraged very well. Additionally, most of the tools have robust AAA capabilities that would enable them to be shared with help desk teams, etc. but these features had been overlooked by the team, despite having been properly configured for their own purposes.

Some investment had been made in third party tools, typically around a specific need. A good example of this would be the Kiwi Cat Tools for ease of device backups. While this functionality existed in other tools, the company wanted a single location for all device configuration files. The customer found that numerous tools existed, but it took the entire team to enumerate them and in a couple cases multiple instances of the same tool had been purchased for usage by different teams.

Certain members of the IT team who were comfortable with writing and using scripts would develop their own toolsets, however these would often not be shared with the rest of the IT team until some specific project jogged the author's memory who would then offer up some script that had been written. In all cases these were very specific and had never been fully socialized, the team decided to develop a website internally to reference these tools and their use cases.

Working with each of the administrators and their areas of responsibility it was easy to understand how they've gotten to this point where substantial investment had been made in a myriad of tools without putting a strategy in place. Each of the teams had acquired or deployed tools to make their lives easier and tended to go with whatever was vendor aligned or free. Taking a step back together from it all and looking at the system in its entirety provided a much different perspective - is this really how we'd design our management infrastructure if we built it from the ground up? Clearly not, so what next? Looking at the tools current deployed, it was obvious that substantial duplicate functionality existed as well as a number of gaps, especially as it pertained to any one specific team's visibility.

Enumerating the existing tools, processes and use cases highlighted how much organizations actually do spend on tools while complaining that they don't have the visibility needed. Open lines of communication between teams, the development of an official or virtual "tools team", and careful consideration of products purchased are key to the success of running the IT team properly. Highly custom scripts and those who can write them can be of great value to an organization, however this value is wasted if the team at large isn't aware of these scripts and how to best utilize them.