Last year, some Nagios employees shared past IT stories that still haunt them to this day. After posting the article, a few of you submitted your own horror stories for us to share! We are thrilled to share some of them with you for this year’s collection.
Following Orders
Back in 2007, I worked 12-hour day and night rotations at a pretty large datacenter. One evening, I received a turnover from the day shift, and I was informed that a configuration change had gone horribly wrong. This was one in a series of changes that had gone horribly wrong, and all because someone on our admin desk decided that they knew what the person who wrote the change actually wanted versus what they actually wrote.
After the day shift leaves, the boss comes in. Starts ranting about how none of the admins should be interpreting changes. If we encounter something where the provided commands do not work as expected, we need to roll back the change and email the client, letting them know the situation and that the change failed. Very specifically, “From now on, you copy/paste everything out of the change orders, and you react to the output.”
Not too soon after, a client wanted to clean up files after an Oracle upgrade. It was irritating because I wasn’t allowed to interpret the change that was provided, and this change absolutely needed some interpretation. As I’m copying and pasting commands out of the change, it happens. I came across one command that was screwed up. I tried to escape the command as fast as I could, but the array of 15k RPM SAS drives was too fast for the slow reaction of meat and bone. I took the box out of service; fortunately, others were ready to take its place, and I dutifully began drafting up the incident report.
-M.B.
Web Hosting 101
Not too long ago, my organization hired a local web designer to overhaul our site. It was clear he was newer to the game but had a decent portfolio of example sites that he had created, and his prices were fair. Months into the design and consulting, it was clear that the project was not going as planned and that we had to move in a different direction. We did a little more research and found a larger company that specialized in creating websites tailored to our market.
Shortly after the consultations, the new web team began their work. Their first order of business was to ask for security credentials and retrieve the current website data that resided on the previous web developer’s servers. After they logged into the back-end of the website, they noticed a number of oddly placed and disorganized files and folders. A closer inspection revealed that these folders were not related to our website in any way but to the first web developer’s personal files and even some inappropriate photos! The personal directories were only the beginning of the story because it turned out that every website this developer was hosting was open to public access, modification, and browsing. Our new web developer discovered that the security permissions were not at all up to par and left several holes for basically anyone to delete or modify our site.
For multiple reasons, we were happy that the new web development team knew the proper ways to build a site from front to back.
-S.H.
Be Our Guest
We hired an IT company to manage our small office. We have 60 employees and a handful of printers, etc. We were a pretty small client for them, I suppose, and we rarely ever called for help. We upgraded our internet and, at the same time, thought it was a good idea to call the IT company to come in and upgrade our wireless network. The plan was to have an internal secured network and a guest network for occasional needs.
The IT company charged us just north of $2,000 to install a few wireless access points and knocked a couple of other tickets off the to-do list while on-site. Everything was great. We had fast internet and a guest network.
About a year later, a tech-savvy visitor came into our office and noticed that, in addition to the internal and guest networks, an additional name appeared on his laptop’s wireless list with a great signal. The wireless name was generic, almost like a model number. He clicked to join it (no password) and could immediately browse the internet. “I’m just going to check one thing for you.” A few moments later, our visitor showed us all of the devices on our network that he was able to access. He was even able to get into one of our company drives that we have mapped on every computer!
Frustrated, we called our IT company, and they told us it was probably a mistake and that we weren’t actually able to do that. We made them visit our office to confirm the security issues, and it turns out the culprit was something they installed in our network to make it easier for them to dial in. We found a new IT company, but it’s a good reminder to always be cautious, even when you get the results you’re looking for.
-N.K.
Nagios is there to alleviate all your IT horrors! Download your free, fully-loaded 30-day trial of all the Nagios products here.
