During the Christmas break at my work, the tech department continues working for a good portion of it. We work almost as long as the gremlins that cause the problems, it seems like most of the time. Normally, I enjoy working with no one in the buildings because that means not only can I get stuff done at my own leisure, but no more work orders come in. Normally, it's a good time to work, however it's not just the people who's computers I fix that can be a major problem. A good portion of the time, some of my worst experiences come from those I work with. In this case, my boss set me up for quite a bit of anger and as I am writing this, problems still persist. So sit back and get some popcorn while I spin you the tale of my network nightmare!
For some background, we had contractors running a new Cat6 network along side the old Cat5 and Cat5e network that was currently in place. This alone was riddled with headaches due to them unplugging network cables of inept teachers, leaving behind particulate from the drop down ceiling, leaving doors unlocked, and even leaving behind empty pizza boxes. I dealt with those problems as they were presented. In the meantime, our Network Administrator was learning how to use Brocade Ruckus switches. All of our current infrastructure was HP Procurves of all various time periods and Ubiquiti edge switches that had a problem with POE burning up the onboard fuses on cards we could not get replaced and could not send the switches back for repair because the warranty was so short. The end goal was to replace all the Ubiquity with Brocade Ruckus switches, stick in new HP Procurves where we need extra connections with no POE in rooms that were air conditioned because all the current equipment were in electrical rooms. Since I am an IT Specialist, I was not involved in the major networking choices or configurations.
One of the days when getting ready for our big cutover, the Network Manager had a heart attack and went to the hospital to be put into a chemically induced coma because of a 90% blockage in his heart. So now we have a half configured network, equipment everywhere and the ONLY network person and only person who knew what was going on with the network was not only hospitalized, but unable to in anyway tell us where his notes were to get this stuff going. So as we inch closer to the break to do the cutover, my boss decides to have me check everything over and try to work out what was going on and figure out where everything was. I reluctantly did so.
As I found a saved half-working configuration, I grabbed an extra switch and began to mess with it to try and make a generic configuration that would suffice for a quick copy-paste deployment to get us started. A few days before we went into the break, another tech had to leave because of a death in the family and would not be back until the end of the break.
Already by now you are probably thinking that all the signs are saying to not do the cutover now. The fates are all stacked against it. However, it continues to get even worse. It gets so bad that I contemplated just simply walking out.
Now we get to the last day before the Christmas break. My boss had decided to come with me to the location and review everything so we could set it up for the cutover. At this point, I am highly against trying to do the cutover because of the ramifications should we screw up and not be able to correct it before the break is over. Ramifications that I am now suffering. So we go through everything we can, I install and configure as much as I possibly can in a forlorn hope to not be eaten alive by the project. At the end of the day, before we left, I was told something that I am still flabbergasted by. As my boss was on his way out and we were chatting...
"Well, I won't be able to help with this next week. My wife said I need to clean the house because we have guests coming over. Could you work on it the following week (this is the week OF Christmas, just FYI) when I am available to help?"
Not only is that excuse one of the most irritating things I have ever heard uttered, seemingly a slap in the proverbial face of work-ethic, logic, and common decency, but the reason I was making sure to keep those days clear was specifically because HE PUT IT ON OUR WORK CALENDAR TO DO IT THAT WEEK. I know what you might be thinking. It's horrible, but can it really get any worse? Well, get ready to cringe so bad that your face may just stick in that position for days to come.
The next day, I show up to work and wandered around like a lost puppy trying to find some guidance or shelter. The assistant coordinator, the second in command, asked me a question.
"Is <boss> coming in to work today? You were the last one to talk to him and I haven't seen him yet today."
ARE YOU KIDDING ME? How is it that my boss can have the audacity to not only excuse himself from a task that he himself scheduled, but he doesn't have the balls to even inform the other workers that his wife said he couldn't come out to play for the day. His wife must have his spine and balls in her purse under lock and key. By now, I'm fuming inside. I'm pretty sure I have a brain tumor and a few ulcers from this. I let the assistant boss know what I was told. She asked if I was going to need help doing the cutover, like I actually was project planning and all that, to which I said no because I don't even know what I'm doing.
Shortly after that, I guess my boss' wife let him run free for a few minutes, as he showed up. This was not to do work, but instead to leave the assistant boss with a list of what I needed to get done with the help of what was left of an 11 person department that was down 5 people, as two were already taking there vacation as well. Shortly after, he leaves and we all hop in our cars and drive on down.
We all get there and I'm still lost because I got thrown into a project already started without myself. As we walk into the building, the assistant boss who I would have assumed would take the role of boss and delegate roles, instead asks me what I want everyone to do. Some may have viewed it as them putting their faith in my abilities. I, however, viewed it as people looking for ways to absolve themselves of responsibility for the impending and certain failure that will (and has) occur. After taking a guess and just scattering people in hopes maybe someone would get something right, I took one of the other techs I consider a friend with me to listen to me complain while he helped me do my guess work. I would have grabbed another one of the techs, but I didn't want it to be obvious that I was just trying to keep my sanity and complain than working on something I had only guess work with.
So after day one of hooking up equipment, I had assumed it was all good and we all left. Oh boy was I wrong. We got network alerts galore! So on to day two. I grabbed a couple of techs to help me stare at stuff in hopes divine intervention may happen to fix the problem. Eventually we called in the ISP fiber engineer person to help us to see if the fiber may be the problem. When he got there, he assured us that he was as lost as we were. So we talked, tested, and fought the network. Day three was more of the same. With some luck, the network went up for a bit and I hightailed it out of there for a week Holiday vacation. The whole week was email alert after email alert that it wasn't working.
Over that week, my boss went in to try to fix the problem. It was never fixed while he was there. The week I got back, we tried to figure it out more. I got spanning tree set up and that seemed to get parts of the network working, so I took it at that and left. After school was back in session, I faced many work orders about network stuff not working. A couple of weeks later, I disabled all rapid spanning tree because the HP Procurves refused to accept a new root for RSTP, and just did plain old legacy STP. Over the course of the next couple of weeks, I found not one, not two, but three loops that were created by the cabling guys who helped move devices over after the cutover. I also found out that because of the change in contractors, there was a 50% failure rate when they were certifying the work that they had to redo.
I was swallowed up and now seem to live in the belly of the beast. Despite everything appearing fine, the network still has problems with VOIP quality randomly failing, WiFi connections dropping, fluctuating speeds when checking with speedtest.net, and a general disdain for work at the moment. On the lighter side, our Network Administrator is alive, awake, and back at work with plans to retire as soon as he hits the mark he needs.
I think the moral to the story is quite simply to not let work get to you, because then you have stress and problems at work. Or something like that. I'm not really good at morals.
No comments:
Post a Comment