Friday, March 23, 2018

Techs from the Crypt: Holiday Network Nightmare

During the Christmas break at my work, the tech department continues working for a good portion of it. We work almost as long as the gremlins that cause the problems, it seems like most of the time. Normally, I enjoy working with no one in the buildings because that means not only can I get stuff done at my own leisure, but no more work orders come in. Normally, it's a good time to work, however it's not just the people who's computers I fix that can be a major problem. A good portion of the time, some of my worst experiences come from those I work with. In this case, my boss set me up for quite a bit of anger and as I am writing this, problems still persist. So sit back and get some popcorn while I spin you the tale of my network nightmare!

For some background, we had contractors running a new Cat6 network along side the old Cat5 and Cat5e network that was currently in place. This alone was riddled with headaches due to them unplugging network cables of inept teachers, leaving behind particulate from the drop down ceiling, leaving doors unlocked, and even leaving behind empty pizza boxes. I dealt with those problems as they were presented. In the meantime, our Network Administrator was learning how to use Brocade Ruckus switches. All of our current infrastructure was HP Procurves of all various time periods and Ubiquiti edge switches that had a problem with POE burning up the onboard fuses on cards we could not get replaced and could not send the switches back for repair because the warranty was so short. The end goal was to replace all the Ubiquity with Brocade Ruckus switches, stick in new HP Procurves where we need extra connections with no POE in rooms that were air conditioned because all the current equipment were in electrical rooms. Since I am an IT Specialist, I was not involved in the major networking choices or configurations.

One of the days when getting ready for our big cutover, the Network Manager had a heart attack and went to the hospital to be put into a chemically induced coma because of a 90% blockage in his heart. So now we have a half configured network, equipment everywhere and the ONLY network person and only person who knew what was going on with the network was not only hospitalized, but unable to in anyway tell us where his notes were to get this stuff going. So as we inch closer to the break to do the cutover, my boss decides to have me check everything over and try to work out what was going on and figure out where everything was. I reluctantly did so.

As I found a saved half-working configuration, I grabbed an extra switch and began to mess with it to try and make a generic configuration that would suffice for a quick copy-paste deployment to get us started. A few days before we went into the break, another tech had to leave because of a death in the family and would not be back until the end of the break.

Already by now you are probably thinking that all the signs are saying to not do the cutover now. The fates are all stacked against it. However, it continues to get even worse. It gets so bad that I contemplated just simply walking out.

Now we get to the last day before the Christmas break. My boss had decided to come with me to the location and review everything so we could set it up for the cutover. At this point, I am highly against trying to do the cutover because of the ramifications should we screw up and not be able to correct it before the break is over. Ramifications that I am now suffering. So we go through everything we can, I install and configure as much as I possibly can in a forlorn hope to not be eaten alive by the project. At the end of the day, before we left, I was told something that I am still flabbergasted by. As my boss was on his way out and we were chatting...

"Well, I won't be able to help with this next week. My wife said I need to clean the house because we have guests coming over. Could you work on it the following week (this is the week OF Christmas, just FYI) when I am available to help?"

Not only is that excuse one of the most irritating things I have ever heard uttered, seemingly a slap in the proverbial face of work-ethic, logic, and common decency, but the reason I was making sure to keep those days clear was specifically because HE PUT IT ON OUR WORK CALENDAR TO DO IT THAT WEEK. I know what you might be thinking. It's horrible, but can it really get any worse? Well, get ready to cringe so bad that your face may just stick in that position for days to come.

The next day, I show up to work and wandered around like a lost puppy trying to find some guidance or shelter. The assistant coordinator, the second in command, asked me a question.

"Is <boss> coming in to work today? You were the last one to talk to him and I haven't seen him yet today."

ARE YOU KIDDING ME? How is it that my boss can have the audacity to not only excuse himself from a task that he himself scheduled, but he doesn't have the balls to even inform the other workers that his wife said he couldn't come out to play for the day. His wife must have his spine and balls in her purse under lock and key. By now, I'm fuming inside. I'm pretty sure I have a brain tumor and a few ulcers from this. I let the assistant boss know what I was told. She asked if I was going to need help doing the cutover, like I actually was project planning and all that, to which I said no because I don't even know what I'm doing.

Shortly after that, I guess my boss' wife let him run free for a few minutes, as he showed up. This was not to do work, but instead to leave the assistant boss with a list of what I needed to get done with the help of what was left of an 11 person department that was down 5 people, as two were already taking there vacation as well. Shortly after, he leaves and we all hop in our cars and drive on down.

We all get there and I'm still lost because I got thrown into a project already started without myself. As we walk into the building, the assistant boss who I would have assumed would take the role of boss and delegate roles, instead asks me what I want everyone to do. Some may have viewed it as them putting their faith in my abilities. I, however, viewed it as people looking for ways to absolve themselves of responsibility for the impending and certain failure that will (and has) occur. After taking a guess and just scattering people in hopes maybe someone would get something right, I took one of the other techs I consider a friend with me to listen to me complain while he helped me do my guess work. I would have grabbed another one of the techs, but I didn't want it to be obvious that I was just trying to keep my sanity and complain than working on something I had only guess work with.

So after day one of hooking up equipment, I had assumed it was all good and we all left. Oh boy was I wrong. We got network alerts galore! So on to day two. I grabbed a couple of techs to help me stare at stuff in hopes divine intervention may happen to fix the problem. Eventually we called in the ISP fiber engineer person to help us to see if the fiber may be the problem. When he got there, he assured us that he was as lost as we were. So we talked, tested, and fought the network. Day three was more of the same. With some luck, the network went up for a bit and I hightailed it out of there for a week Holiday vacation. The whole week was email alert after email alert that it wasn't working.

Over that week, my boss went in to try to fix the problem. It was never fixed while he was there. The week I got back, we tried to figure it out more. I got spanning tree set up and that seemed to get parts of the network working, so I took it at that and left. After school was back in session, I faced many work orders about network stuff not working. A couple of weeks later, I disabled all rapid spanning tree because the HP Procurves refused to accept a new root for RSTP, and just did plain old legacy STP. Over the course of the next couple of weeks, I found not one, not two, but three loops that were created by the cabling guys who helped move devices over after the cutover. I also found out that because of the change in contractors, there was a 50% failure rate when they were certifying the work that they had to redo.

I was swallowed up and now seem to live in the belly of the beast. Despite everything appearing fine, the network still has problems with VOIP quality randomly failing, WiFi connections dropping, fluctuating speeds when checking with, and a general disdain for work at the moment. On the lighter side, our Network Administrator is alive, awake, and back at work with plans to retire as soon as he hits the mark he needs.

I think the moral to the story is quite simply to not let work get to you, because then you have stress and problems at work. Or something like that. I'm not really good at morals.

Wednesday, March 14, 2018

Techs from the Crypt: I don't understand!

Realistically, I have not been a tech for very long, however I work on four different sites normally and interact with quite a lot of people. In my travels, I have come across some humorous, horrifying, and down right strange scenarios. I will now share them with you as part of an attempt at a running series I am calling "Techs from the Crypt" as homage to a favorite show of mine as a kid, Tales from the Crypt! Get it? Did I really need to explain it?

Moving on, I will make up names where needed, not so much to protect the people involved, but more so because I don't care enough to remember anyone's name. Luckily my time in retail has taught me how to cover up my anti-social nature and severe anger towards the more remedial tasks I'm expected to perform.

As a tech, I often come across a lot of people who say something along the lines of "I don't know," or "I don't understand." I often feel myself rewording what I say many times over to make a point or even get some acknowledgement that there is thought behind the vacant stare and hysterical smile of someone completely distraught because "the Internet is broken," or "I didn't do anything and it's no longer working." I'm sure almost every field you can be in has some variation of this interaction, be it with co-workers or clients. So now I submit to you a story about a lady who even when I think back on it, I find hard to believe she actually did all the things she did.

I work for a school system, so most of the people I help are teachers, those we trust to educate our youths. This particular case was at an elementary school. I received a work order about a teacher, we shall call her Mrs. Lego (part of an inside joke I may mention later), could not log into her Google account. Now when an account is setup on our system, a Google account is created. The problem is that you cannot log on to the account until after you change your password and then it gets synchronized with Google. I was quite confident in dealing with these cases as it was the beginning of the year and a lot of new teachers sent in identical work orders. With that, I marched off to the school to get the new teacher squared away.

So far it all seemed routine, and then I met the teacher. Now, let me be clear, she was and is a very nice lady who has never been intentionally rude to me even when I almost lost my temper. However, I would not say Mrs. Lego is the most receptive of people. After explaining to me her problem, I explained to her that she needed to change her password and showed her how. Now, our passwords have complexity requirements that are more strong than some of the other unconnected systems certain groups of staff need to use, usually due to just software limitations. Mrs. Lego tried to change her password to one she had previously setup on one of these particular systems. After it rejected her twice, I asked her to tell me the password. I then explained to her that because of the complexity requirements, it needs more to it, like a special character, maybe an exclamation mark at the end or something simple to remember like punctuation.

"But I want it to be the same password for everything," stated Mrs. Lego quite adamantly.

"Well, I understand that, but I cannot change these requirements. To access Google through your account, it's required to change your password."

"But I need them to be the same, or else what's the point?"

"To access Google, you NEED to change your password."

"Well, why can't I use this one?"

"Because there are requirements that need to be met."

"I don't think you're understanding me. I have my password for that set. I want them to be the same passwords."

"I get that, but unless you change both passwords, we can't make this one the same."

"No, you don't understand, I want my password-"

"To be the same as the other one, I get what you're saying but I cannot do that for you."

"Oh... well then there is no point in changing my password, how do I go back?"

"To access your Google account you HAVE to change your password."

"But I don't want to unless they can match."

This is the shortened version, as this conversation then continued on for some more time. Finally, my patients were gone. She was convinced I couldn't understand what she wanted so now she wants everything back but wants to access her Google account. I was so fed up, I did the only thing I could within my power. I fired up Google Admin Console and manually entered her default password and reluctantly left that be. However, it does not end there, oh no my friends. You see, she was a new teacher and had questions. Many questions. We have instructional people for such questions, but I was nice enough (dumb enough is more like it) to attempt to help to the best of my ability.

You see, at one point they thought it was a good idea to try to continue the use of old outdated computer by installing Ubuntu on them. As a Linux user, I informed them many times after I started and came across this that Ubuntu os a full featured OS, it is not lightweight or good for repurposing old computers. Mrs. Lego had two and one with Windows still on it. The Ubuntu computer log in automatically and the Windows they are recommended to use a class login. After explaining to her the class login and showing her the Windows computer, we discussed the Ubuntu. I explained to her they really are just there for web browsing, there is no Microsoft office or the like on them.

"So, they're not real computers?"

"No, they are just older computers that the schools are trying to reuse to save money."

"At the school system I come from, they stripped out the guts of old computers and called them Linux machines."

"...uh... yeah... same thing?..."

"Well, can I get real computers instead?"

"... uh... tell you what... put... put in a work order and... I'll see if I can scrap together some parts and get Windows on them..."

"Oh, that would be great, what do I put down?"

So I gave her word for word what to put down and then made a mental note to upgrade the RAM so it could handle Windows and try to get this wonderful woman out of my hair.

Then for a bit, we talked about the tech and differences between school systems. I thought it was over and I had weathered the storm, then while I was mid-sentence, she walked over to the door and said she needed to go pick up the kids... and walked out before I could even respond.

It's at times like those that I wonder how people get there in life, or how they often seem better off than me. Perhaps it's just the chipping away of my soul that makes the other side seem so much better. Mrs. Lego went on to terrorize our instructional techs after I told her to submit a work order for them to come by and walk her through the tech.

You see, when one of them was talking to her, he passed the comment that this stuff is easy, jut plug it together, like Lego...

"But I don't have any Legos."

And that, my friends, is the story of Mrs. Lego and why I don't understand. I hope you enjoyed. I have a few other stories I hope to get down before I forget too much detail and make it hard to put in a decent story form.

Tag Cloud

.NET (2) A+ (5) ad ds (1) addon (4) Android (4) anonymous functions (1) application (9) arduino (1) artificial intelligence (1) backup (1) bash (6) camera (2) certifications (3) comptia (5) css (2) customize (11) encryption (3) error (13) exploit (5) ftp (1) funny (4) gadget (4) games (3) GUI (5) hardware (16) haskell (6) help (14) HTML (3) imaging (2) irc (1) it (1) java (2) javascript (13) jobs (1) Linux (19) lua (1) Mac (4) malware (1) math (6) msp (1) network (13) perl (2) php (3) plugin (2) powershell (8) privacy (2) programming (24) python (10) radio (2) regex (3) repair (2) security (16) sound (2) speakers (2) ssh (1) story (5) Techs from the Crypt (5) telnet (1) tools (13) troubleshooting (11) tutorial (9) Ubuntu (4) Unix (2) virtualization (2) web design (6) Windows (16) world of warcraft (1) wow (1) wx (1)