Showing posts with label Techs from the Crypt. Show all posts
Showing posts with label Techs from the Crypt. Show all posts

Friday, May 2, 2025

Techs from the Crypt: OSPF Flood Wars!

 It's been a while since I posted anything, but I wanted to try to get back into posting stuff. What better way to start than another tech horror story! Currently I am working as a Senior Network Engineer, a recent promotion I am quite proud of. One of the reasons I earned my current title is due to my involvement in a lot of troubleshooting sessions and overall network improvement. With that, it makes sense strange issues tend to land in my lap. One of these issues that happened to land in my lap was an OSPF Flood War. Not my first one, so I figured easy enough. We had four devices showing up in logs with intermittent connectivity. With a bit of luck and some careful planning, I managed to get on each one and fixed what was showing as duplicates. With solid connectivity restored I closed out the tickets, only took me most of a day since it was a remote site.


The next day, I got a message about another OSPF Flood War message for the same four buildings. Connectivity was still solid. All four buildings had no errors, but the upstream did. I went over the same four buildings and the upstream device multiple times for most of the day. Still no luck what was making the messages show. Digging through everything I could find, I decided to ask around.


One of the new engineers I had helped earlier working on getting phones setup at the same remote site, but on a completely different building. This building did not show up in any of the logs. With no other ideas, I got into the device for the building. It finally revealed itself when I went into the switch. The voice network is on a separate VRF and the point to point VLAN on the uplink was in the voice VRF and the downstream one was on the default. Thus, two VRFs leaked into each other across one device that never showed up in any errors logs. Two days of staring at devices and it was that simple. It was a nice and easy fix of just moving things into the appropriate VRF.


With all that, moving forward I am hoping to work on some more technical posts focused on network, since I guess that's my life now. I may also consider enlisting the help of all the new fancy AI tools and if I do, I will disclose that on the post. Just a consideration because sometimes my thoughts are too boiled down and dry to make a coherent post as opposed to just like a bullet list of information. Just a thought. Hope you enjoyed my story!

Saturday, January 11, 2020

Techs from the Crypt: Haunted Office

So at one of my schools, the principle jokes around about her office being haunted. Generally, if a weird problem can occur, it happens here. My main problem I originally had with this room was it took replacing her VoIP phone twice until there was no static. Two brand new phones did not work, but a random used one from another place worked just fine. On top of this, there was always a minor problem somewhere along the line.

So one day I am heading into work and I get an email. It was a long chain of emails forwarded to me that I didn't read right away and instead just saw the immediate message to change out her computer. Now for changing out an office computer, it is a pain. There are extra pieces of software that would need to be installed, some of which I don't do. So rather than bothering asking anyone, I do my usual of I'll deal with it when I deal with it. I grab a spare computer just in case and then skim the email chain.

The long story short is after we had new bus radios set up, they get static when anyone takes the hand-helds into the principles office. They think it is Bluetooth or something messing with it, which I can't say I've seen before and the computer doesn't have Bluetooth or a wireless card. I have a CB radio with built in Bluetooth and that doesn't cause any problems whatsoever. The reason they think it is the computer is because the static stops when it's off. Well, computers can be connected to and powering many other things, so that is not definitive troubleshooting.

I get to the school and she shows me what she does when it happens. Oddly enough, there was static in her office, but not out. So I stay in clear line of sight out the door and the range is far below what Bluetooth should do, about 10' at best had static. She did have a wireless mouse, so I unplugged the dongle. Still had static. I unplugged everything except the monitor, still static. I turned off the computer, static gone. Now this is weird and should not be happening. I turn the computer back on, no static until a certain point of booting up. This was really strange.

I proceed to dance around the room with the hand-held looking for any dead areas on the static. There were. It had very distinct, equally spaced dead areas. My random thought is that is very wave-like. I thought back to when the computer was booting up and when the static came on. There was a point where there was static, a space, then solid static. It was when the monitor auto-adjusted. I turned the monitor off, no static. Not anywhere in the room.

Here is my best conspiracy-theory guess. The monitor was a much older LCD display that uses a crap ton of power with a controlled refresh rate. I think that refresh modulation, or whatever, was going at a frequency with enough power through old enough electronics to generate EMI detectable within the frequency the hand-held was set for. Makes enough sense for me to say sure, why not. It is either that, or the monitor needs an exorcism.

So I grabbed a newer LCD monitor that weighed about 1/5th the original and put that in. No more static.

Let this story be a lesson to everyone out there. When repairing anything, actually troubleshoot the problem. If It wasn't for my reluctance to do something that would take a lot of time and effort, I would have wasted a lot of time and effort and not have a problem solved.

Saturday, December 14, 2019

Techs from the Crypt: Job Interview Gone Wrong

I was debating for a short while whether to write this or not as it was fairly recent. Obviously my decision is to write it. Now usually I get on with people just fine and working retail I tend to stay in a neutral position and let others steer the conversation. I know there are some personality types I just don't get along with and admit it is a problem with myself to a degree. Unfortunately it seems to be fairly common among the tech savvy areas of things. I go along the philosophy of if it works for what I need, I go with it unless there is a reason not to. So most of the tech in my house ends up with some odd hack or it is just sort of there without much fine tuning because I have no reason to. Quite a lot of tech people I seem to run into are the kinds who... their way is the right way and if there's a problem they see, even if it's not a problem, you should just do what they say because they know best. What does this have to do with a job interview? Well, I recently had a job interview where I almost hung up on the person because they were everything I dislike about certain IT people.

So on LinkedIn I get a message about a position. It looked automated with some grammatical errors, but I've learned to take chances and just go for anything because you never know what door could open for you. I say that I'm interested and get a request for a number and a good time to chat about the position. I give my cell and say afternoon. Perhaps that was an error on my part, but by afternoon I mean 12pm to 1pm because that's what I always heard it for growing up in upstate NY. Generally up there we considered after 3 to enter into the evening, so I expected at the latest sometime before 3 I would get the call.

The time came and past (me on lunch), so I forgot about it and was driving home from work. Around 3.40pm and I get the call. I don't have Bluetooth or anything for my car because I keep things relatively downgraded miss it. When I get home, it's a message from the guy asking to either talk or schedule a time to talk. I thought we did schedule it, but like I said afternoon but some people that means something different.

I call him back and we start to talk. I explain a bit about what I do and why I may be looking for a better opportunity which usually gets a good response, but it fell flat. I may have misread because I just took some extra strength Dayquil to keep a head cold at bay. I chalk it up to that in my head and move on. A few oddities stuck out to me in the conversation.

The first was related to hard drives. He asked about regular and solid state so I gave the spiel about the differences and using SSD to boot with HDD for storage for speed and savings on storage capacity. He then asked about M.2, which I'm bad with names of stuff. Tell him I can't remember and when he mentioned what it is the alphabet soup of PCIE NVME came to mind and I remembered because I just bought my first laptop with that kind of storage. So I chime in and mention that yes I actually have a couple of laptops at home with those kinds of storage and it's my first time using them. He asked about what they use at the school system I work at and I tell him mostly HDD because it's relatively cheap storage and there are a few SSD to boot from on some machines. I get told by him that no new servers have regular HDD because the speed is too slow and at the very least they get SSD for cost effectiveness. He then asks why we have HDD and I explain we are a public school system, we don't have the money to be cutting edge. In the back of my mind I'm wondering where he's been because HDD are still in wide use, still manufactured, and very common. Last I knew a lot of places aim for large storage with redundancy for the best value possible. I shake that off and just assume maybe he's more of the high-end side of things.

The next thing to stand out was when we started talking about Internet speeds and stuff. I know enough networking to get me by in most situations, but I don't do it on a daily basis so I can take a moment to remember things. He asked about my plan and all that and I tell him right now I have a 30 up 10 down. We go for a while and then he goes "Megabytes or Megabits?" I say "Megabits, that's how you measure network speeds." So then he asks me my throughput in megabytes. I tell him I'm not sure off the top of my head, just divide it by 8. He says "yeah, that's right so it's like 2 or something... 2.75." No, it's not but whatever you say. Just for those that don't know, you can estimate with the two closest whole numbers, in which 3*8 is 24 and 4*8 is 32, so it's between 3 and 4. As I said, I was on Dayquil and I also try to be nice and neutral because I worked retail long enough to know how to avoid most arguments.

We continue on, I am starting to get annoyed because the guy keeps either trying to correct me or tell me what the answer is to things I can't remember all the specifics on. Part of my problem is my job is so generalized I can't always shift gears and I'm sure a lot of techs out there know what I'm talking about when the shifts and alphabet soups get confusing after a while.

The questions started getting rather personal and weirded me out a bit, I was wondering if the guy was even being serious at this point. Questions like do I have a lab at my house, what do I do in my free time, what my WAN IP is. I tell him what I'm comfortable with telling him and tell him I do not know my WAN IP.

"Well, are you at your computer?"
No, and it's off right now.
"Well, if you logged into your router, what would it say?"
Well, I have little bit of a weird setup with two routers, the business class Comcast one and my own personal one, so one would be a 10 dot something and the other would be an actual WAN.
"Oh, I see. It's not weird, it's wrong."

At this point, I'm thinking who the hell is this guy? I didn't call him for tech support.

No, it's just a network in a network.
"Yes, it's called a double NAT and that's wrong."
Well, it works for what I need it to and I don't have any problems with it.
"Well, it's wrong. You should put your Comcast router in bridge mode and then it will work right."
I don't have any problems with it and I just go with what works.
"You should listen to me and spend the 20 minutes to put it in bridge mode. I know what I'm talking about, I do this all the time. It would get rid of any weird issues you have with connectivity or VPN."
I haven't had any problems with vpn or connectivity. I'll fix a problem when it becomes a problem.

At this point I honestly wanted to shout a few expletives, tell him to hire an HR or PR guy to talk to people, and hang up on him. I get I do things in ways they might not be done. At work I get keyboards for different chromebooks to connect to other models with some pliers and modify them, doing things not by the books to get the results I want when I want is what I do. I also didn't ask for help with my setup, my setup works for me.

I really did consider just hanging up on him before considering telling him to get some help from an HR or a PR guy to get help talking to people. I opted to finish the call as calmly as I could. After I finished and had some time to think it over, I sent a response on LinkedIn. For your reading pleasure, here's what I said:

I appreciate you taking the time for that interview, but after reflecting upon the conversation, I no longer have interest in a position for your company. The main reason is because your way of talking struck me as a type of person I could not work with or for. I wish you luck with your search.

That was about the best I could think to say because I really wanted to be rude. I'm sure someone may think I am over-reacting or even in the wrong. Maybe I should listen to the high and mighty people who know of problems I don't even know I have. As far as a job interview going wrong, this was a first that went in this sideways direction. I've interviewed with people who have tried to cheat me like a car salesman, people who have no interest, people who think I am clueless, people who right out of the gate feel I have no business getting to that stage of the interview process, but never someone who wants to give me tech support I never asked for.

In reflection as a whole, I am truly growing tired with the stereotypical "IT pros."  My best advice to any IT people would be learn to laugh, shrug things off, and let things go. If you can't, shout them randomly on the Internet, then move on.

Thursday, July 25, 2019

Techs from the Crypt: Renovations

So recently at one of my job sites, they are doing major renovations. Part of those renovations is building a man trap entrance in the entrance. There was an office already next to the doors, so they decided to use that as the reception. There wasn't anything that I needed to do, as it was all the job of maintenance and contractors. All the ethernet was already ran to that room.

After being questioned about it, I told them who they needed to talk to and that everything should already be setup. So I figured they could just plug everything in and be ready to go. That was at the beginning of the summer. Now, with school about to start, they were getting everything in place. I got a work order about the internet not working. The setup is a little weird, it goes to a POE phone, then to the computer as a daisy chain. Often times people plug the ports in backwards, so the phone won't get POE and turn on.

So I went to the job site and hooked up the stuff. Nothing. That's a little weird, I thought. So I moved around the connections in the room and over on the switch. Still nothing. So I figure maybe they pulled a connection loose when moving things around.

I get out my network tester and plug it in. It reads 11 feet with no connection. That's a little strange when the cord alone is only 8 feet. So I pull off the plate and pull out the ports. Everything looked secure. This is strange. Sitting back on the floor, I run through the scenario in my head and survey the area.

That's right, they put a window in the wall. They couldn't, could they? In disbelief I grab all the cords and yank them out of the wall.






They cut THROUGH  the cables. They cut through FIVE ethernet cords and said absolutely nothing about it. I don't know which was worse, the fact that it happened or me expecting better of people. Of all the lazy, half-assed, idiotic, and just plain stupid things I've seen done, this is one I won't soon forget. Just the other year we paid a contractor to run those drops, and this year they paid someone to screw it up. Now I have to waste my time because some idiots couldn't be bothered to check before they cut.

A word to anyone in or to be in the IT field, expect stupid. When presented with all options, the dumbest reason is usually the real reason.

Friday, March 23, 2018

Techs from the Crypt: Holiday Network Nightmare

During the Christmas break at my work, the tech department continues working for a good portion of it. We work almost as long as the gremlins that cause the problems, it seems like most of the time. Normally, I enjoy working with no one in the buildings because that means not only can I get stuff done at my own leisure, but no more work orders come in. Normally, it's a good time to work, however it's not just the people who's computers I fix that can be a major problem. A good portion of the time, some of my worst experiences come from those I work with. In this case, my boss set me up for quite a bit of anger and as I am writing this, problems still persist. So sit back and get some popcorn while I spin you the tale of my network nightmare!

For some background, we had contractors running a new Cat6 network along side the old Cat5 and Cat5e network that was currently in place. This alone was riddled with headaches due to them unplugging network cables of inept teachers, leaving behind particulate from the drop down ceiling, leaving doors unlocked, and even leaving behind empty pizza boxes. I dealt with those problems as they were presented. In the meantime, our Network Administrator was learning how to use Brocade Ruckus switches. All of our current infrastructure was HP Procurves of all various time periods and Ubiquiti edge switches that had a problem with POE burning up the onboard fuses on cards we could not get replaced and could not send the switches back for repair because the warranty was so short. The end goal was to replace all the Ubiquity with Brocade Ruckus switches, stick in new HP Procurves where we need extra connections with no POE in rooms that were air conditioned because all the current equipment were in electrical rooms. Since I am an IT Specialist, I was not involved in the major networking choices or configurations.

One of the days when getting ready for our big cutover, the Network Manager had a heart attack and went to the hospital to be put into a chemically induced coma because of a 90% blockage in his heart. So now we have a half configured network, equipment everywhere and the ONLY network person and only person who knew what was going on with the network was not only hospitalized, but unable to in anyway tell us where his notes were to get this stuff going. So as we inch closer to the break to do the cutover, my boss decides to have me check everything over and try to work out what was going on and figure out where everything was. I reluctantly did so.

As I found a saved half-working configuration, I grabbed an extra switch and began to mess with it to try and make a generic configuration that would suffice for a quick copy-paste deployment to get us started. A few days before we went into the break, another tech had to leave because of a death in the family and would not be back until the end of the break.

Already by now you are probably thinking that all the signs are saying to not do the cutover now. The fates are all stacked against it. However, it continues to get even worse. It gets so bad that I contemplated just simply walking out.

Now we get to the last day before the Christmas break. My boss had decided to come with me to the location and review everything so we could set it up for the cutover. At this point, I am highly against trying to do the cutover because of the ramifications should we screw up and not be able to correct it before the break is over. Ramifications that I am now suffering. So we go through everything we can, I install and configure as much as I possibly can in a forlorn hope to not be eaten alive by the project. At the end of the day, before we left, I was told something that I am still flabbergasted by. As my boss was on his way out and we were chatting...

"Well, I won't be able to help with this next week. My wife said I need to clean the house because we have guests coming over. Could you work on it the following week (this is the week OF Christmas, just FYI) when I am available to help?"

Not only is that excuse one of the most irritating things I have ever heard uttered, seemingly a slap in the proverbial face of work-ethic, logic, and common decency, but the reason I was making sure to keep those days clear was specifically because HE PUT IT ON OUR WORK CALENDAR TO DO IT THAT WEEK. I know what you might be thinking. It's horrible, but can it really get any worse? Well, get ready to cringe so bad that your face may just stick in that position for days to come.

The next day, I show up to work and wandered around like a lost puppy trying to find some guidance or shelter. The assistant coordinator, the second in command, asked me a question.

"Is <boss> coming in to work today? You were the last one to talk to him and I haven't seen him yet today."

ARE YOU KIDDING ME? How is it that my boss can have the audacity to not only excuse himself from a task that he himself scheduled, but he doesn't have the balls to even inform the other workers that his wife said he couldn't come out to play for the day. His wife must have his spine and balls in her purse under lock and key. By now, I'm fuming inside. I'm pretty sure I have a brain tumor and a few ulcers from this. I let the assistant boss know what I was told. She asked if I was going to need help doing the cutover, like I actually was project planning and all that, to which I said no because I don't even know what I'm doing.

Shortly after that, I guess my boss' wife let him run free for a few minutes, as he showed up. This was not to do work, but instead to leave the assistant boss with a list of what I needed to get done with the help of what was left of an 11 person department that was down 5 people, as two were already taking there vacation as well. Shortly after, he leaves and we all hop in our cars and drive on down.

We all get there and I'm still lost because I got thrown into a project already started without myself. As we walk into the building, the assistant boss who I would have assumed would take the role of boss and delegate roles, instead asks me what I want everyone to do. Some may have viewed it as them putting their faith in my abilities. I, however, viewed it as people looking for ways to absolve themselves of responsibility for the impending and certain failure that will (and has) occur. After taking a guess and just scattering people in hopes maybe someone would get something right, I took one of the other techs I consider a friend with me to listen to me complain while he helped me do my guess work. I would have grabbed another one of the techs, but I didn't want it to be obvious that I was just trying to keep my sanity and complain than working on something I had only guess work with.

So after day one of hooking up equipment, I had assumed it was all good and we all left. Oh boy was I wrong. We got network alerts galore! So on to day two. I grabbed a couple of techs to help me stare at stuff in hopes divine intervention may happen to fix the problem. Eventually we called in the ISP fiber engineer person to help us to see if the fiber may be the problem. When he got there, he assured us that he was as lost as we were. So we talked, tested, and fought the network. Day three was more of the same. With some luck, the network went up for a bit and I hightailed it out of there for a week Holiday vacation. The whole week was email alert after email alert that it wasn't working.

Over that week, my boss went in to try to fix the problem. It was never fixed while he was there. The week I got back, we tried to figure it out more. I got spanning tree set up and that seemed to get parts of the network working, so I took it at that and left. After school was back in session, I faced many work orders about network stuff not working. A couple of weeks later, I disabled all rapid spanning tree because the HP Procurves refused to accept a new root for RSTP, and just did plain old legacy STP. Over the course of the next couple of weeks, I found not one, not two, but three loops that were created by the cabling guys who helped move devices over after the cutover. I also found out that because of the change in contractors, there was a 50% failure rate when they were certifying the work that they had to redo.

I was swallowed up and now seem to live in the belly of the beast. Despite everything appearing fine, the network still has problems with VOIP quality randomly failing, WiFi connections dropping, fluctuating speeds when checking with speedtest.net, and a general disdain for work at the moment. On the lighter side, our Network Administrator is alive, awake, and back at work with plans to retire as soon as he hits the mark he needs.

I think the moral to the story is quite simply to not let work get to you, because then you have stress and problems at work. Or something like that. I'm not really good at morals.

Wednesday, March 14, 2018

Techs from the Crypt: I don't understand!

Realistically, I have not been a tech for very long, however I work on four different sites normally and interact with quite a lot of people. In my travels, I have come across some humorous, horrifying, and down right strange scenarios. I will now share them with you as part of an attempt at a running series I am calling "Techs from the Crypt" as homage to a favorite show of mine as a kid, Tales from the Crypt! Get it? Did I really need to explain it?

Moving on, I will make up names where needed, not so much to protect the people involved, but more so because I don't care enough to remember anyone's name. Luckily my time in retail has taught me how to cover up my anti-social nature and severe anger towards the more remedial tasks I'm expected to perform.

As a tech, I often come across a lot of people who say something along the lines of "I don't know," or "I don't understand." I often feel myself rewording what I say many times over to make a point or even get some acknowledgement that there is thought behind the vacant stare and hysterical smile of someone completely distraught because "the Internet is broken," or "I didn't do anything and it's no longer working." I'm sure almost every field you can be in has some variation of this interaction, be it with co-workers or clients. So now I submit to you a story about a lady who even when I think back on it, I find hard to believe she actually did all the things she did.

I work for a school system, so most of the people I help are teachers, those we trust to educate our youths. This particular case was at an elementary school. I received a work order about a teacher, we shall call her Mrs. Lego (part of an inside joke I may mention later), could not log into her Google account. Now when an account is setup on our system, a Google account is created. The problem is that you cannot log on to the account until after you change your password and then it gets synchronized with Google. I was quite confident in dealing with these cases as it was the beginning of the year and a lot of new teachers sent in identical work orders. With that, I marched off to the school to get the new teacher squared away.

So far it all seemed routine, and then I met the teacher. Now, let me be clear, she was and is a very nice lady who has never been intentionally rude to me even when I almost lost my temper. However, I would not say Mrs. Lego is the most receptive of people. After explaining to me her problem, I explained to her that she needed to change her password and showed her how. Now, our passwords have complexity requirements that are more strong than some of the other unconnected systems certain groups of staff need to use, usually due to just software limitations. Mrs. Lego tried to change her password to one she had previously setup on one of these particular systems. After it rejected her twice, I asked her to tell me the password. I then explained to her that because of the complexity requirements, it needs more to it, like a special character, maybe an exclamation mark at the end or something simple to remember like punctuation.

"But I want it to be the same password for everything," stated Mrs. Lego quite adamantly.

"Well, I understand that, but I cannot change these requirements. To access Google through your account, it's required to change your password."

"But I need them to be the same, or else what's the point?"

"To access Google, you NEED to change your password."

"Well, why can't I use this one?"

"Because there are requirements that need to be met."

"I don't think you're understanding me. I have my password for that set. I want them to be the same passwords."

"I get that, but unless you change both passwords, we can't make this one the same."

"No, you don't understand, I want my password-"

"To be the same as the other one, I get what you're saying but I cannot do that for you."

"Oh... well then there is no point in changing my password, how do I go back?"

"To access your Google account you HAVE to change your password."

"But I don't want to unless they can match."

This is the shortened version, as this conversation then continued on for some more time. Finally, my patients were gone. She was convinced I couldn't understand what she wanted so now she wants everything back but wants to access her Google account. I was so fed up, I did the only thing I could within my power. I fired up Google Admin Console and manually entered her default password and reluctantly left that be. However, it does not end there, oh no my friends. You see, she was a new teacher and had questions. Many questions. We have instructional people for such questions, but I was nice enough (dumb enough is more like it) to attempt to help to the best of my ability.

You see, at one point they thought it was a good idea to try to continue the use of old outdated computer by installing Ubuntu on them. As a Linux user, I informed them many times after I started and came across this that Ubuntu os a full featured OS, it is not lightweight or good for repurposing old computers. Mrs. Lego had two and one with Windows still on it. The Ubuntu computer log in automatically and the Windows they are recommended to use a class login. After explaining to her the class login and showing her the Windows computer, we discussed the Ubuntu. I explained to her they really are just there for web browsing, there is no Microsoft office or the like on them.

"So, they're not real computers?"

"No, they are just older computers that the schools are trying to reuse to save money."

"At the school system I come from, they stripped out the guts of old computers and called them Linux machines."

"...uh... yeah... same thing?..."

"Well, can I get real computers instead?"

"... uh... tell you what... put... put in a work order and... I'll see if I can scrap together some parts and get Windows on them..."

"Oh, that would be great, what do I put down?"

So I gave her word for word what to put down and then made a mental note to upgrade the RAM so it could handle Windows and try to get this wonderful woman out of my hair.

Then for a bit, we talked about the tech and differences between school systems. I thought it was over and I had weathered the storm, then while I was mid-sentence, she walked over to the door and said she needed to go pick up the kids... and walked out before I could even respond.

It's at times like those that I wonder how people get there in life, or how they often seem better off than me. Perhaps it's just the chipping away of my soul that makes the other side seem so much better. Mrs. Lego went on to terrorize our instructional techs after I told her to submit a work order for them to come by and walk her through the tech.

You see, when one of them was talking to her, he passed the comment that this stuff is easy, jut plug it together, like Lego...

"But I don't have any Legos."

And that, my friends, is the story of Mrs. Lego and why I don't understand. I hope you enjoyed. I have a few other stories I hope to get down before I forget too much detail and make it hard to put in a decent story form.