It's been a while since I posted anything, but I wanted to try to get back into posting stuff. What better way to start than another tech horror story! Currently I am working as a Senior Network Engineer, a recent promotion I am quite proud of. One of the reasons I earned my current title is due to my involvement in a lot of troubleshooting sessions and overall network improvement. With that, it makes sense strange issues tend to land in my lap. One of these issues that happened to land in my lap was an OSPF Flood War. Not my first one, so I figured easy enough. We had four devices showing up in logs with intermittent connectivity. With a bit of luck and some careful planning, I managed to get on each one and fixed what was showing as duplicates. With solid connectivity restored I closed out the tickets, only took me most of a day since it was a remote site.
The next day, I got a message about another OSPF Flood War message for the same four buildings. Connectivity was still solid. All four buildings had no errors, but the upstream did. I went over the same four buildings and the upstream device multiple times for most of the day. Still no luck what was making the messages show. Digging through everything I could find, I decided to ask around.
One of the new engineers I had helped earlier working on getting phones setup at the same remote site, but on a completely different building. This building did not show up in any of the logs. With no other ideas, I got into the device for the building. It finally revealed itself when I went into the switch. The voice network is on a separate VRF and the point to point VLAN on the uplink was in the voice VRF and the downstream one was on the default. Thus, two VRFs leaked into each other across one device that never showed up in any errors logs. Two days of staring at devices and it was that simple. It was a nice and easy fix of just moving things into the appropriate VRF.
With all that, moving forward I am hoping to work on some more technical posts focused on network, since I guess that's my life now. I may also consider enlisting the help of all the new fancy AI tools and if I do, I will disclose that on the post. Just a consideration because sometimes my thoughts are too boiled down and dry to make a coherent post as opposed to just like a bullet list of information. Just a thought. Hope you enjoyed my story!
