Worst Network Ever

One day in the mid 1990s, I got a call from a company looking for "help with their network". Ok, cool, that's what I do. I told them my rate, they made the token grumble, and said come on out.

I get there, and things are a mess. The office is down, none of their computers can see their server. Pretty obviously a cabling problem. Good, I am good with those.

This particular office used a Coax Ethernet (10BASE-2, for those who love cryptic names). Coax Ethernet was wired in a "daisy chain" form, a cable runs from the back of your computer to two other computers, and all the systems are in this chain. The two end computers have a "terminator" to provide a proper end to the network. If the cable breaks ANYWHERE, because both ends are no longer properly terminated, the entire network is down.

Coax Ethernet networks was a solution that "worked" when it was developed, but it wasn't designed for big offices. That "one break anywhere breaks everything" thing is really a time-bomb waiting to go off. At least once, it was someone thinking, "I don't use my computer", so they unplugged it and disconnected all the cables from the back. A few minutes later, they heard the screaming around the office about "THE COMPUTERS ARE DOWN!", and quietly sung their own praise for not using those dang things.

Around 1990, the Ethernet standards were enlarged to include a new wiring system called twisted pair. Instead of going computer-to-computer in a very fragile design, twisted pair systems went from each computer to a central hub. While twisted pair wire was relatively cheap compared to coax, the cost of the hubs early on was astronomical -- several hundred dollars a port early on (the first one I put in myself was "only" about $80/port!). It was a big investment, but people loved it because it helped isolate problems. Anyone could unplug their computer without impact, and if they damaged a wire that caused problems throughout the network, it could be isolated at the hub, found and disconnected without impacting the rest of the wiring plant. By 1992, I only installing coax in some very special cases (just a few machines, all in a row, no need to bury wire in walls, etc., where the physical layout was so easy to work with and simple enough that the reliability wouldn't be an issue), and no one else was, either.

So, around 1995, this company calls me out. Not only do they have a coax network, but it is a large number of offices, wires are buried in walls. Worse, they use an "alternative" wiring system, it was very expensive in the day. It was intended to get around the problems of coax networks without using a hub (in fact, I think it predates twisted pair Ethernet, so "A" for effort). I wish I remembered the name of the product, but it was similar to EAD sockets but I don't think it was compatible with the EAD standard. They eliminated the "T" connector at the back of every computer by using a special drop cable from the wall to the computer, which was actually TWO chunks of coax, and at the other end was the magic jack, which would solve the "unplugged cable" problem by bypassing the loop to the back of the computer and completing the circuit between the two segments of coax hitting the back of the computer. When you plugged the drop cable into the jack, the wire in the wall was "opened", routed up the drop cable, back down, and along to the rest of the computers in the segment. Neat idea. Plus, they made it "tool free" -- you could attach their wall sockets to the coax in your wall with just a wire cutter, no tools were needed, everything was connected using just finger pressure.

Neat idea...but they implemented it completely wrong.

Since the coax was attached to the jacks using just finger pressure, the connections weren't very strong, but that's ok, they are supposed to be on a wall plate, immobile in the walls. Except, the person that did this didn't do that. The wire came out of the wall, into the wall plate (which was laying lose on the floor), then out of the wall plate back into the wall. So, poor mechanical strength, exposed to people moving stuff around it.

This system was electrically imperfect, the manufacturer knew this, and only recommended it for segments of ten or fewer nodes. This office had something like 40 drops attached on one segment. All those electrical imperfections added up, resulting in a network that barely worked at the best of times.

So...when I get there, I grab my diagnostic tools. There are two tools I usually used to test coax networks, an ohm meter and a "Time Domain Reflectometer" (TDR -- or "cable scanner") -- basically a kind of radar gun that sends a quick pulse down the wire, and depending on how it comes back, it can tell me if the wire is shorted or broken, AND how far away it is to that flaw. (if the wire is properly terminated, nothing comes back, which is as it should be).

Well, the TDR is of limited use -- you have to open the coax segment up, and test each "subsegment" you just created by splitting it. Problem is, this EAD-like system is designed around NOT opening up the segment. So putting the TDR in requires breaking the cable (at a "wall" plate), attaching the TDR and seeing what it says.

That's ok, the ohmmeter is, I have found, REALLY good at testing coax. If both terminators are in place, it's 25ohms. One terminator? 50 ohms. Short somewhere? Zero. Sure, the electrical purist will say, "That's DC testing! That's doesn't really count!" True, but if the right parts are used properly, DC testing is really really good at telling you how things are going to behave and helps you find the problem.

Well, these are not the right parts and they are not used properly. The ohmmeter comes back with a really weird value, but translation is basically, "this is messed up".

Ok, I'm going to have to start splitting the network, running tests in each direction, and chasing problems down. Problem is, when split, the TDR is even more confused than the ohmmeter. It's a mess. But slowly, I tracked down where the problem was, fixed it, and put it back together. I used my portable test server to verify that the network was working again.

At this point, I find out they have taken down their server, because when things don't work, reboot, right? Their server was running Novell Netware, which I'm really good with. I say "ok, let's power this thing up", and the person I've been dealing with pulls out a stack of papers, which are the startup instructions for their server. NO! Netware, when properly set up, you just turn on and it does everything... But no, they seriously had two or three pages of instructions of things to type at the command line to start the thing up. And it turns out, they aren't correct, so I have to open the machine up to identify the network card.

When the machine is open, I discover it is a machine of pretty powerful capabilities, quite new, overkill, even. But I discover the processor fan is laying on the bottom of the computer case..so the poor computer has been overheating badly. Worse, I dont' see any sign that the fan was ever properly attached. So I fixed that, found the info I needed, built up the startup scripts so the machine could be just powered on, and fire it all up to test it, and sure enough, it comes right up.

At this point, I'm sitting on the floor, next to the server, waiting for it to finish booting. I ask the person working with me, "What does your company do, anyway?"...kinda odd, but I've been diving into their problems, and I really don't know a thing about my new client. Usually, I try to learn a bit about them first, but they were down, so we had different priorities.

"We are computer consultants, we set up networks for businesses", he tells me.

I'm going to admit this is probably my most unprofessional moment of my self-employment life, but I lost it, I was sitting on the floor at the time working on the server, I literally rolled backwards and started laughing. "YOU ARE MY COMPETITION??", and I laugh some more.

Literally the worst designed, worst implemented small office network I've seen, built in-house by a guy who was out at that moment, selling his services to others. I so wanted their customer list.

I told them their cabling plant was a nightmare. Good products, but implemented completely wrong, and thus, making problems they were intended to prevent. 10BASE-T was very much a thing, they had no excuse not to be using it. And I went on my way.

Turned out they were a slow pay. A couple months after my bill was sent in, they still hadn't paid. I don't tend to get too pushy with customers, things often work out just fine. If this customer never paid me, I would be slightly mad, but happy to never deal with their problems again. Besides, that cable plant was so bad, I was sure I'd hear from them again. And yes, I did..."Our network is down again"
"funny, you haven't paid me from the last time"
"uh...we'll cut you a check as soon as you connect me back up to the printer"

This cycle repeated several times, and quickly lead to me changing my billing policy -- coax repair was charged at double my normal hourly rate, and a one hour minimum as I explained here.

Worst Network I've seen. Screwy customer, too. It turned out they were big into local government contracts. They apparently had been certified as "minority owned", which greased their way into many government deals. Funny thing, though...never met this "minority" owner...or even any high managers that were not white men. Obviously, checking the right qualification boxes mattered more than results to their customers, so ... yeah, no, I really didn't want their customer list after all.