The Two Trad Towers

I run a home server. It holds the family photos, the movies, the backups, and a dozen small services that everyone in the house has quietly come to depend on without ever thinking about it. One evening in August I changed a single environment variable on one container and lost the entire machine for five hours.

The config change was harmless. What cost me the evening was my reaction to it.

What it looked like while I had no idea

The web interface would load, look completely normal for about four seconds, and freeze. SSH alternated between refusing the connection and timing out, with no pattern I could find. Plex went offline, came back, went offline again. Ping ran at fifty to eighty percent packet loss.

That packet loss number is the part I should have paid attention to. Total loss would have meant something was simply down. Fifty to eighty percent sustained, for hours, means something is actively fighting, and I had never seen my network do that before.

Four hours of wrong theories

In order, and with rough time spent on each:

• The container did it. Ninety minutes. I removed the container, removed its config, and removed two others for good measure. No change, which should have told me something and didn't.
• The disks are dying. Forty minutes of reading drive health stats with real dread. The drives were fine. They have been fine the whole time.
• The network switch is failing. An hour, including physically swapping a cable and a port. I want to be clear that I did this while the actual answer was visible on a screen in the next room.
• The router hates me. Forty minutes of rebooting network equipment, which is the troubleshooting equivalent of shouting.

The answer was in my router's client list the entire time. There were two devices with my server's name on them. I had one server. I looked at that list at least three times during the first hour and my eyes went straight past it, because I was looking for something to be missing, and instead something had been added.

The actual cause

The file that defines the network bridge had been cut off partway through a write, which is exactly what you would expect from losing power mid-save. The first six lines survived. Everything that named a physical network card was gone.

With no physical card to inherit an identity from, the operating system did the only sensible thing available to it and generated a random one. Every network device has a hardware address that is supposed to be permanent and unique, and most home networks quietly depend on that. Mine handed out my server's fixed IP address based on it.

New hardware address, new device as far as the router was concerned. It refused to hand over the reserved address to this stranger and offered a different one instead.

Then it got worse without my help

The server kept insisting on the address it remembered. It would claim the address, hear its own claim echo back off the network, conclude that some other machine was impersonating it, defensively give the address up, delete its route to the internet, and immediately ask for the address again.

Every five seconds. For hours. My server spent the evening in an argument with itself, and every service on it failed in that same rhythm, because nothing can hold a network connection open on an address that keeps evaporating.

The fix

I rewrote the config file by hand at the console with the network definitions restored, and set a static IP address instead of relying on the router's reservation. One reboot and everything came back.

The static address is the part worth copying. A router reservation is a promise made by the router, and it is keyed to a hardware address I had assumed was permanent. A static address is a decision the machine makes about itself, and it survives the exact failure that caused all of this.

If you run something like this, check three things

None of these take more than a few minutes, and any one of them would have saved me an evening.

• Is your critical infrastructure on a static IP, or a router reservation? Reservations are easier to set up and they fail in ways that are genuinely hard to diagnose. Anything that other machines depend on finding should decide its own address.
• Do you have a backup of your boot configuration? Mine lives on a USB flash drive, which is a normal design for this kind of server and also the least durable storage in the building. I now back it up on a schedule. I did not before, and rewriting that file by hand was luck rather than preparedness.
• Can you see your router's client list? Learn what normal looks like on it while nothing is wrong. An unfamiliar device is a signal you will only recognise if you know the baseline.

Total cost: five hours, one deeply confused household, and a lasting suspicion of any troubleshooting step that begins with "just reboot it."