Nobody Was in Florida
My production server drowned in its own stuck processes a thousand miles from the nearest human, stayed dead for two days, and the business numbers went up anyway. Then we taught it to reboot itself.
Monday morning I opened my laptop to red errors and asked my AI partner to run a systems check. The answer came back in three parts. My email app was frozen. My morning automations had failed. And my production server, a Mac that lives in Florida while I was in Indiana, was running a load average of 128 on a machine whose normal is six.
Load average, if you don't speak server, is roughly how many things are trying to run at once. Six is a busy morning. One hundred twenty eight is a machine drowning.
Here's what happened, as best the logs can testify. Something wedged the mail apps overnight. Every fifteen minutes, my scheduled jobs would reach out to those apps with a little script, and every one of those scripts hung forever instead of finishing. They piled up like cars behind a stall. A hundred stuck processes later, the machine stopped answering the network entirely. Not slow. Silent. We pinged it from a second machine sitting on the same shelf and got nothing.
And nobody was in Florida.
The smart plug that can't save you
The classic fix for a dead remote server is a smart plug. Kill the power, restore the power, the box reboots, everyone goes back to breakfast. I'd even considered putting one in.
It would have done nothing. The server is a laptop. Pull its wall power and it shrugs and switches to battery, still frozen, now also discharging. The one remote recovery everyone reaches for first was defeated by the machine having a built in UPS. I did not appreciate that until Monday.
So the honest options were: wait two days until I could fly back and hold the power button, or hope the machine's own kernel eventually killed the runaway and clawed its way back. We planned for the first and hoped for the second.
The part that surprised me
Here's the sentence I keep turning over: the business didn't notice.
The donor emails I needed that morning already existed, sitting safely in drafts. The day's call sheet got built by hand from the same data files, in about a minute, because the data lives in synced storage and not on the dead machine. The command center software that normally runs in Florida got started on my laptop with one command, because the whole thing lives in version control and doesn't care where it runs. My monthly numbers got refreshed from the official reports and every single metric came in ahead of pace.
A production server was hard down for the entire day, and the measurable output of the operation went up.
That's not luck. That's two months of a specific discipline: nothing important lives in only one place, nothing gets to say it's fine without proof, and every failure we ever hit gets a rule written about it. The server dying was the final exam for all of it, scheduled by the universe with no advance notice, on a Monday, during month end close.
Teeth, not alarms
The last Build Note was about preflights, the idea that nothing should get to say ready until it proves someone is listening. This week taught the sequel: a machine that can detect it's dying should be allowed to do something about it.
So the fix wasn't a louder alarm. Alarms need someone in Florida. The fix was three layers of teeth.
First, the health snapshot the server publishes every fifteen minutes now counts those little scripts. A pileup is the earliest symptom, visible an hour before the drowning, and now it's on the chart.
Second, the snapshot job got surgical powers. If load is high and the script count is absurd, the exact double signal from Monday, it kills the stuck processes itself and reports what it amputated. Narrow, loud, and limited to the three programs implicated. Self surgery before the kernel goes under.
Third, the dead man switch. A tiny watchdog now runs with root privileges, below all the userland trouble, sampling load every five minutes. Three consecutive samples over threshold, meaning fifteen straight minutes of drowning with no recovery, and it reboots the machine. Everything on that box is built to come back on its own after a restart. So the worst case outage went from two days and a plane ticket to about fifteen minutes and a log entry. There's a disarm file for maintenance days, because a watchdog that shoots you during a big build is worse than none.
We tested the logic before trusting it, obviously. A single spike doesn't fire it. Three sustained breaches fire it exactly once. The disarm holds. The preflight religion applies to the watchdog too.
What I'd tell you
If you run anything unattended, ask the question before the universe asks it for you: what happens when this machine stops answering and nobody can touch it?
If your answer involves a smart plug, check whether the machine has a battery. If your answer involves an alarm, ask who's listening at 7 AM on a Monday. If your answer is nothing, your answer is a plane ticket.
Mine is a watchdog now. The server comes back Wednesday when I press the button, and after that, it never needs me to press it again.
MG
For the engineers.
That was the operator's view. Underneath it sits a small watchdog daemon, a self surgery script, and a health snapshot job, wired together to turn a dead machine into a fifteen minute outage instead of a two day one. I wrote the full engineering breakdown as a companion piece. If you want the wiring diagram, that one is for you.
New notes land first in the weekly letter.
You're in. Next note lands Friday. ✦