Five Weeks Dark. I Built a Night Watch.

The iMac in Indianapolis went dark on June 11 and stayed dark for five weeks. I was in Florida the entire time. I had no idea.

That machine is not a spare. It is the backup target for the Mac that runs everything I build. Every night at two in the morning, the production machine in Florida mirrors itself twelve hundred miles north to that iMac. For five weeks, that job was writing into the void, and nobody was watching.

Five weeks in the dark

The root cause was a chain, not a single thing. A power failure knocked the iMac out. Automatic restart after a power loss was turned off, so when the electricity came back the machine simply stayed off. Somewhere in the same window the internet there failed over to its backup router, and the backup router was dead. I could not reset it from Florida, because the only way to reach it ran through the thing that was broken.

That is the part that stayed with me. It was not that a machine failed. Machines fail. It was that nobody outside that room could tell which layer had failed. The Mac, the router, or the internet itself. From a thousand miles away all three look identical. They look like nothing at all.

So I flew up. And the actual bug I went to fix was not the power. It was the blindness.

Ten minutes of hardening

The first repair took ten minutes and a keyboard.

One command told the Mac to never sleep, to never let its disks sleep, to restart itself automatically after a power failure, and to accept a wake signal over the network. Automatic restart is the headline. That one setting is the exact failure mode that caused the outage, and it ships turned off. I also switched Remote Login back on and disabled key expiry on my private network, which kills a sneakier version of the same outage: the power is fine, the machine is fine, and the credential for reaching it just aged out.

Then I verified it instead of assuming it. I connected from my traveling laptop, then ran a ping from the Florida machine across the private network to the Indiana machine and watched it come back at zero percent loss. The nightly backup was proven working end to end, not hoped to be working. Verification, not vibes.

A watchdog outside the blast radius

Here is the rule I paid for the hard way. A monitoring system running on the machine it monitors is a diary, not an alarm. It tells you what happened, later, and only if it survived.

So the watchdog had to live somewhere else entirely. It is a Raspberry Pi 4 sitting in Indianapolis. Different machine, different city, different power grid, different internet provider from the production Mac in Florida. If Florida goes down, Indiana notices.

Cost: about sixty dollars of hardware already sitting in a drawer, plus a 128GB memory card delivered the same day. Total build time, including every detour: roughly two hours.

No monitor was ever attached

I never plugged a screen into that Pi. Not once, not for a second.

You can bake the entire configuration into the card before the machine ever boots. The imaging tool writes the hostname, the user account, and the remote access setting straight into the boot partition. On first boot the Pi configures itself and announces its name on the local network. No keyboard, no monitor, no video cable.

The detour is worth writing down, because it is the kind of thing that makes people quit. I missed the remote access toggle on the first flash. The Pi came up, answered a ping in five milliseconds, and then refused the connection. Host alive, service absent. The fix is not to reflash the card and start the evening over. It is to pull the card, create an empty file named ssh in the boot partition, eject it, and reboot. The Pi looks for that file on startup and switches the service on for you. Sixty seconds, beginning to end.

Which taught me something I did not know that morning: connection refused is good news during a headless boot. Refused means the host answered you and declined. Something is alive in there. A timeout is the bad one.

Nine eyes and a butler named Carson

The watcher itself is Uptime Kuma, self hosted and open source, running in a single container that restarts itself whenever the Pi reboots. Nine monitors, checking every sixty seconds.

Seven are ordinary web checks: my production command center plus six public properties. The interesting two are pings across my private network, one to each Mac.

Those two pings are the triage layer, and they are the whole reason the evening was worth it. Site down but the private network up means the machine is fine and the tunnel or the CDN broke. Both down means the machine itself is gone. That is a diagnosis I can now run from my phone, in a parking lot, in about four seconds. Five weeks ago I could not run it at all.

There was a bonus I did not plan for. Every web monitor also tracks certificate expiration, and within minutes it flagged one property whose certificate renews in forty one days. If that automatic renewal ever quietly fails, I hear about it before a single visitor sees a browser warning.

The alerts route to a Telegram bot named Carson. The household staff naming convention has been running for a while now. Jeeves runs Florida, Alfred watches orders, Benson dispatches, Florence strategizes. Carson keeps the night watch. His first transmission went out at 19:11, a test message. If everything goes the way I want it to, that is also his last message for a very long time.

Two things will eat your evening when you wire up a bot like this. An unauthorized error means the token dropped a character somewhere in the copy. A missing chat means the bot is not permitted to start the conversation, so you have to message it first.

Six days later it earned its keep

Yesterday morning, six days after the build, the same iMac dropped off the network again.

Different cause this time. Not power. The machine had been up for forty two days. Its network client updated itself overnight and needed to be reauthorized, and when I reauthorized it, it rejoined as a brand new node with a brand new address. Every script pointing at the old address was talking to a ghost.

The system caught all of it. The backup job went red, and the watch told me which layer had failed and, just as usefully, which layers had not. I was on site, so I fixed it before lunch: repointed the alias, accepted the new host key, restarted the job, and had a clean full snapshot by 9:27.

The part I am most pleased about is the part nobody sees. The data was never at risk for a second. A fallback I built earlier copies the critical files into iCloud every night, and it fired exactly as designed while the primary path was broken. Layers. A night watch is not one thing that never fails. It is several things that fail at different times.

Observability is not a big company luxury. In one evening, for the price of a nice dinner, on hardware that was already in a drawer, a one person operation got monitoring in a separate state, automatic recovery from power loss, remote hands, and a butler who texts. And none of the unlock was the hardware. The whole build ran headless and conversational. Flash a card, paste a command, read the error out loud, fix it, keep moving. The machine never needed a screen, because the operator already had one.

Something is awake now while I sleep. That is the entire product.

MG