TL;DR

• Backups and restores are different operations: a backup is a copy, a restore is the whole system rebuilt under pressure. Green dashboards prove nothing about the second one.
• My backups worked while my restores didn't exist. The gap between those two facts is where the worst evenings live.
• The fix is boring by design: a written restore procedure, drilled on a schedule, on the same class of metal that will be in the room when it's real.

Time for the full sermon, and it opens with a confession that will sting anyone who's ever run a homelab with confidence: for most of this cluster's life, my backups worked and my restores did not exist. Backups can work and restores can fail miserably, and the gap between those two facts is where I spent some of the worst evenings of this whole project.

The category error everyone makes

A backup is a copy. A restore is the entire system, rebuilt, under time pressure, from whatever the copy actually contains. These are not the same operation wearing different hats, and the industry's relationship to them proves it: everyone has backups, and almost nobody has proven restores. The backup job shows green in the dashboard. The restore path has never been run. The dashboard cannot tell you that the snapshot you've been dutifully taking every night doesn't actually boot.

And the failure isn't exotic. It's the mundane things: the restore procedure assumes a tool you no longer have installed. The snapshot references a machine whose MAC address is baked into some config. The backup is complete but the process to use it lives in your head, and your head, at 1am, mid-incident, is not a durable storage medium.

The cloud papers over this so smoothly you forget it's papering. When you run on managed services, "restore" is often literally a button, or a support ticket, and the provider's DR runbook is someone else's tested problem. The moment you're on your own metal, every assumption that made the button work - the provider's tooling, their network, their ability to hand you a fresh instance with your tags on it - evaporates. And self-hosting tools are worse: a huge amount of the self-hosted ecosystem silently assumes either that you're in the cloud, or that you're not, and are happy handing over API keys to something like a hosted backup SaaS to make the problem go away. I'm not. That's a decent chunk of why this blog exists.

The cold start I drilled, and the restore I didn't

There's a variant of this that bites even people who do test restores, and it's the cold start: the scenario where nothing is running at all. Not "the database pod crashed, restore it" - the full "the machine is off, the cluster doesn't exist, build everything from the state you have."

Here's the irony I'll own outright: cold start is the one thing I tested. Repeatedly. Rebuilding the cluster from nothing was drilled into me early - wipe the machine, boot it, bring the cluster up from whatever state I had, again and again, until it stopped being scary. That habit was probably the most valuable thing the Pi-and-NUC era gave me, and it's a large part of why the McGyver cluster was rebuildable at all. But there's a trap hiding in that, and I fell into it: a rehearsed cold start quietly becomes a substitute for a tested restore. "I can always rebuild from scratch" feels like a recovery story. It isn't. It's a disaster-recovery story for the day you don't mind the downtime, the data loss, and the evening of yak-shaving. The scenario where the cluster is fine and only one component is dead - that's the scenario I never rehearsed, and it's the one that bit.

Adeptus Mechanicus recovery

The resulting recoveries deserve their own paragraph, because the phrase that comes to mind when I remember them is one I'm only half-joking about: successful recovery was an act worthy of the Adeptus Mechanicus. Incense and liturgy. The right incantations typed in the right order at the right time, a libation of coffee, and a genuine, sincere prayer that the thing came up - because the procedure wasn't really a procedure. It was a sequence of steps I'd discovered worked, once, in a specific order, dependent on machine states I couldn't reproduce and timing I couldn't control.

That's what an untested restore path actually is: liturgy. You're not operating a system; you're performing a rite, and the system's willingness to come back is a matter of faith, not engineering. Every time it worked, it worked because of things I couldn't name. And every time it worked, I learned nothing - because success without understanding is not a tested process, it's a lucky one.

The hardware answer, which is really a philosophy answer

The three EliteDesk machines with i5-9500T chips came in recently, and it's tempting to tell the story as "I bought better hardware and the problems went away." That's not the story. The hardware arrived because the cluster had been through situations where recovery was liturgy, and I refused to keep running a household on faith.

The EliteDesks changed the math in two specific ways, and neither is raw horsepower:

1. Fungibility. Three identical machines - the property from last post. A restore that fails on one machine's quirks is a restore that fails identically on all three, which means testing it once tests it three times, and means the recovery procedure is the same procedure regardless of which box died. The Adeptus Mechanicus liturgy existed because every machine was a unique ritual. Identical machines make the ritual a procedure.
2. Enough nodes to test against. You cannot rehearse a cold start on the cluster you're afraid to break. With three identical nodes, one of them is always expendable. Wipe one, restore onto it, prove the path, wipe it again. The test cluster stopped being hypothetical and became a machine with a power cable.

So the hardware didn't fix recovery. The hardware made practicing recovery cheap enough that I actually do it, which is what fixed recovery.

The principle

Which brings me to the actual content of this post, because it's a principles post, and the principle fits in one line:

Backup and restore are one operation, and restore is the half you must prove.

Concretely, what that means for me now:

• A backup that has never been restored is a hypothesis, not a backup. Green dashboards count for nothing. The only unit of evidence that matters is a restore you've personally completed.
• Restore includes cold start, but cold start is not a substitute for restore. They're different scenarios: rebuild everything from a dark room, versus bring one dead component back while the rest of the cluster is fine. I drilled the first relentlessly and skipped the second entirely - and the second is the one that bit.
• The procedure must be written and boring. If it involves incantations, discovered sequences, or anything that sounds like liturgy, it isn't a procedure yet - it's a memory that will fail you exactly once, at the worst time. Write it down. Then follow it by rote during a test. Then rewrite it, because the rote-following will find the gaps.
• Schedule the drill. Not "when I get around to it." A restore test that isn't on a calendar is a backup you've decided not to verify. Mine is on one.

Where this is going

The next post introduces a player that's been offstage since the beginning: the NAS. It runs TrueNAS, it holds everything the cluster actually cares about, and its connection to Kubernetes is a piece of open source whose API upstream has scheduled for decommissioning. That's a story in its own right, and it's where the principle from this post earns its first real payoff. The war story after it is the exam: a CNPG (CloudNativePG) disaster, in detail, where my cold starts were drilled and boring, my restores were neither, and the first component-level restore that had to happen for real found out what my backups were actually worth. So did I.

If you take one thing from this post, take the line: backup is a verb, restore is the exam. Before the exam: the NAS.