<!--
.. title: Part 12: The WAL ate my disk, the bucket ate my keys, and the NAS stopped posting.
.. slug: the-wal-ate-my-disk-the-bucket-ate-my-keys-and-the-nas-stopped-posting
.. date: 2026-10-10 19:44:27 UTC
-->

## TL;DR

- Three separate [CNPG](https://cloudnative-pg.io) failures, each at a different layer: WAL archives filled local disk when shipping stalled; a restore job refused to honour valid IRSA credentials; and the NAS stopped posting entirely.
- The WAL lesson: local disk budget must include the failure window of archive shipping. Coupling, not capacity.
- The credentials lesson: I refused static IAM user keys on principle, walked away from S3 as a recovery path, and kept S3 only as an extra copy layer. [ZFS](https://openzfs.github.io/openzfs-docs/) snapshots to a separate zpool are the real recovery path now.
- The NAS failure was self-inflicted: my own future-proofing upgrade (newer CPU + HBA) is what stopped it posting. Survivable only because critical workloads had already moved off it.

Here are the CNPG headaches in detail. Not the drill version. The real ones, and not one incident either: three separate failures, each with its own personality, each hitting a different layer. I'm telling them in the order they arrived.

Some setup, because war stories make no sense without it. CNPG (CloudNativePG) is the operator that runs the live [PostgreSQL](https://www.postgresql.org) in my cluster. It stores the database data where [Kubernetes](https://kubernetes.io) says to store it, and it ships WAL archives and base backups to object storage, S3 in my case, over an IAM role the cluster assumed. On paper, that's a tidy design. The database lives on fast storage, the history lives somewhere cold and cheap, and the operator glues them together. I built all of that, it was green, and for a while it was fine.

Then the WAL started eating the disk.

**The disk that filled itself**

The detail I hadn't appreciated about CNPG is that WAL accumulates on local storage and only gets cleaned up once the archiver has successfully shipped it. That sounds obvious when I write it out, but it means the disk fill isn't a backup problem in the usual sense. It's a coupling problem. If the shipping side stalls, even briefly, the write side keeps going, and the two of them meet somewhere ugly: local disk full, PostgreSQL unable to write, and the operator in a state I'd describe as technically alive.

```mermaid
flowchart LR
    PG["CNPG pod<br/>(PostgreSQL)"] -->|"writes data + WAL"| DISK["Local storage"]
    DISK -->|"archiver ships"| S3["S3:<br/>WAL archives + base backups"]
    S3 -.->|"shipping stalls"| WAL["WAL accumulates,<br/>nothing cleaned up"]
    WAL -.->|"coupled to the write side"| FULL["Local disk full -<br/>PostgreSQL unable to write"]
```

The first fix was almost embarrassing: larger disks, reboot the pods, done. No heroics. That's worth saying out loud because the instinct when a database stops writing is to assume a deep, dramatic cause. Sometimes it's a disk that needs to be bigger.

But the underlying lesson is the boring one: your database's local storage budget is not just sized for your data. It's sized for your data plus however long your archive shipping can quietly fail before anyone looks. The failure mode is invisible right up until it isn't.

**The restore that didn't trust my credentials**

Now the part that actually hurt.

When the recovery had to happen for real, I reached for the documented CNPG restore path: bootstrap the cluster from a backup, restore from S3, replay the WAL. The cluster, running as it did with IRSA, assumed an IAM role through its service account. That's how the *backup* had always worked. The restore is a different job, though, and here's the irony I've been chewing on since: the restore process did not honor the IRSA environment variables. The role that backed the data up without a single stored credential was not available to the job restoring it.

I found this out the way you find out these things, which is at 1am with a half-restored cluster and an error message about missing credentials. The backup existed. The bucket was reachable. The data was all there. And the restore job sat there refusing to read it because the thing doing the restoring was not the thing that had been granted access.

There's a homelab version of "finding out at 1am," and it's almost worse. In a datacenter, the error message finds you. At home, nobody pages you: the cluster fails overnight, you sleep right through it, and the way you find out is the next day, when the inbox is silent - no new mail, not even spam. When the spam is the last canary standing, that's your monitoring doing its level best.

The documented escape hatch, at every step, was the same: hand the restore job a static IAM user key, a long-lived credential, and let it pull from the bucket. I want to be honest about what I did here, because the honest part isn't "I fixed it," it's "I refused."

I refused. That env-var handling was a bug in the restore path, not a limitation of my setup, and papering over it with a static key would have meant accepting a design I knew was worse: long-lived credentials in a place I'd spent the [Proxmox](https://www.proxmox.com) era learning not to put them, just to work around a process that couldn't read the identity the platform was already handing it. So I didn't do it. I escalated it in my head to a matter of principle, I ground against it, and I'm not going to pretend that grind was efficient. It was expensive. It was also, I still think, correct - and it's a discipline I'd been building since the Proxmox days, long before this cluster existed.

Because the real fix turned out to be walking away. That was the strategic decision: the S3 restore path had failed me in a way that told me something structural about it. It wasn't that I couldn't make it work; it's that making it work required abandoning a credential model I hold to for everything else, for this one path, this one tool. So the question stopped being "how do I make this restore work" and became "should this restore path exist at all."

**The honest retreat: ZFS snapshots**

And that's how the backup strategy changed, and not in the direction the textbooks point.

I walked away from [barman](https://pgbarman.org) cloud and the S3 pipeline for recovery entirely, and moved onto truenas-csi snapshots proper: ZFS snapshots of the CNPG volumes, croned to a separate zpool. I want to call this what it felt like: a retreat with scars. I didn't leave S3 because it's bad storage. It's eleven nines of durability, it's excellent at its job, and the irony of the whole episode is that I had a perfectly durable copy of my data that I could not reliably *download*. Storage that can't hand your data back when you need it isn't a backup, it's a mausoleum.

So the S3 pipeline now has a different, humbler job in my setup: not recovery, just an extra copy, the layer I reach for when the layers above it have already failed and there's time to fight the tooling. The thing I trust to actually save the live database on a bad night is a ZFS snapshot on a different pool, one I can restore in minutes without consulting anyone's key store. Backup is a verb. Restore is the exam. I picked the path where the exam is one I can actually sit.

**The unplanned version**

And then the NAS, the machine holding all of that, refused to post.

Out of nowhere. Kinda. Because the trigger was, technically, me. I'd swapped in a newer CPU and added an HBA card to hang the disks off - a future-proofing move, part of a plan to eventually migrate to another server. After that upgrade the machine couldn't initialize all of its RAM, and after enough fault-tracing the HBA came out of the picture entirely, and it has stayed out since. So it wasn't a slow trickle of SMART errors I'd ignored, and it wasn't the machine falling over on its own schedule. It was a box that worked, got upgraded in the name of being more future-proof, and stopped posting in the space between two boots. The hardware didn't telegraph the failure. My upgrade calendar did.

The last two posts were about migration as a planned exercise: an API deprecation with a clock on it, a zpool swap scheduled because the storage layer was getting touched anyway. This was the other version, the one you don't schedule - except this time I'd half-scheduled it myself, and the upgrade meant to reduce the future risk was the thing that caused it. The machine that holds your data and your snapshots and your restore path is suddenly a brick, and the cluster keeps asking it for volumes.

I've called this the tech surgeon moment before, and it held: don't panic, stabilize, work the list. But there's a difference between that discipline as a philosophy and needing it at 1am while a live database waits, and the difference is that this time the list had already been written. The critical workloads were already moving off the NAS and onto the three-node cluster, not because I'd predicted the failure, but because the SPOF had been bothering me since long before the NAS actually died. The NAS dying didn't force the migration. It confirmed the one I'd already started.

That's the part I want to leave you with, because it's the least dramatic and most useful thing in this whole story: the NAS failure was survivable not because of anything I did that day, but because of the decision I'd made weeks earlier that the NAS should never be the only copy of anything. When it went down, the work moved. The snapshots were on a separate pool, the critical data had a second home, and the restore paths had been tested. None of that made the failure fun. It just made it an evening instead of an era.

And there's a quieter corollary, the one the HBA taught me: hardware failures in a homelab don't always announce themselves, and they don't always arrive on their own schedule either. Sometimes they arrive *because of you*, on the exact afternoon you were trying to make things better. The discipline that saved this situation wasn't predicting the failure. It was assuming the NAS could disappear on any given day and making sure that day would be survivable - regardless of whether the cause was a dying disk, a bad slot, or a well-intentioned upgrade of my own.