TL;DR

• Storage classes here are picked by failure mode, not spec sheet: the question isn't "how fast?" but "what happens when it dies?"
• Local tier: Longhorn on the node SSDs at RF=2 - data plus one replica, one-disk tolerance, the same as RAID5. Critical workloads (CNPG databases, mail, Authentik) live here so a node can die and they keep answering.
• NAS tier: NVMe-oF for block, NFS for files - fast enough, and easy offline backups. Dies with the NAS, for an afternoon, which is acceptable for the merely-inconvenient tier.
• Bucket in the basement: SeaweedFS serving the S3 shape locally, offsite copies staying genuinely offsite. Live and offsite remain separate concerns. Once again.

The recent posts were fires. A WAL directory quietly ate a disk. A restore job refused to believe my credentials, and I walked away from S3 as a recovery path rather than hand over static keys. And a future-proofing upgrade left the NAS unable to post, on an afternoon I had personally scheduled. None of it ended in catastrophe, and the reason wasn't luck - it was decisions made before the fires: critical workloads already moved off the NAS, tested restore paths, backups treated as a verb.

This post is the map of where those decisions landed. No war story - the inventory. Where the data actually sleeps, on what hardware, at what replication factor, and why my storage classes are named after criticality instead of performance. If the recent posts were the fires, this is the fire map: the picture you want pinned to the wall before the first smoke, not sketched from memory after it.

Pick storage by failure mode, not by spec sheet

The instinctive way to shop for storage is performance: IOPS, latency, price per gigabyte, whichever benchmark wins. I've stopped shopping that way. Around here, the question a storage class answers isn't "how fast is it?" but "what happens when it dies?"

That inversion changes everything downstream. A workload that must never stop needs storage whose failure mode is "one node gone," not "one box gone." A workload that would merely sulk for an afternoon can happily live behind a single NAS. Backup tooling that insists on speaking S3 doesn't need S3-the-cloud-provider; it needs the shape of S3. Every choice below follows from asking the failure question first. Performance comes second - and honestly, none of these tiers is slow enough to complain about.

The local tier: Longhorn on the node SSDs

Each node carries local SSDs - 256GB each, decent throughput, decent latency, the kind of disks that make a database feel good about its life choices. The pack even had a Kioxia in it, which is the parts-bin lottery paying out for once. I'll take my wins where they come.

Longhorn spreads PVCs across those local disks, and the storage classes here run at RF=2: two replicas, on two different nodes. On a three-node cluster, that's the number I keep landing on - it tolerates one node loss, which is the failure Longhorn exists to absorb here, and a third replica would spend a third of the disk budget on a second concurrent failure that this cluster isn't designed to survive anyway. RF=2 is the honest setting, not the cheap one.

The critical storage class is where the must-work things live. CNPG databases for services that have to work run on Longhorn PVCs on the same RF=2 class - durability and failover, the property that a node can drop dead and the database keeps answering, because its other replica doesn't care. Mail and Authentik are on that class too. This is the post-12 lesson turned into topology: the NAS being unable to post once taught me that "it's on the NAS" and "it's safe" are different sentences. The answer wasn't a better NAS. It was a storage class whose blast radius is a node, not a box.

The NAS tier: NVMe-oF and NFS

The NAS is still very much in the story - it's just no longer load-bearing for the must-work tier. Two storage classes hang off it, and they exist for different reasons.

The first is NVMe-oF - block storage over the network, chosen for throughput. My nodes still run 1GbE links today; the NAS is on LACP at 10Gbps. Even so, NVMe-oF yields very decent throughput from those modest node links: the protocol wastes very little overhead, and with the NAS side aggregated at 10Gbps, the limit you hit is the node's own NIC, not the box behind it. For "inconvenient if down" block storage, that's more than enough.

The second is NFS - the file tier. Some services think in files rather than blocks: they want to share a directory, let someone poke at the contents directly, or generally behave like the filesystem they believe they're on. NFS is the honest answer to that. And it has a quiet superpower: offline backups of files are child's play. A file tree is rsync-able, visible, copyable - no snapshots, no zvol gymnastics. For data that wants to be files, NFS makes the backup story nearly boring. After the last two posts, I am extremely pro-boring.

The bucket in the basement: SeaweedFS

One thing post 12 established: the S3-shaped API is the lingua franca of backup tooling. Half the ecosystem speaks it fluently, and the trap was pointing that fluency at someone else's infrastructure and discovering the identity machinery mid-restore.

So the lingua franca now lives at home. There's a SeaweedFS running locally - S3-compatible, backed by PVCs on the NAS. Backup tools that want an S3 endpoint get one; it just happens to live inside my walls, on storage I control, in the failure domain I chose. The cloud-shaped API without the cloud-shaped dependency.

And the layering stays honest: live data and offsite data remain separate concerns - once again, because it keeps being the rule. Live data is in the cluster or on the NAS. The NAS gets its ZFS snapshots to a separate zpool, exactly as described two posts ago. And the offsite copy is offsite - a different location, a different failure domain, because "the house burned down" is a failure mode no amount of replication inside the house can fix.

The whole map

Which makes the current state of storage, written down at last:

• Must work - CNPG databases, mail, Authentik: Longhorn PVCs on local SSDs at RF=2, failover across nodes. Survives a node loss. Survives a NAS loss.
• Inconvenient if down - everything else: NAS storage classes, NVMe-oF for block, NFS for files. Survives a node loss trivially, because it isn't on the nodes. Dies with the NAS, for an afternoon.
• Backup layers - ZFS snapshots to a separate zpool on the NAS; SeaweedFS serving the S3 shape locally for the tooling that wants it; offsite copies of what matters.
• The rule underneath all of it - live and offsite are separate concerns. Once again.

flowchart TB
    subgraph NODES["Three identical nodes - local SSDs"]
        LH["Longhorn, RF=2<br/>two replicas on two nodes"]
    end
    subgraph MUST["Must work"]
        CNPG["CNPG databases"]
        MAIL["Mail"]
        AUT["Authentik"]
    end
    CNPG --> LH
    MAIL --> LH
    AUT --> LH
    subgraph NASBOX["The NAS - dies with it, for an afternoon"]
        NVME["NVMe-oF<br/>(block)"]
        NFSD["NFS<br/>(files)"]
        SW["SeaweedFS<br/>(S3 shape, on NAS PVCs)"]
        ZP["ZFS snapshots<br/>to a separate zpool"]
    end
    OTHER["Everything else<br/>(inconvenient if down)"] --> NVME
    OTHER --> NFSD
    LH -.->|"snapshots"| ZP
    SW -.->|"offsite copies"| OFF["Offsite<br/>(different failure domain)"]

What this buys is simple: the fires from the last two posts now have bounded blast radius by construction. A node can die and the mail keeps flowing. The NAS can die for an afternoon and the must-work tier doesn't notice. The S3-shaped tooling keeps speaking its language to a listener I control. And the whole thing is written down instead of remembered - which, if you've read this far, you'll recognize as the actual point.