TL;DR

• Authentik predates the cluster and survived every migration: one identity provider, one user database, one place where access is decided.
• It speaks everything the estate needs - OIDC, SAML, LDAP, SCIM - so every app that supports a real auth protocol points at it. If an app doesn't, that's a mark against the app.
• It's a dedicated identity provider rather than a generalist: source of truth is a role that belongs to exactly one system, and Nextcloud wasn't built for it.
• MFA is tiered, with WebAuthn hardware keys for the sensitive profiles. And nobody lives as admin - least privilege applies to self-hosted apps exactly as much as to anything in a cloud.

Post 14 ended with a promise: now that the floor is mapped, we build on it. But you don't build the house first. The first thing any building gets, once the slab is poured, is a door. And on a cluster, the door is identity. Who is allowed in, through which entrance, with what proof, and what are they allowed to touch once inside.

That's the post. Authentik.

It predates the cluster

The thing about the keystone services is that the good ideas mostly survived the migrations. Authentik did. I had it running back in the Proxmox and docker-compose days, before any of this cluster existed, for the same reason I run it now: I was tired of every application keeping its own user list.

If you've run more than four self-hosted services, you know the shape of the problem. Nextcloud has users. Gitea has users. Jellyfin has users. Every app has a user database, a password policy, a password reset flow, and a notion of "admin" that means something slightly different from its neighbour's. Individually, each one is fine. Collectively, you're running a small federation of tiny identity providers that don't talk to each other, and you are the federation.

Authentik collapsed all of that into one place. One login. One place where accounts live, one place where passwords are reset, one place where "is this person allowed in" gets answered. And the reason it spread from the old estate to the new one is that it's a genuinely good fit for Kubernetes - the whole thing deploys as workloads, backed by the storage classes from last post, and it speaks every protocol the ecosystem expects. OIDC and SAML for the modern stuff. LDAP for the grumpy stuff. SCIM for provisioning, so accounts can be pushed out to things rather than pulled. Satellites, so to speak: the identity provider stays in one place and the endpoints reach out to it wherever they are.

The result is embarrassingly simple to describe: everything on the cluster that supports a real auth protocol points at Authentik. One way in. If an app doesn't support it, that's now a mark against the app, not a reason to stand up a second user database.

Why not let an app be the source of truth

The obvious alternative route: run Nextcloud, and let it be the user database. It has users, it has groups, it can share credentials around. Plenty of homelabs do exactly that and it works.

I didn't, on purpose. Nextcloud is a file platform that happens to know about users. Authentik is an identity provider, full stop. It exists to hold the user base, to federate it, to answer authentication questions, and nothing else. When something is designed for exactly one job, I've learned to give it that job instead of asking a generalist to do it badly. In a world where MS stands for MicroSlop and identity products exist mostly to make you buy more identity products, Authentik is the best I've used and the best I've tested. Open source, no upsell, and it does the one thing it promises.

There's a design principle hiding in there, the same one that shaped the storage map: source of truth is a role, and a role should belong to exactly one system. The user database lives in Authentik. Everything else - every app, every group mapping, every ACL - defers to it. The moment two systems both believe they hold the truth, you get drift, and drift in identity is how "who had access to what?" stops having an answer.

The keyring

Single sign-on would be enough of a reason, but the part I actually care about is what happens after the password. Authentik does MFA properly, including WebAuthn - real hardware-backed authentication, not an app that generates six digits.

So the policies are tiered. Regular access wants a second factor. The sensitive profiles - the ones where "someone guessed a password" would be a genuinely bad afternoon - want a hardware key. YubiKey-class devices. If the credential can't leave the fob, phishing gets a lot less interesting, because the attacker needs to steal the plastic along with the password.

And that policy is enforced by the identity provider, not by each app. This is the quiet payoff of centralisation: I write "this flow requires WebAuthn with a hardware token" once, and it applies everywhere the flow runs. Before, under the federation of tiny user databases, that sentence wasn't even writable.

Nobody logs in as admin

The last piece isn't a feature, it's a habit, and it applies to self-hosted software exactly as much as to anything I'd deploy at work: least privilege. Not as a compliance checkbox. As a daily operating mode.

The admin account in Authentik exists. It is not the account I live in. Day to day I'm a plain user, with the rights a plain user needs, and elevation is an event that happens when an event is needed. Same rule for the apps behind it: you don't get admin on Nextcloud because you have admin on the cluster, and I don't get admin anywhere just because I own the hardware.

It costs a little friction and it buys something real. When you spend your days as a regular user, the blast radius of a session hijack, a fat-fingered click, or a misbehaving app is what a regular user can touch. Which is, most days, nothing that matters. The cloud world figured this out years ago - nobody sensible runs production as root - and there's no reason a self-hosted cluster should run on a less serious model just because the hardware lives in the house.

What it buys, written down

• One identity provider, one user database, one place where access is decided.
• Protocols for everyone: OIDC, SAML, LDAP, SCIM. The apps that matter point at it.
• MFA and WebAuthn, tiered by profile, with hardware keys for the sensitive ones.
• Least privilege as a habit: nobody, including me, lives as admin.

TL;DR

• Storage classes here are picked by failure mode, not spec sheet: the question isn't "how fast?" but "what happens when it dies?"
• Local tier: Longhorn on the node SSDs at RF=2 - data plus one replica, one-disk tolerance, the same as RAID5. Critical workloads (CNPG databases, mail, Authentik) live here so a node can die and they keep answering.
• NAS tier: NVMe-oF for block, NFS for files - fast enough, and easy offline backups. Dies with the NAS, for an afternoon, which is acceptable for the merely-inconvenient tier.
• Bucket in the basement: SeaweedFS serving the S3 shape locally, offsite copies staying genuinely offsite. Live and offsite remain separate concerns. Once again.

The recent posts were fires. A WAL directory quietly ate a disk. A restore job refused to believe my credentials, and I walked away from S3 as a recovery path rather than hand over static keys. And a future-proofing upgrade left the NAS unable to post, on an afternoon I had personally scheduled. None of it ended in catastrophe, and the reason wasn't luck - it was decisions made before the fires: critical workloads already moved off the NAS, tested restore paths, backups treated as a verb.

This post is the map of where those decisions landed. No war story - the inventory. Where the data actually sleeps, on what hardware, at what replication factor, and why my storage classes are named after criticality instead of performance. If the recent posts were the fires, this is the fire map: the picture you want pinned to the wall before the first smoke, not sketched from memory after it.

Pick storage by failure mode, not by spec sheet

The instinctive way to shop for storage is performance: IOPS, latency, price per gigabyte, whichever benchmark wins. I've stopped shopping that way. Around here, the question a storage class answers isn't "how fast is it?" but "what happens when it dies?"

That inversion changes everything downstream. A workload that must never stop needs storage whose failure mode is "one node gone," not "one box gone." A workload that would merely sulk for an afternoon can happily live behind a single NAS. Backup tooling that insists on speaking S3 doesn't need S3-the-cloud-provider; it needs the shape of S3. Every choice below follows from asking the failure question first. Performance comes second - and honestly, none of these tiers is slow enough to complain about.

The local tier: Longhorn on the node SSDs

Each node carries local SSDs - 256GB each, decent throughput, decent latency, the kind of disks that make a database feel good about its life choices. The pack even had a Kioxia in it, which is the parts-bin lottery paying out for once. I'll take my wins where they come.

Longhorn spreads PVCs across those local disks, and the storage classes here run at RF=2: two replicas, on two different nodes. On a three-node cluster, that's the number I keep landing on - it tolerates one node loss, which is the failure Longhorn exists to absorb here, and a third replica would spend a third of the disk budget on a second concurrent failure that this cluster isn't designed to survive anyway. RF=2 is the honest setting, not the cheap one.

The critical storage class is where the must-work things live. CNPG databases for services that have to work run on Longhorn PVCs on the same RF=2 class - durability and failover, the property that a node can drop dead and the database keeps answering, because its other replica doesn't care. Mail and Authentik are on that class too. This is the post-12 lesson turned into topology: the NAS being unable to post once taught me that "it's on the NAS" and "it's safe" are different sentences. The answer wasn't a better NAS. It was a storage class whose blast radius is a node, not a box.

The NAS tier: NVMe-oF and NFS

The NAS is still very much in the story - it's just no longer load-bearing for the must-work tier. Two storage classes hang off it, and they exist for different reasons.

The first is NVMe-oF - block storage over the network, chosen for throughput. My nodes still run 1GbE links today; the NAS is on LACP at 10Gbps. Even so, NVMe-oF yields very decent throughput from those modest node links: the protocol wastes very little overhead, and with the NAS side aggregated at 10Gbps, the limit you hit is the node's own NIC, not the box behind it. For "inconvenient if down" block storage, that's more than enough.

The second is NFS - the file tier. Some services think in files rather than blocks: they want to share a directory, let someone poke at the contents directly, or generally behave like the filesystem they believe they're on. NFS is the honest answer to that. And it has a quiet superpower: offline backups of files are child's play. A file tree is rsync-able, visible, copyable - no snapshots, no zvol gymnastics. For data that wants to be files, NFS makes the backup story nearly boring. After the last two posts, I am extremely pro-boring.

The bucket in the basement: SeaweedFS

One thing post 12 established: the S3-shaped API is the lingua franca of backup tooling. Half the ecosystem speaks it fluently, and the trap was pointing that fluency at someone else's infrastructure and discovering the identity machinery mid-restore.

So the lingua franca now lives at home. There's a SeaweedFS running locally - S3-compatible, backed by PVCs on the NAS. Backup tools that want an S3 endpoint get one; it just happens to live inside my walls, on storage I control, in the failure domain I chose. The cloud-shaped API without the cloud-shaped dependency.

And the layering stays honest: live data and offsite data remain separate concerns - once again, because it keeps being the rule. Live data is in the cluster or on the NAS. The NAS gets its ZFS snapshots to a separate zpool, exactly as described two posts ago. And the offsite copy is offsite - a different location, a different failure domain, because "the house burned down" is a failure mode no amount of replication inside the house can fix.

The whole map

Which makes the current state of storage, written down at last:

• Must work - CNPG databases, mail, Authentik: Longhorn PVCs on local SSDs at RF=2, failover across nodes. Survives a node loss. Survives a NAS loss.
• Inconvenient if down - everything else: NAS storage classes, NVMe-oF for block, NFS for files. Survives a node loss trivially, because it isn't on the nodes. Dies with the NAS, for an afternoon.
• Backup layers - ZFS snapshots to a separate zpool on the NAS; SeaweedFS serving the S3 shape locally for the tooling that wants it; offsite copies of what matters.
• The rule underneath all of it - live and offsite are separate concerns. Once again.

flowchart TB
    subgraph NODES["Three identical nodes - local SSDs"]
        LH["Longhorn, RF=2<br/>two replicas on two nodes"]
    end
    subgraph MUST["Must work"]
        CNPG["CNPG databases"]
        MAIL["Mail"]
        AUT["Authentik"]
    end
    CNPG --> LH
    MAIL --> LH
    AUT --> LH
    subgraph NASBOX["The NAS - dies with it, for an afternoon"]
        NVME["NVMe-oF<br/>(block)"]
        NFSD["NFS<br/>(files)"]
        SW["SeaweedFS<br/>(S3 shape, on NAS PVCs)"]
        ZP["ZFS snapshots<br/>to a separate zpool"]
    end
    OTHER["Everything else<br/>(inconvenient if down)"] --> NVME
    OTHER --> NFSD
    LH -.->|"snapshots"| ZP
    SW -.->|"offsite copies"| OFF["Offsite<br/>(different failure domain)"]

What this buys is simple: the fires from the last two posts now have bounded blast radius by construction. A node can die and the mail keeps flowing. The NAS can die for an afternoon and the must-work tier doesn't notice. The S3-shaped tooling keeps speaking its language to a listener I control. And the whole thing is written down instead of remembered - which, if you've read this far, you'll recognize as the actual point.

TL;DR

• Nothing happens. The Talos upgrade was uneventful, and that's the point: stability is a habit built from immutability, not luck.
• Nodes can't drift, so the OS stops generating work, and the cluster inherits the calm.
• The quiet is made of boring jobs: snapshots, etcd backups, quiet alerts, an occasional glance at a dashboard.
• Stability is the starting line, not the finish: the real work now is a framework, a workflow, and core services so the cluster scales by scaffolding rather than by heroics.

The day-to-day of running this thing, honestly? Nothing happens.

Not "nothing happens" as in "I have nothing to say." Nothing happens as in the cluster runs, and running is the whole point. The last stretch was all fires. The WAL filled a disk, a restore job refused my credentials, and the NAS stopped posting after an upgrade I made myself. This post is what the calendar looks like when none of that is going on. Which is most days.

Full disclosure: there is no war story in this one. No 1am canary-in-the-inbox moment, no hardware dying mid-upgrade. If you're here for the drama, the last post has it. This one is about why there isn't any.

The Talos upgrade that wasn't a story

When it was time to move Talos to a newer version, I did the thing you're supposed to do: update the config, roll it out node by node, let the cluster drain each one in turn. And nothing happened. No drama, no rollback, no 2am debugging session. The version went up, the pods came back, the cluster kept running.

I keep looking for a lesson in that and the honest one is: there isn't a dramatic one. That's the point. The drill pays off by never being needed. I could tell you about how I staged it, the order I did the nodes, the checks I ran. None of it mattered in the way that makes a story. It mattered the way a handrail matters.

Stability is a habit, not luck

The reason the upgrade was boring is worth naming, because it isn't luck. Talos is immutable: the OS is an image, the config is a file, and the machine cannot drift from what git says it should be. There's nothing to SSH into and "just quickly fix." That sounds like a limitation until you notice what it removes: the slow accumulation of hand-tuned state that makes every server special and every failure unrepeatable. My cluster is stable because my nodes are stable, and my nodes are stable because there is nothing on them that can quietly rot.

There's a real learning curve to that model. For the first while, Talos feels like fighting the OS, because every habit from a normal Linux server is a reflex that doesn't apply. You can't noodle. You edit the config, you apply it, you watch the machine converge or refuse to. Once that clicks, the fight stops. The OS stops generating work. And when the OS stops generating work, Kubernetes inherits that calm: the platform underneath everything can finally be assumed rather than checked.

That's the refreshment part, and I don't use that word lightly. After months where the OS was the thing that woke me up, the OS is now the part that never does.

What the quiet is made of

"The cluster runs properly" is a sentence that hides real work, so here's the ledger of what boring is actually made of:

• The ZFS snapshot jobs from the last post run on their schedule and succeed, and nobody applauds.
• The etcd backup from the PXE post keeps taking its backup, and I keep not needing it.
• A handful of alerts stay quiet. When they do fire, they're right.
• Occasionally I look at a dashboard, or a git log of my node configs, just to see that time has passed without incident.

That's it. That's the whole maintenance story. It's small, and its smallness is the product of the design decisions in these posts, not the absence of design. The boring evening from the PXE post was a promise that operations could be dull. This is the receipt.

Now the real work starts

Here's the turn, and it's the most honest thing I can say about where I am: a stable platform is a starting line, not a finish line. The hardware is fungible, the OS doesn't drift, the databases have tested restore paths, and the whole thing runs without me standing over it. Which removes the last excuse for not doing the thing I actually built this for, which was never "operating Kubernetes." It was running stuff.

And running stuff at any scale needs more than a stable floor. It needs a framework. Right now my deployment story is a collection of manifests that I apply by hand and mostly keep in my head. That's fine at three workloads and stops being fine at thirty. A real workflow: manifests in git, applied by something that isn't me, reviewed like code, with conventions for how a new service joins. Core services that everything else leans on: ingress, DNS, certificates, metrics, the backup schedules as code rather than cron folklore.

That's not glamorous, and it's the actual growth project. The cluster doesn't scale by adding nodes anymore. It scales by building the scaffolding around Kubernetes properly. The platform is done being exciting. The system built on top of it is the next few years of this blog.

TL;DR

• Three separate CNPG failures, each at a different layer: WAL archives filled local disk when shipping stalled; a restore job refused to honour valid IRSA credentials; and the NAS stopped posting entirely.
• The WAL lesson: local disk budget must include the failure window of archive shipping. Coupling, not capacity.
• The credentials lesson: I refused static IAM user keys on principle, walked away from S3 as a recovery path, and kept S3 only as an extra copy layer. ZFS snapshots to a separate zpool are the real recovery path now.
• The NAS failure was self-inflicted: my own future-proofing upgrade (newer CPU + HBA) is what stopped it posting. Survivable only because critical workloads had already moved off it.

Here are the CNPG headaches in detail. Not the drill version. The real ones, and not one incident either: three separate failures, each with its own personality, each hitting a different layer. I'm telling them in the order they arrived.

Some setup, because war stories make no sense without it. CNPG (CloudNativePG) is the operator that runs the live PostgreSQL in my cluster. It stores the database data where Kubernetes says to store it, and it ships WAL archives and base backups to object storage, S3 in my case, over an IAM role the cluster assumed. On paper, that's a tidy design. The database lives on fast storage, the history lives somewhere cold and cheap, and the operator glues them together. I built all of that, it was green, and for a while it was fine.

Then the WAL started eating the disk.

The disk that filled itself

The detail I hadn't appreciated about CNPG is that WAL accumulates on local storage and only gets cleaned up once the archiver has successfully shipped it. That sounds obvious when I write it out, but it means the disk fill isn't a backup problem in the usual sense. It's a coupling problem. If the shipping side stalls, even briefly, the write side keeps going, and the two of them meet somewhere ugly: local disk full, PostgreSQL unable to write, and the operator in a state I'd describe as technically alive.

flowchart LR
    PG["CNPG pod<br/>(PostgreSQL)"] -->|"writes data + WAL"| DISK["Local storage"]
    DISK -->|"archiver ships"| S3["S3:<br/>WAL archives + base backups"]
    S3 -.->|"shipping stalls"| WAL["WAL accumulates,<br/>nothing cleaned up"]
    WAL -.->|"coupled to the write side"| FULL["Local disk full -<br/>PostgreSQL unable to write"]

The first fix was almost embarrassing: larger disks, reboot the pods, done. No heroics. That's worth saying out loud because the instinct when a database stops writing is to assume a deep, dramatic cause. Sometimes it's a disk that needs to be bigger.

But the underlying lesson is the boring one: your database's local storage budget is not just sized for your data. It's sized for your data plus however long your archive shipping can quietly fail before anyone looks. The failure mode is invisible right up until it isn't.

The restore that didn't trust my credentials

Now the part that actually hurt.

When the recovery had to happen for real, I reached for the documented CNPG restore path: bootstrap the cluster from a backup, restore from S3, replay the WAL. The cluster, running as it did with IRSA, assumed an IAM role through its service account. That's how the backup had always worked. The restore is a different job, though, and here's the irony I've been chewing on since: the restore process did not honor the IRSA environment variables. The role that backed the data up without a single stored credential was not available to the job restoring it.

I found this out the way you find out these things, which is at 1am with a half-restored cluster and an error message about missing credentials. The backup existed. The bucket was reachable. The data was all there. And the restore job sat there refusing to read it because the thing doing the restoring was not the thing that had been granted access.

There's a homelab version of "finding out at 1am," and it's almost worse. In a datacenter, the error message finds you. At home, nobody pages you: the cluster fails overnight, you sleep right through it, and the way you find out is the next day, when the inbox is silent - no new mail, not even spam. When the spam is the last canary standing, that's your monitoring doing its level best.

The documented escape hatch, at every step, was the same: hand the restore job a static IAM user key, a long-lived credential, and let it pull from the bucket. I want to be honest about what I did here, because the honest part isn't "I fixed it," it's "I refused."

I refused. That env-var handling was a bug in the restore path, not a limitation of my setup, and papering over it with a static key would have meant accepting a design I knew was worse: long-lived credentials in a place I'd spent the Proxmox era learning not to put them, just to work around a process that couldn't read the identity the platform was already handing it. So I didn't do it. I escalated it in my head to a matter of principle, I ground against it, and I'm not going to pretend that grind was efficient. It was expensive. It was also, I still think, correct - and it's a discipline I'd been building since the Proxmox days, long before this cluster existed.

Because the real fix turned out to be walking away. That was the strategic decision: the S3 restore path had failed me in a way that told me something structural about it. It wasn't that I couldn't make it work; it's that making it work required abandoning a credential model I hold to for everything else, for this one path, this one tool. So the question stopped being "how do I make this restore work" and became "should this restore path exist at all."

The honest retreat: ZFS snapshots

And that's how the backup strategy changed, and not in the direction the textbooks point.

I walked away from barman cloud and the S3 pipeline for recovery entirely, and moved onto truenas-csi snapshots proper: ZFS snapshots of the CNPG volumes, croned to a separate zpool. I want to call this what it felt like: a retreat with scars. I didn't leave S3 because it's bad storage. It's eleven nines of durability, it's excellent at its job, and the irony of the whole episode is that I had a perfectly durable copy of my data that I could not reliably download. Storage that can't hand your data back when you need it isn't a backup, it's a mausoleum.

So the S3 pipeline now has a different, humbler job in my setup: not recovery, just an extra copy, the layer I reach for when the layers above it have already failed and there's time to fight the tooling. The thing I trust to actually save the live database on a bad night is a ZFS snapshot on a different pool, one I can restore in minutes without consulting anyone's key store. Backup is a verb. Restore is the exam. I picked the path where the exam is one I can actually sit.

The unplanned version

And then the NAS, the machine holding all of that, refused to post.

Out of nowhere. Kinda. Because the trigger was, technically, me. I'd swapped in a newer CPU and added an HBA card to hang the disks off - a future-proofing move, part of a plan to eventually migrate to another server. After that upgrade the machine couldn't initialize all of its RAM, and after enough fault-tracing the HBA came out of the picture entirely, and it has stayed out since. So it wasn't a slow trickle of SMART errors I'd ignored, and it wasn't the machine falling over on its own schedule. It was a box that worked, got upgraded in the name of being more future-proof, and stopped posting in the space between two boots. The hardware didn't telegraph the failure. My upgrade calendar did.

The last two posts were about migration as a planned exercise: an API deprecation with a clock on it, a zpool swap scheduled because the storage layer was getting touched anyway. This was the other version, the one you don't schedule - except this time I'd half-scheduled it myself, and the upgrade meant to reduce the future risk was the thing that caused it. The machine that holds your data and your snapshots and your restore path is suddenly a brick, and the cluster keeps asking it for volumes.

I've called this the tech surgeon moment before, and it held: don't panic, stabilize, work the list. But there's a difference between that discipline as a philosophy and needing it at 1am while a live database waits, and the difference is that this time the list had already been written. The critical workloads were already moving off the NAS and onto the three-node cluster, not because I'd predicted the failure, but because the SPOF had been bothering me since long before the NAS actually died. The NAS dying didn't force the migration. It confirmed the one I'd already started.

That's the part I want to leave you with, because it's the least dramatic and most useful thing in this whole story: the NAS failure was survivable not because of anything I did that day, but because of the decision I'd made weeks earlier that the NAS should never be the only copy of anything. When it went down, the work moved. The snapshots were on a separate pool, the critical data had a second home, and the restore paths had been tested. None of that made the failure fun. It just made it an evening instead of an era.

And there's a quieter corollary, the one the HBA taught me: hardware failures in a homelab don't always announce themselves, and they don't always arrive on their own schedule either. Sometimes they arrive because of you, on the exact afternoon you were trying to make things better. The discipline that saved this situation wasn't predicting the failure. It was assuming the NAS could disappear on any given day and making sure that day would be survivable - regardless of whether the cause was a dying disk, a bad slot, or a well-intentioned upgrade of my own.

TL;DR

• The NAS predates the cluster, and the only bridge from Kubernetes to TrueNAS, at the time, was a community CSI driver called democratic-csi. No official one existed.
• Open source volunteers build great things, but when TrueNAS changed the API underneath the driver, the bridge wobbled. Depending on someone else's API has a failure mode all its own.
• The vendor later shipped an official CSI driver, and CSI drivers are not magic: PVCs need recreating, data replayed, restore paths tested. On purpose, before the real move.
• Then I stacked a zpool change on top of it, because apparently one migration wasn't enough of an exam.

Before Longhorn was an option, there was the NAS.

Every cluster needs a place where data goes to outlive the nodes holding it. Mine is a box running TrueNAS, and it's been part of the setup for longer than most of what I've written about so far. This post is about how it got connected to Kubernetes, what that connection is made of, and why the principle from the last post - backup is a verb, restore is the exam - applies to it in a way that isn't obvious until you've been bitten.

The gap between the NAS and the cluster

A NAS doesn't speak Kubernetes. TrueNAS exposes filesystems and ZFS datasets over network protocols; Kubernetes wants a PersistentVolumeClaim it can hand to a pod. The thing in between is a CSI driver: a small program that takes a PVC request and translates it into API calls that create the volume on the other side.

When I set this up, the situation was simple in the worst way: TrueNAS SCALE existed, and the official TrueNAS CSI driver did not. There was exactly one viable path, and it was a community project called democratic-csi.

On the shoulders of volunteers

I want to be fair about one thing first, because the rest of this post is going to sound like a complaint and it isn't one. democratic-csi exists because someone built it. Not a company with a roadmap and an SLA - a person (or a few) who hit the same gap I did, wrote the translation layer, and gave it away. Kubernetes is full of things like this. The controller managers, the webhook validators, the operators, the drivers: an enormous share of the ecosystem is people around the world contributing their evenings to problems they needed solved themselves, and letting the rest of us stand on that. This is one of the genuinely great things about k8s, and about open source in general. When you can't get a piece of infrastructure from the vendor, very often you can get it from a volunteer, and it works, sometimes for the better part of a decade.

For months, in my case. democratic-csi handled every PVC I asked it for, through the Pi-and-NUC era, through the McGyver cluster, without ceremony. Which is exactly the situation that sets a trap.

The trap: you depend on someone else's API

The problem with building on a community driver is that the driver isn't the only dependency. It sits on top of the storage platform's management API, and the storage platform's management API is owned by the storage vendor. One day TrueNAS announced that the API democratic-csi is built on will be decommissioned.

Notice what actually happened there. My code didn't change. democratic-csi's code didn't change. The cluster kept running, the PVCs kept binding, nothing failed. And yet a deprecation announcement had just put a clock on my storage layer, because the floor under the volunteer's project was being removed by a vendor who owed me nothing and the volunteer even less.

This is the part of open source nobody puts on the marketing slide. The incredible contribution-from-strangers model also means your dependency chain includes strangers, their free time, and APIs somebody else decides to sunset. Kubernetes itself is remarkably good at insulating you from this - the CSI contract is a stable interface, so the pod doesn't care what driver sits behind it - but stability at the interface doesn't help the driver keep working.

The vendor saw the same problem

To their credit, TrueNAS didn't just pull the ladder. An official TrueNAS CSI driver now exists, and the gap I filled with a community project got filled by the platform itself. That's the healthy end state.

But here's the detail that turns this from a vendor-comparison footnote into the point of the post: there is no documented migration path from democratic-csi to truenas-csi. I looked. Unless I missed something well hidden, nobody - not the vendor, not the community project - wrote down "here is how you move your running cluster from one driver to the other." And I understand why: it's not really a product problem, it's your problem. The vendor documents their driver, the community documents theirs, and the bridge between them is assumed to be your job.

Fair enough. But "it's your job" is doing a lot of quiet work in that sentence, because what your job actually is here is a restore.

CSI drivers are not magic

The easy mental model is that a CSI driver is a filesystem plug, and swapping it is like swapping a cable. It isn't. A CSI driver creates and manages volumes through its own API calls, and the volumes it created are records on the storage side tied to how that driver named and addressed them. Install a different driver, and your existing PVCs don't just reattach to it. The new driver doesn't know those volumes belong to your workloads, and it may not even create volumes in the same place or the same shape.

So a driver migration, done honestly, looks like this: recreate the PVCs under the new driver, get the data into the new volumes - replayed, restored from backup, or copied volume to volume - then repoint the workloads and prove it. None of that is out of the box. Some of it is straightforward, some of it is tedious, and all of it is a restore-shaped problem: you are rebuilding the storage state of your cluster from some other copy of it, under your own supervision, and finding out what your backups are actually worth.

This is also why I keep saying backup is a verb. The API deprecation was survivable precisely because the question "can I rebuild my PVCs and their data from what I have?" had answers I had tested, not answers I hoped existed. If the answer had been "there's a Velero manifest somewhere," the deprecation announcement stops being an inconvenience and becomes a deadline with teeth.

And then I made it worse, on purpose

Because the storage layer was getting touched anyway, it was tempting to also swap the underlying zpool - new disks, new layout, same NAS. That's the kind of change that feels like a tidy evening of work and is actually two migrations stacked: the CSI driver above, the pool below. If either one is untested, and especially if both are, you find out about it in the worst order possible, at the worst time possible, with a cluster that can't schedule anything that needs storage.

The pool change itself wasn't the scary part. Moving data at the zfs level gives you several honest options - a ZFS-level copy, or plain rsync - so the question was never "how do I move the bytes." The part that actually mattered was the workload underneath it all: CNPG, the live PostgreSQL. That was the big pain point, and what made the whole thing survivable is that at that time I already had a tested recovery path for CNPG. Everything else was file moves. The database was the thing you couldn't afford to guess about, and I didn't have to.