TL;DR

• The NAS predates the cluster, and the only bridge from Kubernetes to TrueNAS, at the time, was a community CSI driver called democratic-csi. No official one existed.
• Open source volunteers build great things, but when TrueNAS changed the API underneath the driver, the bridge wobbled. Depending on someone else's API has a failure mode all its own.
• The vendor later shipped an official CSI driver, and CSI drivers are not magic: PVCs need recreating, data replayed, restore paths tested. On purpose, before the real move.
• Then I stacked a zpool change on top of it, because apparently one migration wasn't enough of an exam.

Before Longhorn was an option, there was the NAS.

Every cluster needs a place where data goes to outlive the nodes holding it. Mine is a box running TrueNAS, and it's been part of the setup for longer than most of what I've written about so far. This post is about how it got connected to Kubernetes, what that connection is made of, and why the principle from the last post - backup is a verb, restore is the exam - applies to it in a way that isn't obvious until you've been bitten.

The gap between the NAS and the cluster

A NAS doesn't speak Kubernetes. TrueNAS exposes filesystems and ZFS datasets over network protocols; Kubernetes wants a PersistentVolumeClaim it can hand to a pod. The thing in between is a CSI driver: a small program that takes a PVC request and translates it into API calls that create the volume on the other side.

When I set this up, the situation was simple in the worst way: TrueNAS SCALE existed, and the official TrueNAS CSI driver did not. There was exactly one viable path, and it was a community project called democratic-csi.

On the shoulders of volunteers

I want to be fair about one thing first, because the rest of this post is going to sound like a complaint and it isn't one. democratic-csi exists because someone built it. Not a company with a roadmap and an SLA - a person (or a few) who hit the same gap I did, wrote the translation layer, and gave it away. Kubernetes is full of things like this. The controller managers, the webhook validators, the operators, the drivers: an enormous share of the ecosystem is people around the world contributing their evenings to problems they needed solved themselves, and letting the rest of us stand on that. This is one of the genuinely great things about k8s, and about open source in general. When you can't get a piece of infrastructure from the vendor, very often you can get it from a volunteer, and it works, sometimes for the better part of a decade.

For months, in my case. democratic-csi handled every PVC I asked it for, through the Pi-and-NUC era, through the McGyver cluster, without ceremony. Which is exactly the situation that sets a trap.

The trap: you depend on someone else's API

The problem with building on a community driver is that the driver isn't the only dependency. It sits on top of the storage platform's management API, and the storage platform's management API is owned by the storage vendor. One day TrueNAS announced that the API democratic-csi is built on will be decommissioned.

Notice what actually happened there. My code didn't change. democratic-csi's code didn't change. The cluster kept running, the PVCs kept binding, nothing failed. And yet a deprecation announcement had just put a clock on my storage layer, because the floor under the volunteer's project was being removed by a vendor who owed me nothing and the volunteer even less.

This is the part of open source nobody puts on the marketing slide. The incredible contribution-from-strangers model also means your dependency chain includes strangers, their free time, and APIs somebody else decides to sunset. Kubernetes itself is remarkably good at insulating you from this - the CSI contract is a stable interface, so the pod doesn't care what driver sits behind it - but stability at the interface doesn't help the driver keep working.

The vendor saw the same problem

To their credit, TrueNAS didn't just pull the ladder. An official TrueNAS CSI driver now exists, and the gap I filled with a community project got filled by the platform itself. That's the healthy end state.

But here's the detail that turns this from a vendor-comparison footnote into the point of the post: there is no documented migration path from democratic-csi to truenas-csi. I looked. Unless I missed something well hidden, nobody - not the vendor, not the community project - wrote down "here is how you move your running cluster from one driver to the other." And I understand why: it's not really a product problem, it's your problem. The vendor documents their driver, the community documents theirs, and the bridge between them is assumed to be your job.

Fair enough. But "it's your job" is doing a lot of quiet work in that sentence, because what your job actually is here is a restore.

CSI drivers are not magic

The easy mental model is that a CSI driver is a filesystem plug, and swapping it is like swapping a cable. It isn't. A CSI driver creates and manages volumes through its own API calls, and the volumes it created are records on the storage side tied to how that driver named and addressed them. Install a different driver, and your existing PVCs don't just reattach to it. The new driver doesn't know those volumes belong to your workloads, and it may not even create volumes in the same place or the same shape.

So a driver migration, done honestly, looks like this: recreate the PVCs under the new driver, get the data into the new volumes - replayed, restored from backup, or copied volume to volume - then repoint the workloads and prove it. None of that is out of the box. Some of it is straightforward, some of it is tedious, and all of it is a restore-shaped problem: you are rebuilding the storage state of your cluster from some other copy of it, under your own supervision, and finding out what your backups are actually worth.

This is also why I keep saying backup is a verb. The API deprecation was survivable precisely because the question "can I rebuild my PVCs and their data from what I have?" had answers I had tested, not answers I hoped existed. If the answer had been "there's a Velero manifest somewhere," the deprecation announcement stops being an inconvenience and becomes a deadline with teeth.

And then I made it worse, on purpose

Because the storage layer was getting touched anyway, it was tempting to also swap the underlying zpool - new disks, new layout, same NAS. That's the kind of change that feels like a tidy evening of work and is actually two migrations stacked: the CSI driver above, the pool below. If either one is untested, and especially if both are, you find out about it in the worst order possible, at the worst time possible, with a cluster that can't schedule anything that needs storage.

The pool change itself wasn't the scary part. Moving data at the zfs level gives you several honest options - a ZFS-level copy, or plain rsync - so the question was never "how do I move the bytes." The part that actually mattered was the workload underneath it all: CNPG, the live PostgreSQL. That was the big pain point, and what made the whole thing survivable is that at that time I already had a tested recovery path for CNPG. Everything else was file moves. The database was the thing you couldn't afford to guess about, and I didn't have to.