TL;DR

• Nodes are special when each one boots its own way; the fix is fungibility - three identical machines, one PXE boot path, one declarative config. Power on, network on, no step three.
• The scheduler makes the scale-out cheap: assignments land, workloads start, nobody carries a machine in their arms.
• etcd backup happened before the madness, and the join/leave symmetry is the real test - decommissioning is boring on purpose.
• Power draw stayed about the same, but the cluster is now patchable one machine at a time with no real downtime.

Here's the cluster as it stands: a Pi 4 holding etcd on an SD card, and a 2013 NUC carrying my mail. Fixing that - without pretending I'd found a datacentre budget - is the job now, and it's where the last seven installments finally cash in. The floppy disks taught me why the layers matter. The 42U rack taught me that provisioning at scale is a solved problem if you refuse to hand-touch anything. The Eucalyptus years taught me that a machine you can describe is a machine you can replace. Everything converges here, on a Tuesday evening, in a rack that used to be a drawer of parts.

The problem with the McGyver cluster

The Pi-and-NUC cluster's failure mode wasn't that it was cheap. It was that every node was special. One SD card, one vintage NUC, one bespoke boot process each - which means the failure of any single node was a unique, unrepeatable archaeology project. When the SD card dies, you don't fail over. You mourn.

The fix for "every node is special" is not better hardware. The fix is making nodes fungible: three identical machines, one identical boot path, one identical config file. Fungibility is what cloud providers actually sell you, and it's what my drawer-parts cluster was missing at the most fundamental level.

PXE boot: the factory floor at home

Talos ships factory-made images - I said that in the last post, and it's the single most load-bearing sentence in this series. The follow-on is that you don't even have to flash them by hand. Talos publishes the artifacts PXE needs: a kernel, an initramfs, and a machine config served over the network. Set up a DHCP server that hands out the right next-server and filename, put the Talos boot artifacts somewhere a machine can TFTP or HTTP-fetch them, and the node's entire provisioning procedure becomes:

1. Plug in power.
2. Plug in network.
3. There is no step three.

The machine boots, fetches the kernel, fetches its config - which is a YAML file in git, versioned, diffable, reviewable - and joins the cluster. If the config is wrong, you fix the file and reapply; the node converges. Nobody opens a case. Nobody finds a monitor. I don't own a portable monitor, remember, and I still don't.

flowchart LR
    NODE["New node<br/>(power + network, no step three)"] -->|"DHCP:<br/>next-server + filename"| DHCP["DHCP server"]
    DHCP -->|"TFTP / HTTP"| BOOT["Talos kernel<br/>+ initramfs"]
    BOOT -->|"machine config:<br/>YAML from git"| JOIN["Node joins the cluster"]
    JOIN --> SEE["Scheduler sees new capacity,<br/>assignments land"]

And here's the part that makes it scale: get it to work for one of three identical machines, and you've got it working for all of them. The second and third nodes are the same box, the same kernel, the same YAML with a different token. Scaling the cluster with hardware becomes as boring as scaling it with VMs on someone else's cloud - which was, not coincidentally, the point. I'm not building a homelab; I'm building a factory for nodes that happens to live in my house.

Declarative config for the boxes themselves

This is where declarative infrastructure stops being a blog-post word and starts being a habit. Every one of my Talos nodes is described by a config file in git. The network interfaces, the extensions (Tailscale, yes - the groundwork from last post), the kernel arguments, the disks - all of it declared, none of it hand-touched. When I want to change something that applies to every node, I change one patch file. When a node dies, I replace it, and the replacement converges to the same declared state, because there is no other state to converge to.

The contrast with my old Proxmox sediment is total. There, every VM was a diary. Here, every node is a printout. If I could hand the printout to a stranger with a new machine, the stranger produces an identical node. That's the whole test, and it's the test cloud providers have been passing for fifteen years.

The Kubernetes half: where the magic gets boring

The boot is only half the story, and honestly the less impressive half. The part that made me stop and stare was what happened after the nodes joined.

No ceremony. No "now manually reschedule these pods." The control plane just sees new capacity, and the machinery that's been idle since the cluster was born - the scheduler, the controllers, the CSI driver - wakes up and does its job. Assignments land. Scheduling starts. Workloads start. Volumes bind to the nodes that can hold them. End to end, the whole thing is transparent, and it's transparent not because I'm clever but because that's what Kubernetes is: a system that watches declared state and moves the world toward it. Adding capacity is not an operation. It's an observation the scheduler makes on its own schedule.

If you've run Kubernetes in the cloud, you know this already - that's what autoscalers exploit. The difference is that here, the "autoscaler" is me plugging in a box, and the plumbing from power cable to running pod is mine end to end. The cloud does the same trick by renting you the metal first. I just cut out the rental.

Etcd backup, or: the one non-negotiable

I'll spoil the ending by telling you the only genuinely critical step, because it deserves its own paragraph and because skipping it is the one mistake in this whole process that has no undo.

Before the madness begins - before any node joins, before any migration, before anything that touches cluster state - take an etcd backup, and test the restore path. Not "I have a snapshot somewhere." Actually restore it once into a scratch cluster, on a boring afternoon, so that the restore procedure is a memory instead of a document. etcd is the cluster's brain; every secret, every config, every spec lives there. Hardware is replaceable by design now. The state in etcd is replaceable by nothing. One backup, one tested restore, and the entire rest of the process can be attempted with the calm of a person who knows there's a floor.

With that floor in place, everything else is reversible. Nodes can be wiped and rejoined. The control plane can be rebuilt from scratch. The only irreversible mistake is the one where the state is gone, and that mistake costs one command to prevent.

Decommission, without ceremony

The other end of the lifecycle deserves equal billing, because it's the mirror image and nobody writes about it. Once the new nodes were in and carrying the load, decommissioning the old ones was easy and - I keep reaching for a better word and coming back to this one - without ceremony.

The Pi 4 and the NUC didn't get a farewell post. They got drained, the workloads rescheduled themselves the way workloads do, and the machines went back in the drawer they came from. No migration weekend. No cutover meeting of one. The cluster doesn't distinguish between "node retired" and "node was never there," because both are just declared state that no longer matches reality, and the controllers reconcile both the same way. I spent more time physically carrying the old machines to the drawer than I spent decommissioning them from the cluster.

That symmetry - join without drama, leave without drama - is the real measure that the design works. A system you can only build is a system you'll be afraid to change. A system you can dismantle at will is a system you actually own.

The power bill, or: what it costs

One honest number, because the blog's whole thesis is trade-offs and I owe you this side of the ledger. The overall power draw is about the same as before - the rack replaced the drawer-parts cluster with machines that are more capable but not dramatically hungrier. What I bought with the same watts is a cluster I can patch one machine at a time, with no real downtime, because there are enough identical nodes that one draining and rebooting is just a scheduler footnote.

That's the trade in one sentence, and it's the blog's thesis wearing work clothes: the same electricity, the same hardware footprint, the same room in the same house - but now the thing it powers is fungible, declarative, and patchable in place. The convenience I gave up is the convenience of hand-tuning a special snowflake. What I got back is a cluster I'd actually trust with my mail.

Which, given the last post, is a higher bar than it sounds.