TL;DR

• Backups and restores are different operations: a backup is a copy, a restore is the whole system rebuilt under pressure. Green dashboards prove nothing about the second one.
• My backups worked while my restores didn't exist. The gap between those two facts is where the worst evenings live.
• The fix is boring by design: a written restore procedure, drilled on a schedule, on the same class of metal that will be in the room when it's real.

Time for the full sermon, and it opens with a confession that will sting anyone who's ever run a homelab with confidence: for most of this cluster's life, my backups worked and my restores did not exist. Backups can work and restores can fail miserably, and the gap between those two facts is where I spent some of the worst evenings of this whole project.

The category error everyone makes

A backup is a copy. A restore is the entire system, rebuilt, under time pressure, from whatever the copy actually contains. These are not the same operation wearing different hats, and the industry's relationship to them proves it: everyone has backups, and almost nobody has proven restores. The backup job shows green in the dashboard. The restore path has never been run. The dashboard cannot tell you that the snapshot you've been dutifully taking every night doesn't actually boot.

And the failure isn't exotic. It's the mundane things: the restore procedure assumes a tool you no longer have installed. The snapshot references a machine whose MAC address is baked into some config. The backup is complete but the process to use it lives in your head, and your head, at 1am, mid-incident, is not a durable storage medium.

The cloud papers over this so smoothly you forget it's papering. When you run on managed services, "restore" is often literally a button, or a support ticket, and the provider's DR runbook is someone else's tested problem. The moment you're on your own metal, every assumption that made the button work - the provider's tooling, their network, their ability to hand you a fresh instance with your tags on it - evaporates. And self-hosting tools are worse: a huge amount of the self-hosted ecosystem silently assumes either that you're in the cloud, or that you're not, and are happy handing over API keys to something like a hosted backup SaaS to make the problem go away. I'm not. That's a decent chunk of why this blog exists.

The cold start I drilled, and the restore I didn't

There's a variant of this that bites even people who do test restores, and it's the cold start: the scenario where nothing is running at all. Not "the database pod crashed, restore it" - the full "the machine is off, the cluster doesn't exist, build everything from the state you have."

Here's the irony I'll own outright: cold start is the one thing I tested. Repeatedly. Rebuilding the cluster from nothing was drilled into me early - wipe the machine, boot it, bring the cluster up from whatever state I had, again and again, until it stopped being scary. That habit was probably the most valuable thing the Pi-and-NUC era gave me, and it's a large part of why the McGyver cluster was rebuildable at all. But there's a trap hiding in that, and I fell into it: a rehearsed cold start quietly becomes a substitute for a tested restore. "I can always rebuild from scratch" feels like a recovery story. It isn't. It's a disaster-recovery story for the day you don't mind the downtime, the data loss, and the evening of yak-shaving. The scenario where the cluster is fine and only one component is dead - that's the scenario I never rehearsed, and it's the one that bit.

Adeptus Mechanicus recovery

The resulting recoveries deserve their own paragraph, because the phrase that comes to mind when I remember them is one I'm only half-joking about: successful recovery was an act worthy of the Adeptus Mechanicus. Incense and liturgy. The right incantations typed in the right order at the right time, a libation of coffee, and a genuine, sincere prayer that the thing came up - because the procedure wasn't really a procedure. It was a sequence of steps I'd discovered worked, once, in a specific order, dependent on machine states I couldn't reproduce and timing I couldn't control.

That's what an untested restore path actually is: liturgy. You're not operating a system; you're performing a rite, and the system's willingness to come back is a matter of faith, not engineering. Every time it worked, it worked because of things I couldn't name. And every time it worked, I learned nothing - because success without understanding is not a tested process, it's a lucky one.

The hardware answer, which is really a philosophy answer

The three EliteDesk machines with i5-9500T chips came in recently, and it's tempting to tell the story as "I bought better hardware and the problems went away." That's not the story. The hardware arrived because the cluster had been through situations where recovery was liturgy, and I refused to keep running a household on faith.

The EliteDesks changed the math in two specific ways, and neither is raw horsepower:

1. Fungibility. Three identical machines - the property from last post. A restore that fails on one machine's quirks is a restore that fails identically on all three, which means testing it once tests it three times, and means the recovery procedure is the same procedure regardless of which box died. The Adeptus Mechanicus liturgy existed because every machine was a unique ritual. Identical machines make the ritual a procedure.
2. Enough nodes to test against. You cannot rehearse a cold start on the cluster you're afraid to break. With three identical nodes, one of them is always expendable. Wipe one, restore onto it, prove the path, wipe it again. The test cluster stopped being hypothetical and became a machine with a power cable.

So the hardware didn't fix recovery. The hardware made practicing recovery cheap enough that I actually do it, which is what fixed recovery.

The principle

Which brings me to the actual content of this post, because it's a principles post, and the principle fits in one line:

Backup and restore are one operation, and restore is the half you must prove.

Concretely, what that means for me now:

• A backup that has never been restored is a hypothesis, not a backup. Green dashboards count for nothing. The only unit of evidence that matters is a restore you've personally completed.
• Restore includes cold start, but cold start is not a substitute for restore. They're different scenarios: rebuild everything from a dark room, versus bring one dead component back while the rest of the cluster is fine. I drilled the first relentlessly and skipped the second entirely - and the second is the one that bit.
• The procedure must be written and boring. If it involves incantations, discovered sequences, or anything that sounds like liturgy, it isn't a procedure yet - it's a memory that will fail you exactly once, at the worst time. Write it down. Then follow it by rote during a test. Then rewrite it, because the rote-following will find the gaps.
• Schedule the drill. Not "when I get around to it." A restore test that isn't on a calendar is a backup you've decided not to verify. Mine is on one.

Where this is going

The next post introduces a player that's been offstage since the beginning: the NAS. It runs TrueNAS, it holds everything the cluster actually cares about, and its connection to Kubernetes is a piece of open source whose API upstream has scheduled for decommissioning. That's a story in its own right, and it's where the principle from this post earns its first real payoff. The war story after it is the exam: a CNPG (CloudNativePG) disaster, in detail, where my cold starts were drilled and boring, my restores were neither, and the first component-level restore that had to happen for real found out what my backups were actually worth. So did I.

If you take one thing from this post, take the line: backup is a verb, restore is the exam. Before the exam: the NAS.

TL;DR

• Nodes are special when each one boots its own way; the fix is fungibility - three identical machines, one PXE boot path, one declarative config. Power on, network on, no step three.
• The scheduler makes the scale-out cheap: assignments land, workloads start, nobody carries a machine in their arms.
• etcd backup happened before the madness, and the join/leave symmetry is the real test - decommissioning is boring on purpose.
• Power draw stayed about the same, but the cluster is now patchable one machine at a time with no real downtime.

Here's the cluster as it stands: a Pi 4 holding etcd on an SD card, and a 2013 NUC carrying my mail. Fixing that - without pretending I'd found a datacentre budget - is the job now, and it's where the last seven installments finally cash in. The floppy disks taught me why the layers matter. The 42U rack taught me that provisioning at scale is a solved problem if you refuse to hand-touch anything. The Eucalyptus years taught me that a machine you can describe is a machine you can replace. Everything converges here, on a Tuesday evening, in a rack that used to be a drawer of parts.

The problem with the McGyver cluster

The Pi-and-NUC cluster's failure mode wasn't that it was cheap. It was that every node was special. One SD card, one vintage NUC, one bespoke boot process each - which means the failure of any single node was a unique, unrepeatable archaeology project. When the SD card dies, you don't fail over. You mourn.

The fix for "every node is special" is not better hardware. The fix is making nodes fungible: three identical machines, one identical boot path, one identical config file. Fungibility is what cloud providers actually sell you, and it's what my drawer-parts cluster was missing at the most fundamental level.

PXE boot: the factory floor at home

Talos ships factory-made images - I said that in the last post, and it's the single most load-bearing sentence in this series. The follow-on is that you don't even have to flash them by hand. Talos publishes the artifacts PXE needs: a kernel, an initramfs, and a machine config served over the network. Set up a DHCP server that hands out the right next-server and filename, put the Talos boot artifacts somewhere a machine can TFTP or HTTP-fetch them, and the node's entire provisioning procedure becomes:

1. Plug in power.
2. Plug in network.
3. There is no step three.

The machine boots, fetches the kernel, fetches its config - which is a YAML file in git, versioned, diffable, reviewable - and joins the cluster. If the config is wrong, you fix the file and reapply; the node converges. Nobody opens a case. Nobody finds a monitor. I don't own a portable monitor, remember, and I still don't.

flowchart LR
    NODE["New node<br/>(power + network, no step three)"] -->|"DHCP:<br/>next-server + filename"| DHCP["DHCP server"]
    DHCP -->|"TFTP / HTTP"| BOOT["Talos kernel<br/>+ initramfs"]
    BOOT -->|"machine config:<br/>YAML from git"| JOIN["Node joins the cluster"]
    JOIN --> SEE["Scheduler sees new capacity,<br/>assignments land"]

And here's the part that makes it scale: get it to work for one of three identical machines, and you've got it working for all of them. The second and third nodes are the same box, the same kernel, the same YAML with a different token. Scaling the cluster with hardware becomes as boring as scaling it with VMs on someone else's cloud - which was, not coincidentally, the point. I'm not building a homelab; I'm building a factory for nodes that happens to live in my house.

Declarative config for the boxes themselves

This is where declarative infrastructure stops being a blog-post word and starts being a habit. Every one of my Talos nodes is described by a config file in git. The network interfaces, the extensions (Tailscale, yes - the groundwork from last post), the kernel arguments, the disks - all of it declared, none of it hand-touched. When I want to change something that applies to every node, I change one patch file. When a node dies, I replace it, and the replacement converges to the same declared state, because there is no other state to converge to.

The contrast with my old Proxmox sediment is total. There, every VM was a diary. Here, every node is a printout. If I could hand the printout to a stranger with a new machine, the stranger produces an identical node. That's the whole test, and it's the test cloud providers have been passing for fifteen years.

The Kubernetes half: where the magic gets boring

The boot is only half the story, and honestly the less impressive half. The part that made me stop and stare was what happened after the nodes joined.

No ceremony. No "now manually reschedule these pods." The control plane just sees new capacity, and the machinery that's been idle since the cluster was born - the scheduler, the controllers, the CSI driver - wakes up and does its job. Assignments land. Scheduling starts. Workloads start. Volumes bind to the nodes that can hold them. End to end, the whole thing is transparent, and it's transparent not because I'm clever but because that's what Kubernetes is: a system that watches declared state and moves the world toward it. Adding capacity is not an operation. It's an observation the scheduler makes on its own schedule.

If you've run Kubernetes in the cloud, you know this already - that's what autoscalers exploit. The difference is that here, the "autoscaler" is me plugging in a box, and the plumbing from power cable to running pod is mine end to end. The cloud does the same trick by renting you the metal first. I just cut out the rental.

Etcd backup, or: the one non-negotiable

I'll spoil the ending by telling you the only genuinely critical step, because it deserves its own paragraph and because skipping it is the one mistake in this whole process that has no undo.

Before the madness begins - before any node joins, before any migration, before anything that touches cluster state - take an etcd backup, and test the restore path. Not "I have a snapshot somewhere." Actually restore it once into a scratch cluster, on a boring afternoon, so that the restore procedure is a memory instead of a document. etcd is the cluster's brain; every secret, every config, every spec lives there. Hardware is replaceable by design now. The state in etcd is replaceable by nothing. One backup, one tested restore, and the entire rest of the process can be attempted with the calm of a person who knows there's a floor.

With that floor in place, everything else is reversible. Nodes can be wiped and rejoined. The control plane can be rebuilt from scratch. The only irreversible mistake is the one where the state is gone, and that mistake costs one command to prevent.

Decommission, without ceremony

The other end of the lifecycle deserves equal billing, because it's the mirror image and nobody writes about it. Once the new nodes were in and carrying the load, decommissioning the old ones was easy and - I keep reaching for a better word and coming back to this one - without ceremony.

The Pi 4 and the NUC didn't get a farewell post. They got drained, the workloads rescheduled themselves the way workloads do, and the machines went back in the drawer they came from. No migration weekend. No cutover meeting of one. The cluster doesn't distinguish between "node retired" and "node was never there," because both are just declared state that no longer matches reality, and the controllers reconcile both the same way. I spent more time physically carrying the old machines to the drawer than I spent decommissioning them from the cluster.

That symmetry - join without drama, leave without drama - is the real measure that the design works. A system you can only build is a system you'll be afraid to change. A system you can dismantle at will is a system you actually own.

The power bill, or: what it costs

One honest number, because the blog's whole thesis is trade-offs and I owe you this side of the ledger. The overall power draw is about the same as before - the rack replaced the drawer-parts cluster with machines that are more capable but not dramatically hungrier. What I bought with the same watts is a cluster I can patch one machine at a time, with no real downtime, because there are enough identical nodes that one draining and rebooting is just a scheduler footnote.

That's the trade in one sentence, and it's the blog's thesis wearing work clothes: the same electricity, the same hardware footprint, the same room in the same house - but now the thing it powers is fungible, declarative, and patchable in place. The convenience I gave up is the convenience of hand-tuning a special snowflake. What I got back is a cluster I'd actually trust with my mail.

Which, given the last post, is a higher bar than it sounds.

The metal, then. With the 42U of beauty from my student years long gone and far overpowered by time, the first version of this cluster was a Raspberry Pi 4 with Kubernetes' control plane on an SD card, and a 2013-era Intel NUC that used to run Eucalyptus. This is madness, we all know that, and while an upgrade was the fix, the defence is the lesson.

Factory-made everything

What makes Talos different from every other homelab Linux isn't only the immutability - I covered that. It's that the entire onboarding path is manufactured for you, in every sense of the word. Talos ships the OS itself from their "factory" as a signed, versioned image for basically any platform you can name: x86_64, arm64, bare metal, cloud, Pi. You pick your platform, your revision, your extensions, and you get an image that is the operating system. There is nothing to install in the classical sense - you flash it, boot it, and the node sits there waiting for instructions over its API like a very patient appliance.

And the Kubernetes part - the part every other route makes you earn - is simply included. If you've ever done this the artisanal way, you know the ritual: install a distro, tune the kernel, configure containerd, then kubeadm init, then debug why the CNI didn't come up, then the control plane endpoint dance, then certificates, then the fifty small flags that the tutorial you followed in 2021 doesn't mention anymore. kubeadm is a good tool, but "good tool you must operate correctly" is homelab drag with a Kubernetes accent. With Talos, talosctl gen config hands you the machine configs and the cluster config in one shot, the node boots with everything wired - etcd, kube-apiserver, containerd, the CNI choice of your config - and the control plane appears. Not "eventually, after an evening of sed." It appears. The first time I watched it happen I kept waiting for the part where it goes wrong, because fifteen years of Linux had taught me that the part where it goes wrong is mandatory. It didn't come. I found that suspicious for about a week.

One genuinely nerdy detail that I'll only mention because this blog is already self-indulgent: Talos doesn't boot into a normal userspace and then start Kubernetes. It boots a minimal init, mounts an immutable rootfs, and launches the whole Kubernetes stack as system services from signed images - with a kernel kexec into a freshly provisioned environment on first boot in some paths, so the machine that ends up running your control plane is not literally the machine that first spun. The OS you booted is not the OS you run. That is the philosophy in one sentence, and it's the reason upgrades can be atomic: the running system is an artifact, not a history.

Tailscale, built in

The second thing that won me over is integration with Tailscale - and I need to be precise, because "Talos + Tailscale" means two different things and the difference matters. Talos ships the Tailscale client as a first-class extension in the machine config. That gives you node-to-node networking: the nodes reach each other over a WireGuard mesh, across networks, without me touching firewall rules or trusting my LAN's flatness. This is about the cluster's plumbing. It is entirely separate from the Tailscale operator, the Kubernetes-native side that hands out IPs and publishes services into your tailnet - that's a story for a later post, and it's a good one, but don't conflate the two. One is how the nodes talk to each other. The other is how other things talk to the cluster.

For me, node-to-node was quietly enormous, though not for the reason you might guess. My network is not the problem - far from it. The VLANs are deliberate, the interfaces each have a purpose, nothing about the physical layout is accidental. The reason Tailscale went into the machine config is future-proofing. I want to be able to provision VMs elsewhere - any cloud, on demand, automatically - and have them join this cluster without me redesigning anything. Labels and selectors are how I intend to keep that sane: pods land where they need to be, and more importantly, where they should not. Longhorn is bound to the rack nodes by labels, so replication cannot creep somewhere it was never meant to go. Workers joining from anywhere is only half the requirement, though - the control plane has to be reachable from anywhere too, and that is where Tailscale does the real work: the mesh makes the API server reachable from any network a worker ends up on, and its ACL policy lets me enforce proper access control over that WireGuard network rather than trusting whatever subnet a node happens to sit behind. That plan hasn't materialized yet. This is the groundwork.

The hardware, and the madness

So: what did I actually run?

The control plane was a Raspberry Pi 4, booted from an SD card. Let me sit with that a moment, because I know exactly what you're thinking and you're right. Running etcd - the component whose entire job is to never, ever lose state, and whose write performance defines your API server's latency - on flash storage whose wear characteristics are famously terrible, whose write endurance is measured in unknowns, and which will absolutely corrupt itself under sustained load... is madness. I knew it when I did it. My defence is this: I already owned the Pi 4, already in a drawer, and drew about as much power as a nightlight. For a learning cluster - "does my design hold up, does my config surface work, can I actually operate this thing" - it was the correct choice, as a choice. What it was not, and I want to be honest about the distinction, was a long-term substrate for anything I cared about.

The worker was better: a 2013-generation i3 NUC, which had spent its previous life running Eucalyptus testing back when I was a TAM and OpenStack's sparring partner was still a punchline. Close to over 10 years old, but 16GB of RAM, and a CPU that predates half the Kubernetes ecosystem's assumptions about modern instructions. But it had a real SSD, real x86, and - this is the part that matters - a history. There's something fitting about a machine that once ran one of Kubernetes' spiritual ancestors coming back to run the real thing.

And it worked fine. For a long while, even. Cluster came up, workloads ran, upgrades happened, the Tailscale mesh held, the immutable-config discipline stuck. This is the trap, by the way - the first homelab cluster always works fine, and "works fine" is the most dangerous state a homelab can be in, because it's the state in which you stop treating it as a prototype and start treating it as infrastructure. I ran this thing long enough that my own e-mail began to depend on it.

Which is, of course, exactly the moment you're not supposed to be running your e-mail on a Pi 4 with etcd on an SD card and a 2013 NUC as the only worker. There is no redundancy. There is no meaningful failure mode for "the SD card wears out" except a full control-plane restore from backups I was confident about the way you're confident about things you've never actually tested under pressure. It was McGyver infrastructure - brilliant, cheap, held together with intent and luck, and completely unacceptable as the substrate for the thing my household actually runs on.

However, my years of planning for failure did bring something: the eBFP CNI with BGP support, the placement rules and otherwise settings that allow for future expansion to be seemless, were in place, ready for the next step.

And so I did what you do when McGyver infrastructure starts carrying real weight: I took it up a notch. That's the next post - what it actually took to turn a drawer-parts cluster into something I'd put my mail on, without pretending I suddenly had a datacentre budget.

Every migration story you read online starts at the destination. Here's the Talos config, here's the cluster, here's the beautiful YAML. What gets skipped is the part that actually explains the decision: what the thing looked like before, and what specifically made the owner finally snap. So before I show you any metal, you get the archaeology - the Proxmox estate I was actually running, the drag it generated, and why the answer turned out to be an immutable, API-driven, deliberately minimal Linux I'd never heard of three months before I installed it.

And the story is far more embarrassing, frankly, but the lessons are the reward in their own way.

What I actually had

The honest inventory of my homelab circa "before": a couple of Proxmox hosts running Fedora VMs, and inside those VMs, Docker - deployed the way most people deploy Docker inside VMs, which is docker-compose up -d over SSH, with the container configs living partially in git, partially in /etc, and mostly in the one place that always loses: my memory. What flags did I pass to that container in 2022? Good question. The container knows. Ask it after the next reboot.

And this wasn't laziness on my part - it's what the software ecosystem made the setup be. Ten years ago, a homelab sysadmin installed services from packages. .deb, .rpm, systemd unit shipped by the maintainer, apt install, done. The distro was the packaging layer, and the packaging layer was someone else's job. Today, almost everything self-hostable ships as a Docker image and only as a Docker image. Which means the sysadmin job hasn't shrunk - it's just moved. Now you are the packaging layer: you write the compose file, you map the volumes, you wire it into systemd (or don't, and let the Docker daemon be your init system, which is its own genre of regret), you handle the update cadence by hand. Every new app is a small packaging project, and every one of those projects is another piece of hand-maintained glue that will silently rot.

Backups, meanwhile, were real but asymmetric. The databases were covered - proper snapshots, taken on a schedule, tested more than never. What was not covered was the system around them: the OS state, the compose files, the eighteen months of small undocumented decisions. I could restore my data. What I could not restore was the machine that served it, because that machine was a sediment layer, and sediment doesn't restore.

If any of this sounds familiar, congratulations, you have a normal homelab. None of it was broken in the "things are down" sense. It was broken in the worse sense: it worked, and the price of keeping it working was me being a full-time, unpaid SRE for a platform nobody but me used.

The real drag

Homelab drag doesn't announce itself. It accumulates the way silt does - each individual deposit is trivial.

It's the Friday night you spend upgrading containerd on one VM because you vaguely remember an advisory, and by the end you're three VMs divergent with no idea which version is running where. It's wanting to try something new - a proper observability stack, say - and realizing the cost isn't learning the new thing, it's re-hand-integrating it with the old sediment. It's apt upgrade on a VM you forgot about, watching it reboot, and feeling genuine dread because you don't actually know what lived on that box anymore. It's the snowflake problem in its purest form: every host is special, and every host being special means every host is a liability.

The real killer, though, was a subtler one. I was a professional platform person spending evenings doing artisanal, hand-crafted infrastructure work - the kind of thing I'd flag in a customer's environment review, gently, with a diagram. I had built, at home, the exact setup I spent my day job walking customers away from: a system whose reliability depended on one specific human remembering things. I was the snowflake. The homelab wasn't under version control; I was the version control, and I had no rollback strategy.

And what made it a conviction rather than a nagging feeling: Kubernetes fixes the interesting half of this - the scheduling, the healing, the declarative state - but it does nothing about the host underneath. A hand-rolled Ubuntu VM running Kubernetes is still a hand-rolled Ubuntu VM. It still drifts. It still has eighteen months of undocumented apt decisions in it. The old joke is that the hardest part of Kubernetes isn't Kubernetes, it's Linux - and if your Linux is artisanal, you've just put a very sophisticated scheduler on top of a very unsophisticated base and hoped the base never embarrasses you. It will. It's Linux. That's the deal.

Enter Talos

While it has drawbacks, getting on the Kubernetes train late also has its advantages: tens of projects, with the successful ones surviving. And lots of presentations provided me plenty of insights on what I should now be looking for. And this is how, I went looking for an operating system that behaves like infrastructure instead of like a pet. The requirements were simple but quite blunt:

1. The host is not mine to mess with. If I can SSH in and hand-edit things, I will, and in three months it's sediment again. I needed an OS where that door is welded shut - not by policy, by design.
2. Everything is declared somewhere, and "somewhere" is git. The node's state should be a file I can diff, not a history I have to excavate.
3. Upgrades should be boring. Atomic, image-based, testable - closer to "flash a new appliance firmware" than "apt dist-upgrade and pray."
4. Minimal surface. If the host is only ever a Kubernetes node, it doesn't need a package manager, a shell, or opinions. Every extra component is another advisory to track forever.

Talos Linux is basically those four requirements turned into a product. It's an immutable, minimal, API-only Linux that runs Kubernetes and nothing else - no SSH, no shell, no package manager. You don't administer a Talos node; you talk to it over an API, and the node's entire configuration is a YAML file. Want to change something? Edit the file, apply it, and the node converges. Want to upgrade? Push a new image version - the OS is itself a container image, more or less, which is a pleasing bit of recursion. There is no state on the node worth preserving, which means there is no state on the node worth fearing.

But here's the part I want to underline, because it's the part everyone skips: I didn't put it on metal. Not first. I installed Talos inside my existing Proxmox VMs - same hosts, same VLANs, same network plumbing, everything reused. The VMs were my training wheels, deliberately.

The reason is boring and physical. I don't own a KVM. I don't own a portable monitor. My "cluster" lives on hardware in a corner, and the moment something goes wrong at the hardware layer - a bad boot, a bricked config, an EFI quirk - the recovery procedure is: unplug the machine, carry it to a desk, find a monitor and a keyboard, sit there with a USB stick like it's 2004. Every bare-metal mistake is a field trip. A VM mistake is a qm destroy and ten seconds. When you're learning a new OS - and Talos makes you relearn things, from disk layout to how you even read logs - you want your mistakes to be cheap, repeatable, and performed from a chair. Proxmox snapshots gave me exactly that: I could break a Talos VM, roll it back, and break it again differently, all without standing up once.

So the graduation path was deliberate: prove the cluster design inside the VMs - the control plane, the CNI, the config surface, the upgrade path - and only move to bare metal once the process was boring. If you're planning the same jump, steal that ordering. The VM phase isn't a detour; it's the rehearsal for the part where mistakes cost a desk trip.

The security posture was what sold me, honestly. I'll write a whole post on it later, but the short version: my prior homelab's attack surface included every box I'd ever forgotten about. Talos' attack surface is an API endpoint, mTLS, and a kernel that ships without a shell to abuse. For someone who cares about data sovereignty and runs his own IAM and email on bare metal, an OS that structurally can't be casually logged into is not a nice-to-have. It's the point.

I want to be fair to Proxmox here, because Proxmox is good software and plenty of people run beautiful clusters on it. The failure wasn't Proxmox's. The failure was mine: Proxmox gives you more ways to touch the system by hand, and I had demonstrated over several years that when given those ways, I used them. The fix wasn't a better hypervisor. The fix was removing the temptation - making the drift physically impossible rather than merely discouraged. (And to be clear: Proxmox stayed. It's a fine tool. It just stopped being the substrate of the thing I actually cared about.)

And so there we are

This is the blog's spine showing up in a specific decision. Bare-metal Kubernetes is worth it to me because it trades a kind of convenience I was abusing for a kind of control I actually wanted - and Talos is the same trade one layer down. Mutable OS with a shell: convenient, in the way credit cards are convenient. Immutable OS with an API: a little friction up front, and in exchange the system can't quietly become something you no longer understand. That's the whole deal. Power and sovereignty, paid for in time and patience.

Interviews reveal a lot. What they can't reveal is the gap I want to talk about, because it never shows up on a CV. It shows up at 3am, when something breaks and the person on call has excellent cloud instincts and no idea where to start looking.

A lot of engineers now start their careers entirely in the cloud, and the troubleshooting instincts you used to pick up from bare metal in your first years are getting skipped. The cloud is a fine place to start a career; it's where the jobs are, and I'm not going to pretend otherwise. But some part of the craft only transfers if you learned it with your hands, and I think we're quietly losing it.

I want to be careful with this, because it's one trivia question away from being a boomer rant. It isn't one. Nobody is better for having suffered through SCSI termination (okay, marginally better, but that's not the argument). The argument is that the cloud hides the layers, and the layers are where the answers live.

What the metal teaches you, without asking

If you read about the rack in my studio, you already know I learned this the slow way. Pulling cable through a room I also slept in. Labeling both ends of every lead because future-me would otherwise be crawling under a desk at midnight. Watching link lights like a hawk because a port that negotiates at 100Mbps instead of 1Gbps doesn't announce itself, it just makes everything mysteriously slow.

That phase teaches you things no course bothers to mention. That every problem is physical somewhere, even the ones that look purely logical. That the machine you can't SSH into still has a serial console, and the serial console will tell you the truth when the network won't. That a failed disk isn't an incident, it's a Tuesday, and the RAID rebuild will run all night while you watch the percentage crawl and question your life choices.

The biggest lesson is an attitude, not a fact: you learn to ask "what am I actually standing on?" Before the OS, there's firmware. Before the firmware, there's hardware. Before the hardware, there's a cable somebody crimped badly, and it's always possible that somebody is you.

Troubleshooting is a layered habit

The actual skill the cloud skips isn't hardware knowledge. It's the habit of isolating from the bottom up.

Is it plugged in? Is there a link? Does it have an address? Can it reach its neighbor? Does DNS resolve? Does the service respond? One question at a time, one change at a time, and you observe after every change instead of shotgunning five fixes at once and hoping one lands. When you've troubleshooted at the physical layer, where the feedback loop is "walk over and look at the light," this discipline gets drilled into you early, and it never leaves.

Now watch what happens when someone who has only known the cloud hits a real problem. The application can't reach the database. The cloud-native reflex is to restart the pod, bump the memory limit, and check the managed database's dashboard. The metal-raised reflex is to ask what exists between the app and the database: a network policy, a route table, a security group, a DNS record, an ARP entry. The pod restarts and nothing changes, because nothing was wrong with the pod.

The instincts transfer to the cloud with almost no loss, and that's the irony: the cloud is just someone else's metal. The VPC is a switch you didn't have to cable. The security group is a firewall you didn't have to configure from a blank file. The managed database is a machine in a rack, in a building, with power and cooling and a disk in it that will eventually die. Someone else babysits the bottom layers, which is a legitimate trade. But you should know what they're babysitting, because when one of those layers leaks, and they do leak, the person who has seen the thing under the thing recovers fastest.

Why I still build it the hard way

This is where the blog's whole thesis shows up again, and I'll keep it short because it's been said before: I run real hardware at home partly for privacy and sovereignty, but partly because it keeps that layered habit alive. When the 1am VLAN lockout happened in my studio, I learned more in two hours than any certification ever taught me, precisely because every layer in the path was mine: my switch, my firewall, my config mistake. There was no provider to blame, no ticket to file, and no abstraction to hide behind.

There's a version of engineering where everything is someone else's responsibility except the top layer. It's comfortable and it pays well and I don't think it makes you a worse person. I just think it makes you a more fragile engineer, and fragility only becomes visible during incidents, which is exactly when it's too expensive.

So if you're early in your career and you've only ever worked in the cloud: go break something you own. A spare box, a Raspberry Pi, a old laptop with Linux on it. Cable it, address it, break the network on purpose, and fix it from a serial console with no dashboard in sight. It's cheap, it's private, and no interview question can take the instincts from you afterwards.

My opinion here is not a small one. It's this: Docker, runc, and the whole container runtime stack work perfectly well on their own - and somewhere along the way, an entire generation of engineers learned to operate the parts without ever meeting them.

Part of my job for years has been interviewing infrastructure and platform engineers, and I want to talk about what those interviews keep revealing - not to dunk on candidates, because the candidates are a symptom, not the disease, so to speak. The disease is that the ecosystem taught people to use abstractions without touching the substrate, and then wondered why nobody knows what a container actually is.

The two failures I see over and over

The first one I've already hinted at: someone tells me about their Kubernetes experience, and after two minutes it becomes clear the experience is using clusters. Deploying to them. Writing manifests against them. Maybe running a managed offering's dashboard. That's real work, and I don't dismiss it at all. But building and operating a cluster - picking a CNI, sizing control planes, troubleshooting a node that won't join, living through an upgrade - is a different skill, and the resume language doesn't distinguish between the two. "Kubernetes experience" covers both, which is a bit like saying you have "car experience" because you've been a passenger.

The second one is more basic and somehow more common: people who list Kubernetes & Docker on their CV, yet then can't answer the fundamental questions. What actually happens when you run docker run? What's the difference between a container and a process? What is an image layer, and where does it live? I'm not asking about runc internals for sport - I'm asking because if you can't answer these, you can't debug the system when it behaves strangely, and it will behave strangely, usually at 2am, usually on the node that has all your stateful workloads.

VMs are not containers, and the difference is not trivia

Then there's the question that separates the people who read about infrastructure from the people who run it: what's the actual difference between a VM and a container?

The correct answer is not "containers are lighter." The correct answer starts with: a virtual machine virtualizes hardware - it has its own kernel, bootstrapped by a hypervisor - while a container shares the host kernel and isolates processes using kernel primitives: namespaces for what they can see, cgroups for what they can use. A container is not a tiny computer. It's a process with an elaborate costume on.

Why do I care that you know this? Because it's not trivia - it's the answer to real operational questions. Can I run a container with a different kernel version than the host? No. Can I get better security isolation by reaching for a VM? Yes, genuinely, sometimes that's the right call. Why did my container's memory limit kill my process with SIGKILL instead of throttling it gracefully? Because cgroups enforce, they don't negotiate. If you know what a container is, these questions answer themselves. If you don't, you're cargo-culting YAML and hoping.

The interview moment I actually love

There's a twist, though, and it's the reason this post is affectionate rather than grumpy: my favorite interview moments aren't the candidates who nail every answer. They're the ones who, when they hit something they don't know, look at me and say, honestly: "I don't know."

That's it. That's the whole move. And it's rarer than it should be, because somewhere along the way our industry decided that not knowing something is a fireable offense in an interview, so people guess, bluster, or worse - confidently state something wrong with the energy of a keynote. I can forgive "I don't know" a hundred times over. What I can't forgive is the person who doesn't know and pretends to, because that's the person who will page you at 2am having already "fixed" something.

The candidates who say "I don't know" and then reason out loud anyway - "well, it's probably X because of Y, but I'd want to check" - are the ones I fight to hire. That behavior in an interview is exactly the behavior you want in an incident. Honesty plus reasoning beats confident nonsense every single time.

CV-driven Kubernetes, a field guide

And I'd be lying if I said the "use Kubernetes without understanding containers" pattern was always innocent. A lot of it is, but some of it is grubbier: tech choices made for the CV, not the system.

My favorite specimen is an engineering manager who once told me, with complete sincerity, that I was insane not to use Kubernetes - because Kubernetes is where Java runs best. I still don't fully know how to parse that sentence, but the part that made it tragic rather than merely wrong was his team was still on Java 8. If you know anything about the JVM, you know Java 8 was essentially container-blind: it read the host's memory and CPU, not the cgroup limits, so a container capped at 4GB happily told the JVM it had 64GB and got OOM-killed for its optimism. (Java eventually fixed this - the fix landed in 8u191 and got real in Java 10+, so it's dated knowledge, but that's exactly the point: it's the kind of thing you only know if you've run a JVM inside a cgroup and watched it die.) He was championing the platform as the JVM's natural habitat while his own apps were running un-containerized, un-aware JVMs from 2014. That one I can almost forgive - a weak opinion loudly held is an occupational hazard of management.

What I can't forgive is the other species: developers who attend a conference talk, or read a blog post about ArgoCD, and then spend the next six months lobbying everyone around them into adopting the whole stack - the manifests, the GitOps flow, the rollout of the rollout - all of it trickling downhill to other people to actually implement and operate. They get the bullet point on the CV: "led Kubernetes adoption." The person who led it is the one rebuilding the cluster at midnight because nobody who chose it had ever run one. I have a long list of grudges against Kubernetes, and the unfair part is that almost none of them are really about Kubernetes. They're about people who chose it as a career move and made it someone else's problem.

So what are containers, really?

Stripped of the marketing: a container image is a bundle of files and metadata. A container runtime (runc, historically via Docker, today usually via containerd) takes that bundle and asks the Linux kernel to do three things: give this process its own view of the world (namespaces), cap what it can consume (cgroups), and start it inside a root filesystem that isn't the host's (pivot_root, or chroot on steroids). That's it. That's the magic. Everything else - Docker the CLI, Compose, Kubernetes, your Helm charts - is orchestration and convenience layered on top of a kernel feature that has existed for years and works just fine by itself.

You can run production workloads with nothing but systemd units and docker run commands. People do. It's not good, in the sense that nobody's built the self-healing, declarative, fleet-scale layer for you - but it's legitimate, in the sense that the container itself doesn't need Kubernetes to exist, any more than a process needs systemd to run.

Where Kubernetes actually earns its keep

Kubernetes is not the thing that makes containers work. It's the thing that makes hundreds of containers across dozens of machines work together - scheduling, rescheduling, healing, declaring intent, reconciling reality with a spec. That's an enormously valuable problem to solve... if you have that problem.

If you have three services and one box, Kubernetes is not the answer; it's a very elaborate way of making a simple system complicated. If you have two hundred services, six clusters, and a team of eight, Kubernetes is not optional; it's the only sane way to stay in control. The skill is knowing which situation you're in - and you can't know that if you never learned what the parts underneath actually do.

So if you're early in your career and the advice you're absorbing is "learn Kubernetes": fine, learn Kubernetes. But, learn Docker first. Actually run a container without an orchestrator. Actually break one. Actually read ps output and find your containerized process living smugly among the host's other processes. Learn what the kernel is doing for you, because the day something breaks - and it will - that's the layer where the answer lives.

The people who know that layer are the ones I hire. The people who pretend to are the ones I remember.

The confession, then: I was late to Kubernetes, and for years I dismissed it with exactly the energy I'd once used against OpenStack.

But I've reread my old objections recently - some of them in the drafts for this very blog - and I'm not going to stand here and tell you I was wrong, because the honest answer is: yes and no. And the "no" part is where most of this post lives.

Start with what I never bought, and still don't. Kubernetes is not a fully managed service, even on AWS. Let's be precise about this, because the marketing has done decades of work blurring it: EKS gives you a managed control plane. The upgrade burden - the part that actually hurts - stays with you. Every Kubernetes version bump drags CSI, CNI, and the entire add-on ecosystem behind it, and every one of those components has to be kept aligned, tested, and maintained, by you, forever. "Managed Kubernetes" is a management service for the part of Kubernetes that rarely breaks, while the part that actively fights you remains your problem. I'll die on this hill holding a decent amount of company.

And when Kubernetes was young, the things that make it tolerable today simply didn't exist. No network mesh to speak of. Network policies: absent - they arrived as a beta in Kubernetes 1.3 in July 2016, and only went stable a year later. Tight, real integration with AWS IAM for role-based access: absent, or at best clunky enough that "clunky" is a compliment - IRSA, the thing that finally made pod-level IAM clean on EKS, landed in September 2019, four years after 1.0. The security best practices everyone now recites like scripture? Ignored, broadly, for convenience - and I watched people confidently expose clusters in ways that would get a platform team fired today. The ecosystem had to grow into something defensible, and for years it wasn't.

flowchart TB
    A["Jul 2015 - Kubernetes 1.0"] --> B["2016 - CNI spec"]
    B --> C["Jul 2016 - NetworkPolicy beta, Kubernetes 1.3"]
    C --> D["Mar 2017 - RBAC stable, Kubernetes 1.6"]
    D --> E["Nov 2017 - EKS announced"]
    E --> F["Feb 2018 - ECS awsvpc task networking"]
    F --> G["Jun 2018 - EKS generally available"]
    G --> H["Sep 2019 - IAM Roles for Service Accounts"]
    H --> I["Nov 2023 - EKS Pod Identity"]

Even ECS, which I eventually came to love, took me a while - I was reluctant until the day the networking drivers caught up - awsvpc task networking, February 2018 - and every task got its own ENI, which meant security groups could finally be done right. Per-task security groups. That was the moment ECS stopped being "the other option" to me and started being the sane one.

That was also the era when "multi-cloud" curdled into a monstrosity. Whole companies and platforms were built around abstracting away every vendor-specific feature, so that no one would ever, ever be locked in. I never drank that. I was firmly, unapologetically pro vendor lock-in - provided you understood what you were actually buying. Because what people call lock-in, done deliberately, is an ecosystem. You buy in, and then you put the cross-services to work: the identity system that already knows your users, the observability stack that already sees your network, the security tooling that's already wired to everything else. Abstract all of that away in the name of portability and what you get isn't freedom. It's the lowest common denominator of three clouds, running at three times the operational cost, so that a migration you'll probably never do stays theoretically possible. Portability is a feature you pay rent on. Lock-in is an investment that pays dividends. I know which side history came down on, and it wasn't the abstraction layer's.

So yes: I was late, I was loud about it, and a fair amount of what I said aged well.

With that said, the truth is far more interesting than either "I was right all along" or "I saw the light." When EKS and its siblings finally became genuinely viable - EKS went GA in June 2018, and by my book the IAM story only stopped being clunky with IRSA the year after - I didn't flip to enthusiasm. I ran the numbers for my situation, and by all account, they said no. And I want to walk you through that decision, because it's the most honest architectural call I've ever made, and it had nothing to do with whether Kubernetes was good.

A fair question at this point would be: why all of my Kubernetes sits outside of AWS? The answer is that it was my decision to make, and I made it deliberately.

EKS still means owning upgrades. In a distributed team of eight engineers, from graduates to seniors, keeping up with the number of clusters we would have needed would have cost an incredible amount of man time - even at a plain Docker level of operations, before you add a version-bump treadmill on top. I would have loved to spend that time on Kubernetes sooner. I genuinely would have: looks great on a CV, get hands-on experience, experiments, and so on. But I could not justify the spend, and pretending otherwise would have been architecture-by-hobby or CV-driven as I call it.

So I pushed for the team to standardize on AWS ECS. And I'll say the thing that sounds like a humblebrag but is actually the entire point: in five years at Sainsbury's, my team and I never once had to perform orchestrator-level maintenance. Not once. Because ECS is fully managed - not "control plane managed," fully managed. The upgrade treadmill doesn't exist because there's no treadmill. Eight engineers of wildly varying depth, zero orchestrator maintenance, five years. You can disagree with the choice; you cannot argue with that arithmetic.

However on-prem - where that trade-off evaporates, where there's no per-cluster version treadmill priced against a small team's calendar, where the machines are mine and the time is mine to spend - I ended up with Kubernetes. No ambivalence, no grudge from the old arguments. Because the same analysis that said "ECS" for a small team on AWS says "Kubernetes" when the constraints are different. That's not inconsistency. That's the whole job. The right orchestrator was never a religion; it was always a function of who's operating it and who's paying for the treadmill. And I do say so even after running workloads with AWS ECS Anywhere for years: doing that I still depend on AWS and a working internet connection. And have to pay for the privilege to run my own hardware.

Which brings me to the thesis of this blog, and why it needed four posts of biography to get to it:

Bare-metal Kubernetes is not a rebellion against the cloud. I love the cloud; the cloud pays for the rack. It's the conclusion of everything I've told you across these posts: the kid who liked knowing where things are, the student who gutted his gaming rig because the architecture was better than the gaming, the Eucalyptus support engineer who watched a little project win every fair fight and still lose every budget, the platform lead who refused to buy portability he'd never use. At home, on metal I own, Kubernetes is the first system that gives me cloud-grade capability - scheduling, self-healing, declarative everything - without asking me to outsource a single piece of the ownership I've been fighting for.

It is also a hobby the way a dog is a hobby. The upgrades are mine. The breakage is mine. I accepted all of it with eyes open, because for me - and I want to stress that this is a me decision, not advice - the trade is worth it.

So yes, I was late. Late to kube. Early to everything else.

And so it begins.

A confession first: I never ran OpenStack.

Not once. Not at work, not at home, not in a lab, not out of curiosity. I refused it - fiercely, the way some people refuse a religion - and I want to explain why, because the reason turns out to be the same reason I was late to Kubernetes, years later, and that's when the name of this blog stops being a joke.

By the time I finished my masters, the private-cloud wars were in full swing, and they were fought the way wars of that era were fought: every vendor had a distribution, every distribution was the only supported way, and every slide deck promised you a private Amazon. But the war I was actually watching had a specific shape, and it was two projects going at each other, and I had already picked a side.

The side I picked was called Eucalyptus. AWS-compatible. Open source. The pitch was simple and, to my mind, correct: if you're going to run your own cloud, it should speak the same API as the real cloud, so you're never locked in and never locked out - the same skills, the same tools, the same muscle memory, on hardware you own. OpenStack was the other camp. The story everyone told at the time - and I can't verify it, but everyone told it - was that OpenStack began as a Python fork of Eucalyptus, started by people who simply didn't want to use Java. Whether or not that's the full history, it tells you everything about the era: a big chunk of the "next generation of cloud" was born as a language preference with a marketing budget.

I picked Eucalyptus for the philosophy. What I didn't know was that it would pick me back.

They hired me as support before I'd finished my masters. I want you to sit with that for a second, because I certainly didn't: I was still a student, and a company whose product I ran in my own studio rack looked at my, let's say, enthusiastic forum presence and my weird little homelab and decided I was employable. Then they moved me to the UK.

And then - within the first few weeks - they made me TAM for Nokia.

Their biggest customer. By a landslide. My onboarding period could be measured in weeks; the customer I was made responsible for was larger than some of the vendors we competed against. I don't recommend this to anyone, and I'd do it again tomorrow. There is no better way to learn a product than to have it be the beating heart of a contract worth more than your future salary, and to be the person whose phone rings when its heart skips.

Nokia meant Finland, and I'll put it on the record: the warmest, most competent people I've ever had the luck to work with. I landed there knowing nobody, and I was adopted within the hour - the welcome never once wore off across the whole engagement. And underneath the hospitality sits a level of competence that raised my own game: they knew their estate down to the last VLAN, they'd read everything before the call started, and their questions were sharper than half the war rooms I've sat in since.

One thing that struck people at the time - and this comes up again and again in my career, so let me get it on the record now - was my age. Everyone put me in my mid-30s. Not only because I looked older, though I did. Because I kept up. I could hold my end of a network conversation, a systems conversation, an architecture conversation, with seniors who had been in the trenches for fifteen years longer than I had, and I could hold it without flinching, because I'd already hit every one of their war stories in miniature, at home, alone, at 1am, in a studio where a rack lived where the wardrobe should have been. The homelab wasn't a hobby. It was a compressed apprenticeship. It just took a while for the industry to price it correctly.

And those were the vendor wars from the inside, for the record: I wasn't in the crowd watching distributions fight. I was in the arena. I knew exactly which competitor features were real and which were roadmap. I knew how the same slide deck was rewritten for three different audiences. None of it made me cynical, exactly. But it did teach me to read vendor announcements the way you read weather reports: with an umbrella in the bag regardless.

Then HP bought us, and the shape of the war changed.

I want to be precise about why, because it was one of the strangest strategic moves I've ever been adjacent to. HP had enterprise customers. Those customers wanted to go to AWS. HP's problem was obvious: if the customers left, HP lost them. So the play was to keep them on-prem by giving them AWS - the same APIs, the same features, the same tooling - via Eucalyptus, so that moving to the real cloud later would be a migration, not a conversion. We weren't the product; we were the bridge. Bare metal couldn't do elastic, and Eucalyptus could, and suddenly the little AWS-compatible cloud nobody had heard of was a retention strategy for one of the biggest hardware companies on earth.

Which meant that while every cloud conference was a contest of investment announcements - who's pouring how many millions into OpenStack this year, as if dollars were a benchmark - we were the very little guys. We had no armada of integrators. We had no roadshow budget. Our name recognition was a rounding error, and I won't pretend it wasn't the single biggest wall we faced: half the objections we hit weren't about the product at all, they were about the logo on the slide.

But here's the part I've carried with me ever since. Wherever we could get through the front door - wherever we actually got a technical evaluation instead of a procurement-level dismissal - we won. Not sometimes. Consistently. Because the moment an engineer put both clouds side by side and tested what mattered - the API compatibility, the actual behavior, the operational reality - the marketing evaporated and we were still standing. Every single one of those wins was earned against the grain of a market that had already decided, before the first test, that the little Java project with the tree name couldn't possibly beat the one everyone was investing in.

It taught me a lesson I've relearned in every fight since: name recognition decides who gets evaluated. It almost never decides who gets kept.

So why does this end with a refusal of OpenStack rather than a grudging adoption? Because everything I'd lived through - the all-inclusive socket, the gutted rig, the Nokia TAM phone, and now the little cloud that won every fair fight and still lost every budget - pointed at one belief I couldn't shake: the private cloud as a product category was a machine for organizations, not for people. And Eucalyptus was the last private cloud I believed in, because it was the only one that admitted what it actually was: an API boundary, not a lifestyle.

You can guess what happened to it. You can also guess what the industry did instead. And you can probably guess what I said, for years, when people told me Kubernetes was the obvious next step - the exact same thing I'd said about OpenStack.

The first thing you should know about a 42U rack is that it is bigger than you think. The second thing is that it is louder than you fear. I learned both in a university studio apartment that measured, generously, eighteen square meters.

It started innocently, the way these things always do. A tower under the desk. Then a second one, because the first one was doing DNS and couldn't also do storage. Then a shelf, because towers stacked on towers are how studios catch fire. Then, one day, someone on a forum was offloading a half-empty rack for the price of transport, and I did the math on my floor space and - this is the part I still can't defend - decided the rack could live where the wardrobe was supposed to go.

I did not have a wardrobe for three years. I had infrastructure.

Which brings us to the noise. Here is the part where every homelab war story tells you the neighbors nearly called the cops - and I have to disappoint you, because mine never minded. Not once. The rack settled into the building the way a refrigerator does: background, ignorable, mine. I'd like to claim this was diplomacy, but honestly? I think the floor just accepted us as a unit, the way buildings accept old radiators. No - the pressure came from a completely different direction, and it took me an embarrassingly long time to see the pattern. Every time I brought someone home, there would come a moment, usually somewhere between the second drink and the first raised eyebrow at the glowing wall of a wardrobe-shaped object, when she would ask the question my heart feared most: "Can you turn it off?"

The neighbors never asked me to power it down. The women I dated did. And I never did. Draw from that whatever conclusions about my twenties you like; the rack and I are still here.

My friends, meanwhile, thought I was insane, and their case was solid. While everyone else my age was out living their best lives between nineteen and twenty-two, I was in a studio with the lights off, working out why a Windows Server 2008 domain controller had opinions about my evening. I was chasing a dream that sounded insane even as I said it out loud: VDI with GPU passthrough - my desktop, running on a blade in the rack, streamed to whatever screen was in front of me. The sacrifice that still stings is the one I made for it. I gutted my gaming rig. Not sold it, not retired it - donated, organ by organ, into the rack. The GPU went into a hypervisor host so that a virtual machine could pretend to be a gaming PC, worse than the real one had been, and I called it progress because architecturally, it was.

And when it worked? When the desktop materialized on the laptop screen, rendered by hardware bolted into a machine humming across the room? It was the single most motivating thing I've ever built. I never went back to a desktop under a desk. That chase - better performance, better design, better architecture - never stopped. It's still going. It's the reason this blog exists.

Not every night was a victory. Some nights I spent hours deep in VLANs, moving interfaces between networks at 1am, learning the specific lesson that only ever teaches itself once: that the switch port you just reconfigured was the one you were connected through. There is a flavor of silence that exists only in a studio apartment where the network has just cut off its own administrator, and the rack is still humming, unreachable, and it is 1am, and you are nineteen. I know that silence intimately. I earned it honestly, more than once.

It was around then that Microsoft and I quietly separated. I stopped using anything of theirs and started what I can only call a love affair with open source: pfSense on the edge, CentOS everywhere else, and VMware ESXi underneath it all - which, at the time, was generous to students in a way that bordered on charity, and which I abused accordingly and without a single shred of guilt. That stack ran my studio for years, and it taught me more than my degree did, and it cost me nothing but sleep.

There are photos from those years. I look at them and I can still tell you, machine by machine, exactly which one ran what, and to what purpose. Which tower was DNS. Which one held the media. Which one was the sacrificed gaming rig, gutted and rededicated, fans still whining like a betrayed animal. Most people's student photos are of parties. Mine are of cabling, and I can read them like other people read albums.

But somewhere in there - between the sacrificed rig and the 1am lockouts and the fourth girlfriend asking, gently, "can it at least sleep when you sleep" - it stopped being a hobby. That's the real shift, and it isn't about size. A hobby stops when you're tired. Infrastructure doesn't get that option. The night I left for a weekend trip and spent the train ride not reading my book but thinking "did I actually confirm the backup cron ran before I locked the door" - that's when I knew the machines weren't toys anymore. They were a small, loud, private utility company with one customer and one employee, and I was both.

People ask what I got out of it, and the honest list is short: a gutted gaming rig I never regretted, a dating life with a recurring structural antagonist, and the most expensive education available in networking, storage, and failure modes - tuition paid in hardware that died exactly when I needed it most.

I was born before ADSL existed. Not before the internet - I'm not that old - but before it became something you lived in. The internet of my childhood was a thing you did on purpose, like going to the library, except the library was in the living room and made a sound... You've got to hear it to understand.

For those who never heard it, and for the nostalgic among us.

The first computer I remember wasn't mine. It was the family's, which meant it was mine only in the sense that I was the one who kept touching it. The hard disk was small enough that installing anything felt like a negotiation. Delete a game to make room for a new one. Copy your save files to a floppy first, because you knew - everyone knew - that the reinstall would go wrong and you'd need to roll back. We kept our lives on 1.44 megabytes, and I don't remember it being painful. I remember it being exact. You counted. You chose. Nothing was abstract yet.

Then the CDs arrived, and with them the smell. You know the one. That faint plastic-and-ink smell when you crack a shrink-wrapped box, the manual still stiff, the disc seated in its little foam ring. An operating system used to arrive as an object. You held the whole thing in one hand. Windows 98 was roughly the weight of a sandwich. I unboxed my first OS like it was a gift, because to me it was - someone had decided what computing was, printed that decision on 200 pages, and handed it to me at a shop.

I didn't know it then, but that's the era I keep returning to when people ask why I run my own servers. Everything since has been a slow removal of edges. The disk became a download, the download became a stream, the stream became something that's just there, on someone else's machine, in a country you'll never visit, under a company you've only ever seen as a logo on a billing page. Convenient. Extremely convenient. I won't pretend I didn't enjoy every step of it.

But the floppy era taught me a lesson I've never unlearned: I like knowing where things are.

The first machine that felt like mine rather than the family's was nothing special. I've seen the spec sheet in my memory so many times that the actual numbers have worn away, and I've decided I prefer it that way. What matters is that it was the first one where the question wasn't "what can this computer do?" but "what can I make this computer do?" That question has cost me an enormous amount of money, sleep, and kitchen table space, and I regret none of it.

So here's where this blog is going, and why the name is a small joke I'll explain later.

Most of what I do for a living is cloud. Real cloud, big cloud, the kind with regions and availability zones and a support plan. I'm not here to tell you the cloud is bad - that would be a strange thing for someone whose paycheck depends on it, and also it would be untrue. The cloud is astonishing. I like the cloud.

But at home, on hardware I can touch, I run Kubernetes. Not on a Raspberry Pi cluster arranged on a shelf for the photo, though I've done that too, and the photo looked great. I mean actual metal. Machines with fans you can hear, disks you can pull, a rack that hums in a corner of the house and draws real watts off a real circuit that I wired. And the honest pitch is not "it's better." The honest pitch is this:

I like knowing where things are. Bare metal is where things are.

When my homelab works, I know exactly which machine, which disk, which cable, which rule in which firewall. When it breaks - and it breaks, regularly, at inconvenient hours - I know exactly which machine, which disk, which cable, which rule. There is no dashboard standing between me and the failure. The failure is physical. I can put my hand on it. Some people would call that a disadvantage. I call it the closest thing to the floppy-disk era that exists in 2026.

The catch, and I will never downplay this on this blog: it costs time. An enormous amount of time. Kubernetes on bare metal is a hobby the way a dog is a hobby - it's fine when you're there, and it knows exactly when you're not. The cloud gives you weekends. The rack does not. If you're reading this deciding which side to pick: the cloud is probably right for you, and I say that without condescension, because for years I picked the cloud too, and it was the right call, and I still make it every day at work.

This blog is about the other side of the ledger. The part you can't bill.