The confession, then: I was late to [Kubernetes](https://kubernetes.io), and for years I dismissed it with exactly the energy I'd once used against [OpenStack](https://www.openstack.org). But I've reread my old objections recently - some of them in the drafts for this very blog - and I'm not going to stand here and tell you I was wrong, because the honest answer is: yes and no. And the "no" part is where most of this post lives. Start with what I never bought, and still don't. Kubernetes is not a fully managed service, even on AWS. Let's be precise about this, because the marketing has done decades of work blurring it: EKS gives you a managed *control plane*. The upgrade burden - the part that actually hurts - stays with you. Every Kubernetes version bump drags CSI, CNI, and the entire add-on ecosystem behind it, and every one of those components has to be kept aligned, tested, and maintained, by you, forever. "Managed Kubernetes" is a management service for the part of Kubernetes that rarely breaks, while the part that actively fights you remains your problem. I'll die on this hill holding a decent amount of company. And when Kubernetes was young, the things that make it tolerable today simply didn't exist. No network mesh to speak of. Network policies: absent - they arrived as a beta in Kubernetes 1.3 in July 2016, and only went stable a year later. Tight, real integration with AWS IAM for role-based access: absent, or at best clunky enough that "clunky" is a compliment - IRSA, the thing that finally made pod-level IAM clean on EKS, landed in September 2019, four years after 1.0. The security best practices everyone now recites like scripture? Ignored, broadly, for convenience - and I watched people confidently expose clusters in ways that would get a platform team fired today. The ecosystem had to *grow* into something defensible, and for years it wasn't. ```mermaid flowchart TB A["Jul 2015 - Kubernetes 1.0"] --> B["2016 - CNI spec"] B --> C["Jul 2016 - NetworkPolicy beta, Kubernetes 1.3"] C --> D["Mar 2017 - RBAC stable, Kubernetes 1.6"] D --> E["Nov 2017 - EKS announced"] E --> F["Feb 2018 - ECS awsvpc task networking"] F --> G["Jun 2018 - EKS generally available"] G --> H["Sep 2019 - IAM Roles for Service Accounts"] H --> I["Nov 2023 - EKS Pod Identity"] ``` Even ECS, which I eventually came to love, took me a while - I was reluctant until the day the networking drivers caught up - awsvpc task networking, February 2018 - and every task got its own ENI, which meant security groups could finally be done *right*. Per-task security groups. That was the moment ECS stopped being "the other option" to me and started being the sane one. That was also the era when "multi-cloud" curdled into a monstrosity. Whole companies and platforms were built around abstracting away every vendor-specific feature, so that no one would ever, ever be locked in. I never drank that. I was firmly, unapologetically *pro* vendor lock-in - provided you understood what you were actually buying. Because what people call lock-in, done deliberately, is an *ecosystem*. You buy in, and then you put the cross-services to work: the identity system that already knows your users, the observability stack that already sees your network, the security tooling that's already wired to everything else. Abstract all of that away in the name of portability and what you get isn't freedom. It's the lowest common denominator of three clouds, running at three times the operational cost, so that a migration you'll probably never do stays theoretically possible. Portability is a feature you pay rent on. Lock-in is an investment that pays dividends. I know which side history came down on, and it wasn't the abstraction layer's. So yes: I was late, I was loud about it, and a fair amount of what I said aged well. With that said, the truth is far more interesting than either "I was right all along" or "I saw the light." When EKS and its siblings finally became genuinely viable - EKS went GA in June 2018, and by my book the IAM story only stopped being clunky with IRSA the year after - I didn't flip to enthusiasm. I ran the numbers for *my* situation, and by all account, they said no. And I want to walk you through that decision, because it's the most honest architectural call I've ever made, and it had nothing to do with whether Kubernetes was good. A fair question at this point would be: why all of my Kubernetes sits outside of AWS? The answer is that it was my decision to make, and I made it deliberately. EKS still means owning upgrades. In a distributed team of eight engineers, from graduates to seniors, keeping up with the number of clusters we would have needed would have cost an incredible amount of man time - even at a plain [Docker](https://www.docker.com) level of operations, before you add a version-bump treadmill on top. I would have loved to spend that time on Kubernetes sooner. I genuinely would have: looks great on a CV, get hands-on experience, experiments, and so on. But I could not justify the spend, and pretending otherwise would have been architecture-by-hobby or CV-driven as I call it. So I pushed for the team to standardize on AWS ECS. And I'll say the thing that sounds like a humblebrag but is actually **the** entire point: in five years at Sainsbury's, my team and I never once had to perform orchestrator-level maintenance. Not once. Because ECS is fully managed - not "control plane managed," *fully* managed. The upgrade treadmill doesn't exist because there's no treadmill. Eight engineers of wildly varying depth, zero orchestrator maintenance, five years. You can disagree with the choice; you cannot argue with that arithmetic. However on-prem - where that trade-off evaporates, where there's no per-cluster version treadmill priced against a small team's calendar, where the machines are mine and the time is mine to spend - I ended up with Kubernetes. No ambivalence, no grudge from the old arguments. Because the same analysis that said "ECS" for a small team on AWS says "Kubernetes" when the constraints are different. That's not inconsistency. That's the whole job. The right orchestrator was never a religion; it was always a function of who's operating it and who's paying for the treadmill. And I do say so even after running workloads with AWS ECS Anywhere for years: doing that I still depend on AWS and a working internet connection. And have to pay for the privilege to run my own hardware. Which brings me to the thesis of this blog, and why it needed four posts of biography to get to it: Bare-metal Kubernetes is not a rebellion against the cloud. I love the cloud; the cloud pays for the rack. It's the conclusion of everything I've told you across these posts: the kid who liked knowing where things are, the student who gutted his gaming rig because the architecture was better than the gaming, the [Eucalyptus](https://github.com/eucalyptus/eucalyptus) support engineer who watched a little project win every fair fight and still lose every budget, the platform lead who refused to buy portability he'd never use. At home, on metal I own, Kubernetes is the first system that gives me cloud-grade *capability* - scheduling, self-healing, declarative everything - without asking me to outsource a single piece of the ownership I've been fighting for. It is also a hobby the way a dog is a hobby. The upgrades are mine. The breakage is mine. I accepted all of it with eyes open, because for me - and I want to stress that this is a *me* decision, not advice - the trade is worth it. So yes, I was late. Late to kube. Early to everything else. And so it begins.