Skip to main content

Part 5: The parts work on their own

My opinion here is not a small one. It's this: Docker, runc, and the whole container runtime stack work perfectly well on their own - and somewhere along the way, an entire generation of engineers learned to operate the parts without ever meeting them.

Part of my job for years has been interviewing infrastructure and platform engineers, and I want to talk about what those interviews keep revealing - not to dunk on candidates, because the candidates are a symptom, not the disease, so to speak. The disease is that the ecosystem taught people to use abstractions without touching the substrate, and then wondered why nobody knows what a container actually is.

The two failures I see over and over

The first one I've already hinted at: someone tells me about their Kubernetes experience, and after two minutes it becomes clear the experience is using clusters. Deploying to them. Writing manifests against them. Maybe running a managed offering's dashboard. That's real work, and I don't dismiss it at all. But building and operating a cluster - picking a CNI, sizing control planes, troubleshooting a node that won't join, living through an upgrade - is a different skill, and the resume language doesn't distinguish between the two. "Kubernetes experience" covers both, which is a bit like saying you have "car experience" because you've been a passenger.

The second one is more basic and somehow more common: people who list Kubernetes & Docker on their CV, yet then can't answer the fundamental questions. What actually happens when you run docker run? What's the difference between a container and a process? What is an image layer, and where does it live? I'm not asking about runc internals for sport - I'm asking because if you can't answer these, you can't debug the system when it behaves strangely, and it will behave strangely, usually at 2am, usually on the node that has all your stateful workloads.

VMs are not containers, and the difference is not trivia

Then there's the question that separates the people who read about infrastructure from the people who run it: what's the actual difference between a VM and a container?

The correct answer is not "containers are lighter." The correct answer starts with: a virtual machine virtualizes hardware - it has its own kernel, bootstrapped by a hypervisor - while a container shares the host kernel and isolates processes using kernel primitives: namespaces for what they can see, cgroups for what they can use. A container is not a tiny computer. It's a process with an elaborate costume on.

Why do I care that you know this? Because it's not trivia - it's the answer to real operational questions. Can I run a container with a different kernel version than the host? No. Can I get better security isolation by reaching for a VM? Yes, genuinely, sometimes that's the right call. Why did my container's memory limit kill my process with SIGKILL instead of throttling it gracefully? Because cgroups enforce, they don't negotiate. If you know what a container is, these questions answer themselves. If you don't, you're cargo-culting YAML and hoping.

The interview moment I actually love

There's a twist, though, and it's the reason this post is affectionate rather than grumpy: my favorite interview moments aren't the candidates who nail every answer. They're the ones who, when they hit something they don't know, look at me and say, honestly: "I don't know."

That's it. That's the whole move. And it's rarer than it should be, because somewhere along the way our industry decided that not knowing something is a fireable offense in an interview, so people guess, bluster, or worse - confidently state something wrong with the energy of a keynote. I can forgive "I don't know" a hundred times over. What I can't forgive is the person who doesn't know and pretends to, because that's the person who will page you at 2am having already "fixed" something.

The candidates who say "I don't know" and then reason out loud anyway - "well, it's probably X because of Y, but I'd want to check" - are the ones I fight to hire. That behavior in an interview is exactly the behavior you want in an incident. Honesty plus reasoning beats confident nonsense every single time.

CV-driven Kubernetes, a field guide

And I'd be lying if I said the "use Kubernetes without understanding containers" pattern was always innocent. A lot of it is, but some of it is grubbier: tech choices made for the CV, not the system.

My favorite specimen is an engineering manager who once told me, with complete sincerity, that I was insane not to use Kubernetes - because Kubernetes is where Java runs best. I still don't fully know how to parse that sentence, but the part that made it tragic rather than merely wrong was his team was still on Java 8. If you know anything about the JVM, you know Java 8 was essentially container-blind: it read the host's memory and CPU, not the cgroup limits, so a container capped at 4GB happily told the JVM it had 64GB and got OOM-killed for its optimism. (Java eventually fixed this - the fix landed in 8u191 and got real in Java 10+, so it's dated knowledge, but that's exactly the point: it's the kind of thing you only know if you've run a JVM inside a cgroup and watched it die.) He was championing the platform as the JVM's natural habitat while his own apps were running un-containerized, un-aware JVMs from 2014. That one I can almost forgive - a weak opinion loudly held is an occupational hazard of management.

What I can't forgive is the other species: developers who attend a conference talk, or read a blog post about ArgoCD, and then spend the next six months lobbying everyone around them into adopting the whole stack - the manifests, the GitOps flow, the rollout of the rollout - all of it trickling downhill to other people to actually implement and operate. They get the bullet point on the CV: "led Kubernetes adoption." The person who led it is the one rebuilding the cluster at midnight because nobody who chose it had ever run one. I have a long list of grudges against Kubernetes, and the unfair part is that almost none of them are really about Kubernetes. They're about people who chose it as a career move and made it someone else's problem.

So what are containers, really?

Stripped of the marketing: a container image is a bundle of files and metadata. A container runtime (runc, historically via Docker, today usually via containerd) takes that bundle and asks the Linux kernel to do three things: give this process its own view of the world (namespaces), cap what it can consume (cgroups), and start it inside a root filesystem that isn't the host's (pivot_root, or chroot on steroids). That's it. That's the magic. Everything else - Docker the CLI, Compose, Kubernetes, your Helm charts - is orchestration and convenience layered on top of a kernel feature that has existed for years and works just fine by itself.

You can run production workloads with nothing but systemd units and docker run commands. People do. It's not good, in the sense that nobody's built the self-healing, declarative, fleet-scale layer for you - but it's legitimate, in the sense that the container itself doesn't need Kubernetes to exist, any more than a process needs systemd to run.

Where Kubernetes actually earns its keep

Kubernetes is not the thing that makes containers work. It's the thing that makes hundreds of containers across dozens of machines work together - scheduling, rescheduling, healing, declaring intent, reconciling reality with a spec. That's an enormously valuable problem to solve... if you have that problem.

If you have three services and one box, Kubernetes is not the answer; it's a very elaborate way of making a simple system complicated. If you have two hundred services, six clusters, and a team of eight, Kubernetes is not optional; it's the only sane way to stay in control. The skill is knowing which situation you're in - and you can't know that if you never learned what the parts underneath actually do.

So if you're early in your career and the advice you're absorbing is "learn Kubernetes": fine, learn Kubernetes. But, learn Docker first. Actually run a container without an orchestrator. Actually break one. Actually read ps output and find your containerized process living smugly among the host's other processes. Learn what the kernel is doing for you, because the day something breaks - and it will - that's the layer where the answer lives.

The people who know that layer are the ones I hire. The people who pretend to are the ones I remember.