## TL;DR - The network inside a cluster is a black box by design: pods egress NATed behind the nodes, exactly like workloads NATed in a VPC. I don't care what that network is; I treat it as a VLAN with no route to its L3. - BGP and load balancing alone aren't features at the top of my list. eBPF was. That's why I picked [Cilium](https://cilium.io). - eBPF takes a little kernel-space understanding to click, but once it does it's obvious: programs that run safely inside the kernel, attached to the exact points where packets flow. No user-space detour per packet. - I liked it enough to argue about it. At an AWS Summit, after a talk from Liz Rice, I argued this tech should exist as native [Docker](https://www.docker.com) network drivers. I lost that argument, but I still think the idea was right. - Later in the series I want to test [Tetragon](https://github.com/cilium/tetragon), eBPF used for security observability. Same genius, pointed at a different problem. - Cilium is more work than the default CNI, and that's the point: the network fabric is the one layer where investing early pays dividends for the life of the cluster. I didn't pick it randomly. - How traffic gets *to* the cluster - that's the ingress story, next post. ## The black box you're allowed to not care about Here's the first thing I want to say about cluster networking, and it's the opposite of what most Cilium posts open with: the pod network is not interesting to me. I don't care which CIDR the pods live on, I don't route to it, I don't want my LAN even having a path to its L3. The cluster defines its own network; my services live in it; when they need to talk to the outside world they egress NATed behind the node IPs, same as any workload in a VPC egresses through a NAT gateway. That part of the design costs me nothing and asks nothing of me, and that's exactly what I want from it. If you came from [Proxmox](https://www.proxmox.com), this maps cleanly: the pod network is just another VLAN I don't extend into the LAN. The difference is that in a cluster, the "VLAN" is defined by a CNI instead of a switch, and the NAT story is handled by the data plane instead of a router on the edge of the segment. Which raises the only question that matters when picking the CNI: what is the data plane, actually? ## The primary reason: eBPF The honest answer for why Cilium and not one of the other excellent CNIs is that BGP and load balancing alone aren't features at the top of my list. eBPF was. Plenty of CNIs will do BGP peering and service load balancing; they were table stakes in my shopping, not the prize. What I actually wanted was the data plane. eBPF takes a little kernel-space understanding before it clicks. You have to know what the kernel is doing when a packet arrives - which functions it walks through, where the packet is copied, where a decision like "route this" or "drop this" gets made. If you've spent time in that world, whether from firewall work, from tcpdump archaeology, or from simply having wondered *where exactly* in the stack a rule is applied, the idea lands instantly. Once it clicks, it's obvious. eBPF is a way to run small, verified programs inside the kernel, attached to the exact points where packets flow. Instead of shunting every packet up to user space, through a proxy or a plugin, and back down, the decision happens where the packet already is. That's the whole trick. The overhead isn't "lower per benchmark," it's *architecturally absent* - the processing moves to the same place the packet is. ```mermaid flowchart LR subgraph USP["User-space data plane"] direction LR P1["Packet arrives
(kernel)"] --> UP["Detour up to
user space"] --> PROXY["Proxy / plugin
decides"] --> DOWN["Back down
into the kernel"] end subgraph EBPF["eBPF data plane"] direction LR P2["Packet arrives
(kernel)"] --> PROG["eBPF program decides
where the packet already is"] end ``` That's the genius of it, and it's why I was evangelizing about it before I had a cluster to run it on. ## The argument I lost (but not really) I had the pleasure of arguing with Liz Rice after her presentation at an AWS Summit. For those who don't know the name: she's one of the people who made eBPF *legible* to the rest of us, and her talks are the reason many people understand what it is. My argument was that this technology belonged in native Docker network drivers. I wanted container networking to have this data plane available without needing [Kubernetes](https://kubernetes.io) as the entry ticket. If the genius of eBPF is that it works at kernel level, in the packet path, then container networking is exactly the place it should shine - and at the time, the Docker-native story for it was thin. I lost that argument. eBPF went on to conquer the world through Kubernetes CNIs anyway, so in hindsight the market solved it from the other direction. But I don't think the idea was wrong: this is a kernel capability, not a Kubernetes feature, and it deserves to be usable wherever containers run. It just happened to find its mass adoption through k8s first. ## Where this shows up in practice I don't want to turn this into an eBPF tutorial - Liz's talks do that better than I ever will. But the practical consequences for the cluster are worth naming, because they're the reasons the fabric earns its place: - **Service traffic is handled in the kernel path.** Load balancing, NAT, and service translation happen where the packet is, not in a user-space proxy that every byte must traverse. - **Observability stops being an add-on.** Because the data plane is eBPF, the CNI can see - and tell you about - flows at a level a plugin-based CNI fundamentally cannot. - **Policies are enforced where packets flow.** NetworkPolicy isn't a suggestion relayed to some other component; it's a program at the kernel hook. And the road ahead: later in the series I want to test Tetragon, which points the same eBPF machinery at security - watching syscalls and process behavior from inside the kernel. It's the same genius applied to a different problem, and it's on the list. ## The investment that pays dividends The honest part, because this post has been suspiciously positive so far: Cilium is more work than the default. It's more concepts to learn, more knobs than a [flannel](https://github.com/flannel-io/flannel)-style overlay, more things that can be misconfigured in ways that only show up at 1am. If you're standing in front of your first cluster wondering whether to just take the simple CNI and move on: the simple CNI is a legitimate answer, and I won't pretend otherwise. So why did I pay the price? Because networking is the one layer where investing early pays back for the rest of the cluster's life. Storage classes can be migrated. Ingress controllers can be swapped. Applications can be redeployed. But the network fabric is the substrate everything else stands on - and the decisions made there are the ones you live with. Picking a data plane that treats the kernel as the platform, rather than a plugin-hosting afterthought, means every capability that arrives later - observability, security enforcement, load balancing modes, BGP - is an extension of what's already there, not a bolt-on fighting it. That's why I didn't pick it randomly. The BGP announcements that power the ingress story? Possible because the data plane was built for it. The Tetragon experiment waiting on the roadmap? Possible for the same reason. The credproxy work I teased last post leans on Cilium plumbing too. Each of those would have been a fight on a different fabric. Here, they're features being switched on. It's the same lesson as the [etcd](https://etcd.io) backup from post 9, generalized: the work you do before you need it is the work that's still working when you need it. Up-front cost, compounding returns. That's not an argument for gold-plating everything - most layers should stay boring. It's an argument for knowing which one layer is worth it. In a cluster, networking is that layer.