TL;DR

• My phone runs SailfishOS - Linux, neither Apple nor Android - and the platform gap is real: mainstream software assumes two ecosystems, and mine is neither.
• Tailscale ships no SailfishOS app, only binaries. So I packaged them into an RPM and built the app around them: driving tailscaled, reporting status, aiming for feature parity with the Android app plus the extras I actually wanted.
• The stock alarm app was the second offender - no snooze interval settings, and disabling an alarm is easier than snoozing it - so that became my second app.
• Both are built on the Forgejo runners from last post, with GLM doing the heavy lifting, and the loop now works end to end: I ask Hermes to plan a feature, a PR eventually appears, I test the built RPM.

The phone choice, and what it costs

Everyone's phone is one of two operating systems. Mine isn't. It runs SailfishOS, which means it runs Linux - actual GNU userland, a real terminal, packages - on the device I carry everywhere. I chose it for the same reasons I chose everything else in this series: I want the thing to be mine, to be inspectable, and to not phone home to a company whose business model depends on my attention. The privacy argument that runs through this whole estate does not stop at the pocket.

But I won't pretend the choice is free. The cost of leaving the duopoly is that every app you take for granted has to either exist for your platform or get built. Mainstream vendors build for iOS and Android and stop there. Your platform is a rounding error on their user metrics, and the binaries - when they bother to ship them at all - arrive with no UI, no integration, and no app.

SailfishOS does have one card to play here: Android app support. An Android compatibility layer - with Aurora Store and Obtainium sitting on top of it - gives surprisingly good coverage for everyday apps, and that closes a lot of the gap I'd otherwise feel daily. But not all apps work, and the ones that misbehave tend to be the ones I care most about. Which is why I've drawn a line: any future app that's purely an API client, I'll take on and code myself rather than fight the compatibility layer to make someone else's wrapper behave.

This post is about the two things I refused to live without, and what building them myself actually looked like.

Tailscale, the app that didn't exist

Tailscale is non-negotiable in this setup. The whole estate's remote access story rests on it - the mesh, the doors, the admin surfaces that are "nobody unreachable" - and a phone without the mesh is a phone cut off from all of it. SailfishOS gets no native app. What it does get is binaries: the client and the daemon, compiled for the right architecture, downloadable but uninstalled-shaped. No UI. Nothing that drives tailscaled. Nothing that tells you whether you're actually connected.

So the gap between "binaries exist" and "usable app" was mine to fill. I packaged the binaries into an RPM, and built an app around them that starts and drives the daemon, and reports on status - connected or not, what the mesh looks like from the pocket. The target is parity with the Android app where it matters, plus the extras the official apps don't give me and that I actually want. It's not finished, but it's real: the phone joins the mesh through software I built, packaged the way my platform expects, published on my Forgejo.

The alarm, or: the bar was on the floor

The second app came from a more mundane place, which is fitting, because mundane software failures are the ones that stick. The stock alarm app had two flaws that made it unusable for me. First, no settings for snooze intervals - the snooze behavior was fixed, take it or leave it, and my mornings are not the developer's mornings. Second, and worse: dismissing an alarm outright was easier than snoozing it. A half-asleep swipe in the wrong place and the alarm is gone for the day, with no way to tell the difference until the consequences arrive.

That combination - inflexible where I needed flexibility, careless where I needed care - meant the existing app was not going to improve. So I wrote my own. It does what the stock one wouldn't: configurable snooze intervals, and a deliberate friction gradient where stopping an alarm takes more intent than postponing it. It's a small app. It solves a small problem. But it's the clearest demonstration of the difference between using software and being able to change it: when the alarm misbehaves at 6am, I don't file a bug and hope. I own the repo it lives in.

The loop, which last post set up

Here's where the story connects back to the runners. Both of these apps are built on the Forgejo runners from last post - the KEDA-scaled pool that spins up when the queue has work. The push-to-build pipeline that compiles my cluster's tooling also compiles my phone's apps, on hardware I own, from repos I own. The recursion from the GitOps post just keeps deepening: the cluster now builds software for a device that isn't even part of the cluster.

And the development loop itself is where GLM comes in - first 5.2, now 5.3 - doing the heavy lifting on features, with me steering and reviewing. The workflow that's settled in is the one I find quietly remarkable: I ask Hermes to plan a new feature, and later it tells me a PR is ready. I test the built RPM, decide, merge or iterate. The phone in my pocket gets feature development from an agent on my own infrastructure, building from my own repos, through runners that existed because I got tired of maintaining build servers. None of that was the plan when I started this series. It's just what the estate turned out to be able to do once each layer was in place.

What the pocket proves

The thesis of this blog is that owning your stack is worth the time it costs, and the pocket is where that thesis gets its hardest test. A server is easy to make sovereign - it sits in your rack, runs what you tell it, and nobody expects it to run other people's apps. A phone is hostile territory: an entire industry built around two platforms, an app economy that assumes them, and a daily-use device where "it doesn't work" is not an acceptable outcome.

What I've learned living on the minority platform is that the gap is real but shallower than it looks. Tailscale's binaries were there; they needed packaging and a face. The alarm app needed writing, but writing a small app in 2026 is a conversation away from a first draft. The duopoly's real lock-in was never technical difficulty - it was the assumption that if your platform isn't served, you lose. The assumption is wrong for anyone with a cluster, runners, and the patience this series keeps talking about. The phone is the newest tenant of the estate, it doesn't even live in the house, and it's already on the payroll of the house that builds.

I should be honest about the last holdout, though: I still carry a GrapheneOS phone alongside the SailfishOS one. The rule is simple - I do as much as I can on the Linux phone first, and reach for the Graphene one only when I can't do otherwise. What's striking is the direction of that list: it keeps shrinking, and not because I'm porting faster than the industry. It's because more and more app developers have decided to be the authority on which devices are allowed to run their software, and the Graphene phone - which lives by the same defiance as the Linux one - keeps failing their checks alongside mine. Add Google's plans to disable sideloading, and the "escape hatch" phone is inheriting the same cage as everything else. That's precisely why the Linux phone earns its place: it gives me work, but the work buys back freedom that the mainstream platforms are actively selling off. The cage is closing from both sides. Mine is the one with a terminal.

TL;DR

• The last piece of the "code ownership at home" project wasn't storage or identity - it was compute for the builds. Forgejo runs on the cluster, but until now its runners didn't; a push could test itself somewhere that wasn't mine.
• Runners are the textbook scale-to-zero workload: two permanent runners burned resources around the clock on a constrained cluster, for a job that's idle most of its life. KEDA is the tool built for exactly that shape - watching the runner queue and spawning ephemeral pods on demand.
• The philosophical payoff: it makes the homelab aligned with the cloud's best idea - consume resources when you need them and no further - except the invoice is sent to nobody, because the metal is already mine.
• I didn't write the deployment from scratch. There's a helm chart on GitHub that got me 80% there, and getting the last 20% right is the part this post is about - which is the whole beauty of Open Source: somebody already walked the path, and the walking is now cheaper.
• The recursion from post 17 is now complete: the cluster deploys from Forgejo, Forgejo tests itself, and the builds run on the same metal as the code they build.

What the runners were before

Post 22 counted Forgejo's runners among the tenants - "with runners now" - but counted them the way you count a thing that exists, not the way you count a thing that's done. The honest version: runners had been registered and had executed the occasional job, but the provisioning story wasn't one I could describe without embarrassment. It was a runner or two, standing, permanent, boring, and wrong in the way permanent standing things in a cluster are always eventually wrong.

The wrongness was the shape, not the existence. I had two permanent runners, standing, allocated, always warm - two sets of resources consumed around the clock on what is, in the grand scheme of things, a constrained cluster. A CI runner is idle almost all of its life. Jobs arrive in bursts: you push, and for a few minutes you want compute, and then you don't again for hours. Standing runners pay the full price of existence for that pattern - allocated resources, warm pods, something to patch - for a fraction of the actual work. On a cluster where every node counts, two of them on permanent stand-by is the opposite of the frugality the whole estate is built on. Runners standing idle all day are paying rent without living there.

The scale-to-zero pattern was the obvious answer, and Kubernetes has a tool whose entire job is that pattern.

KEDA: the shape of the solution

KEDA - Kubernetes Event-driven Autoscaling - watches a queue, a metric, a stream, anything that can be measured, and scales a workload between zero and whatever the load asks for. When there's nothing to do, the workload doesn't exist. When work arrives, it appears, does the job, and dissolves again.

For a Forgejo runner pool, this is almost comically perfect. The runner is registered once, in the usual way, and waits - not as a pod, but as nothing at all. A push lands, Forgejo's runner queue gains a job, KEDA notices, and a runner pod spins up, picks up the job, runs the build or the tests, and either parks back at zero or waits briefly for the next one. The cluster's contribution to CI becomes: when you need a builder, here's a builder; when you don't, there isn't one.

What this buys is the same trade the whole estate runs on: cost where it's earned, absence where it isn't. The idle runners of the old shape were resources pretending to be a service. The KEDA shape is a service that exists only when used - which is also, I'd argue, the more honest Kubernetes idiom. The platform is for elasticity; using it for CI is the pattern working as intended.

There's a philosophical point under the mechanics, and it's worth spelling out: this makes a homelab philosophically aligned with the cloud's best idea. The thing the cloud actually got right was never the managed services or the dashboards - it was the economic model: consume resources when you need them and no further. That's an idea worth stealing on its merits, and Kubernetes plus KEDA lets me steal it entirely. The elasticity without the landlord, the scale-to-zero without the invoice. My cluster now bills the same way the cloud does - for work done, not for existence - except the invoice is sent to nobody, because I paid for the metal once and it's mine.

One honest wrinkle worth naming: scale-to-zero means the first job of a burst waits for a pod to schedule. It's seconds, not minutes, and in a homelab where the queue is mine and the patience is mine, that's a price I'm happy to pay. But it's a real trade - a cold start is a cold start - and anyone copying this setup should make it with eyes open.

Standing on shoulders

I didn't hand-write the deployment. There's a helm chart on GitHub - a community chart for Forgejo runners with KEDA scaling - and I started from it, the way you're supposed to.

This is the part of the story I want to defend rather than apologize for, because it highlights everything I love about Open Source. Somebody had the problem before me. They solved it, parameterized the parts that vary, and published it. I read their chart, understood their assumptions, kept the 80% that was right, and bent the 20% that wasn't toward my estate - my runner registration, my resource envelopes, the hooks into my storage and identity. The chart gave me the walk; my cluster gave the walk its own route.

The alternative, writing it from zero, buys nothing except the ability to say I did - and loses the review that comes free with a published chart, which is that dozens of people before me already hit its bugs. I'd rather inherit a solved problem and fix my deviation than invent a problem that was solved last year. Standing on shoulders isn't laziness; it's how the whole ecosystem compounds, and a homelab that pretends otherwise is just rebuilding the industry slower and worse.

And the deviations themselves are blog-sized: what I changed, why, and what the chart's defaults assumed about a cluster mine doesn't match. The mechanics of provisioning - the ScaledObject watching the queue, the ephemeral runner pods, the registration flow - are documented well enough elsewhere. What's under-documented is the judgment: which defaults to keep, which to challenge, and how to know the difference. That's the part this post is for.

The loop, closed

Here's where the recursion lands. Post 17 ended with the cluster deploying from Forgejo, and Forgejo running on the cluster - the loop I found genuinely satisfying and slightly circular. The runners close the last gap in that circle: now the CI that decides whether a commit is worth deploying runs on the same metal as the deployment it gates.

The full shape: I push to my forge. My forge's runner queue fills. A pod scales up on my cluster, builds and tests the change. If it passes, the manifests change in the repo, ArgoCD picks them up, and the cluster updates itself. Push, build, deploy - every hop on hardware I own, under policies I set, observed by the monitoring I built. Not one packet of that journey required a third party's goodwill, a paywall, or a quota.

flowchart LR
    PUSH["git push"] --> FJ["Forgejo<br/>runner queue"]
    FJ --> KEDA["KEDA<br/>watches the queue"]
    KEDA -->|"0 to 1:<br/>spawn on demand"| RUN["Ephemeral<br/>runner pod"]
    RUN -->|"build + test"| RES["Job result"]
    RES -->|"pass -<br/>manifests change"| ARGO["ArgoCD"]
    ARGO --> CLU["Cluster<br/>updates itself"]
    CLU -.->|"same metal,<br/>end to end"| PUSH

That's what "code ownership at home" was always pointing at: hosting the whole lifecycle - the forge, the builds, the deployments, the tests - instead of storing my software's home address. It's where my software is born, proven, and shipped.

What this opens

A fully closed loop is also a loaded spring. The next thing this estate needs is something of mine to build - the first self-devved app, the model runner that's been in progress for a while now. With runners provisioned and scale-to-zero proven, the last excuse is gone: the infrastructure to build and deploy my own code is standing, waiting, elastic. The ApplicationSets dev/live story from post 17 has its stage set.

That's the runners done. The house builds houses now.

TL;DR

• DNS has been a gap in this arc since the early posts: every networking post assumed names resolve and none of them explained what actually resolves them. This closes it.
• The resolver for the whole homelab is Blocky, wired as the upstream for the network's DHCP-assigned clients: DoT to the real upstream resolvers, light blacklists for the things worth blocking, and manual entries for the services that should resolve locally at the network level.
• Blocky over pihole, in my view, comes down to one decisive point: a much more comprehensive configuration surface for my needs, and built-in TLS for upstream queries - no additional unbound to stand up and babysit just to have DoT.
• The checklist format is deliberate: this is the small, unglamorous post that verifies the promises the other posts quietly assumed were true.

The gap this post closes

Somewhere around the mail post, DNS started appearing in the margins. Deliverability was "half DNS." The mail server needed a hostname that resolved the way the world expects. The border posts gave the cluster routable addresses and assumed the names would follow. And every reader who's run anything self-hosted knows the truth: DNS is the thing that's fine right up until it isn't, and when it isn't, everything isn't.

So this is the checklist post. Not the most glamorous entry in the series, but the ones before it quietly assumed a working name-resolution layer, and I'd rather verify the assumption than keep it quiet. The checklist format is deliberate, too: each item below is a promise one of the earlier posts made, and this post is where it gets checked off.

The resolver: Blocky

The homelab's resolver is Blocky, and it serves two roles at once, which is exactly why it earned a place on the estate under the filters from post 22: it carries a genuine privacy purpose - it's the point where the whole network's DNS queries get sanitized, encrypted, and filtered before they leave the house - and its operational rent is negligible, because Blocky is very light. A small Go binary that does one job and doesn't ask to be babysat.

Its job: be the single DNS answer for the homelab. DHCP hands out Blocky's address to every device on the network, and from that point on every query from every laptop, phone, and appliance flows through one small chokepoint I control. Which is the whole point: DNS is one of the last places a modern network leaks. Every hostname a device looks up is a small confession about what it's doing, and Blocky is where those confessions get read before they go anywhere.

DoT, and the unbound I didn't have to deploy

Upstream, Blocky talks DNS-over-TLS to real public resolvers. Plaintext DNS on port 53 to the ISP resolver was never an acceptable end of the pipeline - that's the leak the whole exercise exists to close - and DoT closes it without ceremony.

Here's the part that matters in a comparison-shopping sense: I don't have to have an unbound in place somewhere just to have TLS for DNS queries. The classical pihole setup for encrypted upstream is pihole plus unbound - one box to do the filtering, another to do the TLS - which is two services, two failure points, and two things to patch for one job. Blocky does both in one binary: filtering and encrypted upstream in the same config file. For my money that's also where its configuration set is simply more comprehensive than pihole's for what I need: upstreams as first-class config, per-upstream TLS, grouping and failover between them, and the blocking lists as a declarative layer rather than a UI toggle. IMHO, but a fairly settled one at this point.

Blacklists stay light, deliberately. The goal is the network's obvious annoyances - ads, telemetry, the worst tracking networks - not a maximal blocklist that turns DNS into a helpdesk ticket generator for the household. When something legitimate gets caught, the fix is a whitelist entry and ten seconds. The blocking is a convenience that pays rent in privacy; it isn't the point of the deployment.

Manual entries: the network-level pin

The third job, and the quiet reason a network-level resolver earns its keep in a homelab with a cluster: manual entries. Services that should resolve to local addresses, pinned at the network level so every device gets the answer without knowing anything about it.

This is the payoff of the whole design. When the mail server, the forge, or any cluster-fronted service should be reachable from the LAN by name, the answer comes from Blocky's config, not from each device's hosts file or a browser extension or an /etc/hosts gospel that only my machine has. It also means those local answers are version-controlled in the same repo as everything else on the estate - a DNS record is a one-line config change, committed, with the same GitOps discipline as the cluster's own state. Move a service, change a line, and the whole network follows.

And Blocky runs on the cluster itself, which is where BGP earns its keep. Cilium advertises the LoadBalancer address into the network, so the DNS endpoint is a single cluster IP that every subnet and VLAN already knows how to reach - no per-subnet routes to maintain, no "which firewall rule forwards port 53 where" archaeology. The router learns the route via BGP, and Cilium load-balances the queries across the Blocky replicas behind it. Out in the network, DHCP configurations vary - some clients get Blocky's address handed to them directly, others point at the router which forwards upstream to Blocky - but the chain always lands in the same place, and Blocky is in it either way. DNS gets the same treatment as everything else on this estate: a service, horizontally spread, with the network doing the routing homework instead of me.

flowchart LR
    DEV["Every device on the network<br/>(via DHCP)"] --> BLK["Blocky<br/>(cluster IP via BGP,<br/>load-balanced by Cilium)"]
    BLK -->|"manual entries<br/>(version-controlled)"| LOC["Local answers<br/>for estate services"]
    BLK -->|"light blacklists"| DROP["Ads, telemetry, trackers:<br/>blocked"]
    BLK -->|"DoT -<br/>encrypted upstream"| UP["Public resolvers"]
    WORLD["The outside world"] --> R53["Public zones: Route53<br/>(delegated, DNS-01 for certs)"]

One precision, because the next paragraph depends on it: the public zones live in Route53. The private, local side of DNS is self-hosted - Blocky for the LAN, internal records where the estate needs them - but anything the outside world resolves goes through Route53, and that split matters more than it sounds. It means cert-manager needs automation-capable credentials for Route53: DNS-01 challenges against the public zone are how certificates for internal-sounding hostnames get issued at all, and a credential that can do that unattended is a privilege worth being deliberate about. That's its own story; more on that later.

The checklist

Since this is the checklist post, the checklist:

1. Upstream privacy - every client's queries leave the house encrypted (DoT), through resolvers that aren't the ISP's default. ✓
2. Filtering - light blocklists at the network level, applied to every device whether it knows or cares. ✓
3. Local pinning - estate services resolve to local addresses for the whole LAN, from version-controlled config. ✓
4. One component - filtering + TLS upstream + local entries in a single light binary, no unbound companion. ✓
5. The cluster's own DNS - a separate question (CoreDNS and the pod network), which stays inside the cluster and was never this post's problem. ✓ by deliberate scoping.
6. Public zones in Route53 - self-hosted where self-hosting makes sense (the LAN side), delegated where the world needs answers; cert-manager's DNS-01 automation leans on that split. ✓ (credentials story to come).

That's the checklist closed. DNS is boring, which is the highest compliment available for infrastructure - it means it's working, and this post exists so the boring is on the record rather than in my head.

TL;DR

• The scaffolding is done, so here is the honest inventory: the apps that moved in are few, and the list is short on purpose.
• Two reasons decide what gets hosted: it serves a privacy purpose (the mail server, the git server), or it pays for its place. Everything else waits, because self-hosting everything is possible and expensive in the only currencies that matter: time and effort.
• What did make it in: Forgejo with runners, bringing repo ownership home, and Open WebUI plus Hermes keeping AI conversations and files at home - local Ollama embeddings, ZDR providers only for the rest.
• The first self-devved app is in progress: something to run models on the cluster, and the natural trigger for the ApplicationSets dev/live model.
• The plan on the horizon: pointing Hermes, my coding agent, at Forgejo to coordinate long-lived coding tasks - the estate becoming a place where software gets written, not just run.

The inventory

Time to open the doors and count the residents. I've been putting this post off, and the reason for the delay is the honest part: the list is short. After twenty-one posts of scaffolding - storage classes, identity, backups, GitOps, a network border drawn three ways - you'd expect a mansion full of tenants. The reality is a house with a few very deliberate residents.

And I think that's the more useful story to tell. The homelab internet is full of self-hosting victory laps: forty apps on a dashboard, mostly opened twice and forgotten. The count feels like the achievement. It isn't. The achievement is that each thing that lives here earns its place, because every resident is one more thing to patch, back up, monitor, and eventually explain to myself at 2am.

Why so few: two filters

Two filters decide what gets hosted here, and most candidates fail both.

The first filter is privacy purpose. The estate exists, at its core, so that the things that carry my life - mail, code, identity, files - live on hardware I control, under policies I set, encrypted the way I want, with retention I decide. Stalwart got a post because mail is the crown jewel of that argument. Forgejo is in that same category: my repos, my metadata, my history. These aren't apps I prefer to self-host; they're apps that make the whole exercise coherent. If they don't run here, the estate is a hobby. If they do, it's a position.

The second filter is the one nobody warns you about: the cost of getting a thing to work fully self-hosted is enormous and it's not negligeable, so be it. Every service that makes it in arrives with its own private tax - the SSO integration, the backup path, the monitoring, the storage class choice, the upgrade procedure, the question of what happens when it's down. The engineering is the fun part. The ops is the part that multiplies. Anyone setting out to self-host everything: brace yourself. It yields a lot, but the bill is paid in the one resource a homelab never mints more of, which is time. So the bar for entry is not "is this nice to have." It's "does this justify its operational rent."

The result of the two filters, applied honestly: not much, if anything at all, beyond what the cluster needs for itself. And I've stopped feeling bad about that.

What made it in

Three residents, then, and they're instructive precisely because they're few.

Forgejo, with runners now. Post 17 told the story of the GitOps loop and the small satisfaction of the cluster deploying from a forge that runs on the cluster. The runners are the second half of that sentence: CI executing at home too, so the loop - push, build, deploy - runs entirely on my metal, end to end. This is the "bring repo ownership home" project in its final form. Every repository that migrates off GitHub takes one more piece of my development history out of a platform that gates basic features like SSO behind a paywall, and the runners mean it's not just hosting the code anymore, it's building it. The recursion from post 17 has a sequel: the cluster deploys from Forgejo, Forgejo tests itself, and the whole development surface lives at home.

Open WebUI, which earns its place by a different route: it's half of the answer to "how do AI touch this estate without touching my privacy." Together with Hermes, my coding agent, they're how I keep conversations and files at home. Open WebUI fronts a local Ollama for embeddings - that part is fully local, full stop. The heavier conversation models, for now, come through OpenRouter, with a discipline attached: proper zero-data-retention providers only, GLM as the workhorse, and privacy guardrails around what flows out at all. The shape matters more than the specifics: the sensitive layer (embeddings, storage, the interface) is mine; what crosses the border is model inference through providers contractually forbidden from keeping it, and even that only within rails I set. It is, in a strict sense, a want rather than a need - but it's a want held to the same standard as a need. That's why it matters in the inventory: it's proof the filters are filters and not a cage. When something genuinely earns its place as a want, it gets in. Most wants don't, and that's fine.

And the third resident isn't one yet - it's the plan that's shaping the next chapter. Forgejo's API surface, once the runners are stable, is where Hermes comes in: my coding agent, pointed at my own forge, coordinating long-lived coding tasks on my own infrastructure. Code ownership at home, then compute-for-code at home, then the agent that lives in the same house. The estate stops being only a place where software is run and becomes a place where software gets written.

The ApplicationSet-shaped hole

One gap to name while I'm being honest about inventories: I haven't devved a serious app of my own on this cluster yet - though the first one is already in progress, and it's in character for the estate: something to run models on the cluster itself. Post 17 sketched the model - ApplicationSets generating a dev and a live Application from one template, commits flowing to dev, promotion to live when dev has earned it - and it's still a sketch, because nothing of mine has needed it yet. Forgejo, runners, and Open WebUI all landed the ordinary way: manifest in the repo, policy deciding how it syncs.

That first app is exactly the sort of thing the model will get its post for: an inference workload wants a dev face to break safely and a live face to serve from, and with Hermes-on-Forgejo behind it, the loop closes - an agent that writes code all day eventually writes something worth deploying twice.

What this closes

The scaffolding arc is now fully cashed. Floor, door, keys, storage, identity, deployment model, border - and now the census, short as it is. The estate has residents chosen by filter rather than by accumulation, and the two filters - privacy purpose, operational rent - are the real thesis of every scaffolding post that came before: infrastructure is not the obstacle to workloads, it's what makes each workload affordable to keep.

TL;DR

• This is the Tailscale story post 8 deliberately deferred: node-to-node mesh was that post's job; this one is about the operator, and how other things talk to the cluster.
• The frame: not "VPN to get back home," but a powerful networking control plane used properly. Two kinds of doors live side by side: the public door (Authentik-fronted services, whose OIDC endpoints must be publicly reachable) and the private door (admin UIs, backends, dashboards that should never be public at all).
• The operator runs Tailscale as a Kubernetes workload and publishes services into the tailnet selectively - proxy groups choose what's exposed, which is least privilege applied to the network layer.
• The result: every service gets the access model it deserves. Public services get identity-checked front doors. Admin surfaces get existence hidden behind the mesh. No port bounties, no "is your Grafana public?" anxiety.

The deferred story, briefly

Post 8 drew a line on purpose: Talos ships the Tailscale client for node-to-node networking - the mesh that keeps the cluster's plumbing working across networks - and that was all that post needed. It pointed at the operator, the Kubernetes-native side, and said "later." This is the later.

Two kinds of doors

Here's the distinction that makes everything click, and it took me embarrassingly long to state it cleanly. There are two kinds of doors into an estate like mine, and they have opposite requirements.

The first kind: services that must be publicly reachable. Anything doing OAuth or OIDC federation needs its endpoints to be public - that's how the protocol works. If Authentik is the identity provider for an external service, that service needs to reach the OIDC endpoint without a VPN, without a tailnet membership, without any pre-existing trust. Public is not a design flaw there; it's the contract. So those services get the full treatment: CloudFront edge, firewall admission, Traefik routing, Authentik checking identity at the door. The public door is elaborate because the protocols demand it.

The second kind: everything else. Admin UIs. Backends. Database management surfaces. The Grafana nobody else should be logging into. The git server's admin panel. The mail server's interface. For these, the question "who can reach it?" gets a different first answer: nobody who isn't already mine. Not "nobody unauthenticated" - nobody unreachable, unless they're on the mesh. The URL doesn't resolve from the internet. The port isn't listening. The surface doesn't exist, as far as the public internet is concerned. And that's not instead of authentication - it's in front of it. The services behind the tailnet still speak OIDC; Authentik still stands at the door and checks identity exactly as it does on the public side. Tailscale answers "are you even allowed to attempt this," Authentik answers "who are you and what may you touch." Reachability is the outer fence; identity is the door. The two compose, and the composition is the point: private services get both fences, public ones get the identity fence plus the full public chain.

The traditional way to get the second kind was a VPN: a concentrator, listening on a public port, that you dial into. Which works, and immediately recreates the problem: you've just built a public endpoint whose compromise is catastrophic. The door you hid just got a very high-value public copy.

The control plane you get for free

This is where Tailscale stops being "a VPN" and starts being what it actually is: a coordination plane that makes WireGuard behave like infrastructure. The mesh doesn't need listening ports; every node dials out, and the coordination layer brokers the handshake. Key rotation, ACL policy, node identity - all handled by a control plane that was designed by people who thought about the failure modes.

The Kubernetes operator is where it gets interesting for this estate. Tailscale runs as a workload, not as a daemon I install on every node and remember to maintain. When I want a service published to the tailnet, I annotate it, and the operator stands up a proxy pod that joins the mesh and fronts the service. Which service gets published, to which identities, with what access - that's the proxy group story, and it's least privilege applied at the network layer: the network equivalent of what Authentik does for identity. Not every service is in the tailnet. The ones that are, are there on purpose, addressed on purpose, and reachable by identities I explicitly allowed.

The beauty of this arrangement is what it does to my public attack surface. My internet-facing footprint shrinks to the services that genuinely need to be public. Everything administrative goes dark. There is no Grafana login page for a botnet to find, no admin panel leaking its version number, no exposure surface at all - because none of it listens on the internet. The estate keeps its public doors, polished and identity-checked; and everything else simply isn't there.

The posture

So the border now has two faces. The public face: CloudFront at the edge, firewall doing admission, BGP announcing addresses, Traefik routing, Authentik authenticating - a chain of responsibilities, each doing its one job, exactly as the last two posts described. The private face: a mesh that follows the estate out of the house, that needs no inbound ports, that publishes only what I explicitly published, and whose access policy lives in the same declarative world as everything else.

The two faces don't bleed into each other. A service is public or it's tailnet-only; if it's both, each path gets the treatment its exposure deserves. And the mental model scales: when the second cluster from the site-to-site daydream materializes, it joins the same mesh the same way - another set of doors, same policies, same control plane.

flowchart LR
    subgraph PUB["The public door"]
        direction LR
        NET["Internet"] --> CF["CloudFront"] --> FW["Firewall"] --> TR["Traefik"] --> A1["Authentik<br/>identity"] --> SVC["Public services<br/>(OIDC endpoints)"]
    end
    subgraph PRIV["The private door"]
        direction LR
        DEV["My devices"] -->|"dials out -<br/>no listening ports"| TS["Tailscale mesh"] --> OP["Operator<br/>proxy pod"] --> A2["Authentik<br/>identity"] --> ADM["Admin UIs,<br/>backends, dashboards"]
    end

And here's the payoff nobody advertises: because Authentik sits at the center of the public door, moving a service between the two faces is nearly trivial. The service's provider in Authentik - the registration that describes who the app is and where its endpoints live - carries the redirect URLs and the OIDC/SAML endpoints. Change the service's exposure in the network layer (publish it to the tailnet, unpublish it from the edge), update the redirect URL in the Authentik UI to match, and the service has changed continents. Public to private, or private to public, without touching the application, without re-registering anything at the provider, without a migration project. The centralization I chose for identity in post 15 keeps paying rent: one place holds the truth about where each app's front door is, so relocating the door is a form edit.

That's the networking chapter closed: fabric, front door, private door. Three posts, one border, no leaks.

TL;DR

• Ingress is a chain of responsibilities, not one box: a BGP-capable router so I never worry about which node to point at, a firewall doing what firewalls do best (outside the cluster, on purpose), and CloudFront carrying everything HTTPS from the internet's edge.
• BGP-capable router so I never worry about which node to point at; Cilium announces service IPs, the announcement is the contract.
• Firewall stays outside Kubernetes: admission control is its job, in the place it's built for.
• CloudFront carries everything HTTPS from the internet's edge; its documented IP ranges mean "only CloudFront may talk to the firewall's ingress port" is a generatable, versionable rule.
• Traefik is the ingress controller inside - chosen on a track record, not a benchmark - with cert-manager + ACME for certificates.

The question every ingress answer gets wrong

Every ingress question eventually becomes "which box do I point at?" Load balancer IPs, node ports, ingress controllers with their own published addresses - and every one of those answers hides a question nobody asked: who keeps that address true when the cluster changes?

The usual answer is "you do, manually," or "a cloud controller does, somewhere." I wanted neither. My answer is that the question should dissolve entirely: nothing should have to know which node to point at, because the network should already know. That's what BGP buys, and it's why the announcement from last post belongs to this story rather than the fabric one.

The announcement, remembered

To recap the shape: Cilium announces the cluster's service addresses to my router over BGP. The router learns them the way it learns any route, and the cluster becomes just another prefix-holder on my network. From that moment, "which node?" stops being a question anyone answers - the routing table answers it, continuously, and updates itself when nodes come and go. A node dies, the routes adjust, traffic follows the survivors. Nobody re-points anything.

This is the part I keep having to defend, because it sounds exotic: a BGP-capable router in a homelab. It shouldn't sound exotic. It's one more dynamic-routing neighbor, and my firewall handles it for breakfast. What it removes is an entire category of operational debt - the hardcoded edge address, the DNS record that lags reality, the "wait, which node was that again?" archaeology. The announcement is the contract, and contracts don't need maintenance, only honoring.

The firewall's job, done outside

Here's a design decision I feel strongly about: the firewall stays outside Kubernetes. Not "connected to it" - outside it, as in not running on it, not managed by it, not a workload in it.

The reason is the same principle that's run through this whole arc: every component should do the one job it's built for, in the place it's built for. A firewall's job is admission control at the border - deciding what traffic is allowed to enter, from where, to what. That job belongs to the component whose whole existence is that border, the one that's been doing it since before the cluster existed and will keep doing it after the cluster is rebuilt from a git repo and a PXE stick. If I put the firewall inside the cluster, its fate is coupled to the thing it's supposed to be protecting - and a border guard who dies with the fortress isn't a border guard, it's a decorator.

So: packets arrive at the firewall (which learned where the cluster lives via BGP, above), and the firewall does what it does best - the rules. What's allowed in, from whom, to which port. Not the cluster's job. Not the ingress controller's job. The firewall's job, done in the place the firewall is.

Traefik, and the certificates

The last link in the chain is the ingress controller, and here I'll confess to having a favourite: Traefik. This is the one component I chose on familiarity first and ideology second - I've used it extensively before, it works really well, and that track record means more to me than a benchmark. NGINX is a fine web server wearing an ingress costume; Traefik was designed as an edge router from day one, and it shows in how its configuration model maps to what an ingress actually does: routes, entry points, middlewares, services. I wanted an ingress controller I'd stop thinking about, and Traefik is the one I already had stopped thinking about.

Certificates are where it gets interesting, because the ACME challenge for DNS-01 validation needs Route53 credentials, and this estate doesn't do long-lived static keys - post 12 established that hill. The solution is credproxy: a home-backed IRSA-simulation, my own service that mints credentials the way ECS and EKS do, so cert-manager can ask Route53 to answer challenges without anyone holding a permanent key. That mechanism is a chapter of its own, and a fun one - Cilium plays a big role in making it work properly - so I'll leave it teased here rather than crammed into a paragraph.

CloudFront, the front door's front door

And now the piece that surprises people: all things HTTPS come in via CloudFront, which I know sounds like the most cloud-native thing in what is supposedly a self-hosted story. Stay with me.

CloudFront does three things for this setup, and none of them require me to hand anything over:

1. Edge security. TLS termination, DDoS absorption, the world's largest chokepoint standing between the internet and my house. CloudFront's IP ranges are public and well documented - which means my firewall rule set can be precise: the only source allowed to talk to the firewall on the ingress port is CloudFront. Everything else is dropped at the border, unlogged, uncounted. The internet never talks to my house directly; it talks to Amazon, and Amazon talks to me.
2. IaC integration. CloudFront fits into the same IaC automation as everything else in the estate - distributions, behaviors, certificates, all of it in code, reviewed, applied, versioned like every other change I make. The edge is declarative too. My IaC story is a two-tool settlement, and it's deliberate: CloudFormation where I want infrastructure deployed and then forgotten - it's a service, the stack is maintained by AWS, and I never think about state again - and Pulumi where I want real code, because infrastructure written in an actual language gets the same treatment as any other code: functions, types, review, refactor. Yes, Pulumi has state management too - the difference is it's a detail of the tool, not the language I'm forced to think in. What I actually can't stand is HCL: a bespoke DSL pretending to be configuration while failing at both jobs. HashiCorp, once again, sits this one out. OpenTofu is a fine fork, but the moment anyone tells me HCL is a solved problem I remember my last skirmish with a for_each expression, and my stomach settles the debate.
3. One documented source. Because the IP ranges are published, "only CloudFront may connect" isn't a hope, it's a rule I can generate, version, and diff. When Amazon publishes new ranges, my automation picks them up. The front door's guest list writes itself.

The result is a chain, and each link owns exactly one thing: CloudFront owns the edge - TLS, DDoS, and the guest list. The firewall owns admission - what comes in, from whom, to where. BGP owns the addressing - where the cluster's services actually live, updated in real time. Cilium owns the fabric - getting a packet from the edge of the cluster to the right pod. Traefik owns the routing - path, host, service, inside the cluster where HTTP actually becomes a request to a workload.

Every component does the one job it's built for. No box knows more than it needs to. And when a piece fails, the blast radius is one responsibility, not the whole door.

flowchart LR
    NET["The internet"] --> CF["CloudFront<br/>TLS 1 - DDoS absorption - guest list"]
    CF -->|"only CloudFront may talk<br/>to the firewall's ingress port"| FW["Firewall<br/>admission"]
    FW --> ROUT["Router<br/>routes learned via BGP"]
    ROUT --> CIL["Cilium<br/>the fabric"]
    CIL --> TR["Traefik<br/>TLS 2 - routing"]
    TR --> POD["Workload"]

One detail worth pausing on: the certificate story reaches backwards through this whole chain. Traefik terminates TLS again inside the cluster (cert-manager + ACME via credproxy keeps those certs valid), while CloudFront terminates it at the edge - so the connection is encrypted end to end, but it's terminated twice, once at Amazon's border and once at mine, each side with its own certificates and its own renewal story. That's not accidental duplication; it's the chain-of-responsibility model applied to TLS itself.

TL;DR

• The network inside a cluster is a black box by design: pods egress NATed behind the nodes, exactly like workloads NATed in a VPC. I don't care what that network is; I treat it as a VLAN with no route to its L3.
• BGP and load balancing alone aren't features at the top of my list. eBPF was. That's why I picked Cilium.
• eBPF takes a little kernel-space understanding to click, but once it does it's obvious: programs that run safely inside the kernel, attached to the exact points where packets flow. No user-space detour per packet.
• I liked it enough to argue about it. At an AWS Summit, after a talk from Liz Rice, I argued this tech should exist as native Docker network drivers. I lost that argument, but I still think the idea was right.
• Later in the series I want to test Tetragon, eBPF used for security observability. Same genius, pointed at a different problem.
• Cilium is more work than the default CNI, and that's the point: the network fabric is the one layer where investing early pays dividends for the life of the cluster. I didn't pick it randomly.
• How traffic gets to the cluster - that's the ingress story, next post.

The black box you're allowed to not care about

Here's the first thing I want to say about cluster networking, and it's the opposite of what most Cilium posts open with: the pod network is not interesting to me. I don't care which CIDR the pods live on, I don't route to it, I don't want my LAN even having a path to its L3. The cluster defines its own network; my services live in it; when they need to talk to the outside world they egress NATed behind the node IPs, same as any workload in a VPC egresses through a NAT gateway. That part of the design costs me nothing and asks nothing of me, and that's exactly what I want from it.

If you came from Proxmox, this maps cleanly: the pod network is just another VLAN I don't extend into the LAN. The difference is that in a cluster, the "VLAN" is defined by a CNI instead of a switch, and the NAT story is handled by the data plane instead of a router on the edge of the segment.

Which raises the only question that matters when picking the CNI: what is the data plane, actually?

The primary reason: eBPF

The honest answer for why Cilium and not one of the other excellent CNIs is that BGP and load balancing alone aren't features at the top of my list. eBPF was. Plenty of CNIs will do BGP peering and service load balancing; they were table stakes in my shopping, not the prize. What I actually wanted was the data plane.

eBPF takes a little kernel-space understanding before it clicks. You have to know what the kernel is doing when a packet arrives - which functions it walks through, where the packet is copied, where a decision like "route this" or "drop this" gets made. If you've spent time in that world, whether from firewall work, from tcpdump archaeology, or from simply having wondered where exactly in the stack a rule is applied, the idea lands instantly.

Once it clicks, it's obvious. eBPF is a way to run small, verified programs inside the kernel, attached to the exact points where packets flow. Instead of shunting every packet up to user space, through a proxy or a plugin, and back down, the decision happens where the packet already is. That's the whole trick. The overhead isn't "lower per benchmark," it's architecturally absent - the processing moves to the same place the packet is.

flowchart LR
    subgraph USP["User-space data plane"]
        direction LR
        P1["Packet arrives<br/>(kernel)"] --> UP["Detour up to<br/>user space"] --> PROXY["Proxy / plugin<br/>decides"] --> DOWN["Back down<br/>into the kernel"]
    end
    subgraph EBPF["eBPF data plane"]
        direction LR
        P2["Packet arrives<br/>(kernel)"] --> PROG["eBPF program decides<br/>where the packet already is"]
    end

That's the genius of it, and it's why I was evangelizing about it before I had a cluster to run it on.

The argument I lost (but not really)

I had the pleasure of arguing with Liz Rice after her presentation at an AWS Summit. For those who don't know the name: she's one of the people who made eBPF legible to the rest of us, and her talks are the reason many people understand what it is.

My argument was that this technology belonged in native Docker network drivers. I wanted container networking to have this data plane available without needing Kubernetes as the entry ticket. If the genius of eBPF is that it works at kernel level, in the packet path, then container networking is exactly the place it should shine - and at the time, the Docker-native story for it was thin.

I lost that argument. eBPF went on to conquer the world through Kubernetes CNIs anyway, so in hindsight the market solved it from the other direction. But I don't think the idea was wrong: this is a kernel capability, not a Kubernetes feature, and it deserves to be usable wherever containers run. It just happened to find its mass adoption through k8s first.

Where this shows up in practice

I don't want to turn this into an eBPF tutorial - Liz's talks do that better than I ever will. But the practical consequences for the cluster are worth naming, because they're the reasons the fabric earns its place:

• Service traffic is handled in the kernel path. Load balancing, NAT, and service translation happen where the packet is, not in a user-space proxy that every byte must traverse.
• Observability stops being an add-on. Because the data plane is eBPF, the CNI can see - and tell you about - flows at a level a plugin-based CNI fundamentally cannot.
• Policies are enforced where packets flow. NetworkPolicy isn't a suggestion relayed to some other component; it's a program at the kernel hook.

And the road ahead: later in the series I want to test Tetragon, which points the same eBPF machinery at security - watching syscalls and process behavior from inside the kernel. It's the same genius applied to a different problem, and it's on the list.

The investment that pays dividends

The honest part, because this post has been suspiciously positive so far: Cilium is more work than the default. It's more concepts to learn, more knobs than a flannel-style overlay, more things that can be misconfigured in ways that only show up at 1am. If you're standing in front of your first cluster wondering whether to just take the simple CNI and move on: the simple CNI is a legitimate answer, and I won't pretend otherwise.

So why did I pay the price? Because networking is the one layer where investing early pays back for the rest of the cluster's life. Storage classes can be migrated. Ingress controllers can be swapped. Applications can be redeployed. But the network fabric is the substrate everything else stands on - and the decisions made there are the ones you live with. Picking a data plane that treats the kernel as the platform, rather than a plugin-hosting afterthought, means every capability that arrives later - observability, security enforcement, load balancing modes, BGP - is an extension of what's already there, not a bolt-on fighting it.

That's why I didn't pick it randomly. The BGP announcements that power the ingress story? Possible because the data plane was built for it. The Tetragon experiment waiting on the roadmap? Possible for the same reason. The credproxy work I teased last post leans on Cilium plumbing too. Each of those would have been a fight on a different fabric. Here, they're features being switched on.

It's the same lesson as the etcd backup from post 9, generalized: the work you do before you need it is the work that's still working when you need it. Up-front cost, compounding returns. That's not an argument for gold-plating everything - most layers should stay boring. It's an argument for knowing which one layer is worth it. In a cluster, networking is that layer.

TL;DR

• Stalwart runs my email: open source, which I financially support, and honestly the best mail server experience I've had - a comprehensive email solution with all the modern bells and whistles, trying to give people and companies what the big providers gave up on.
• Email at home means full control: encryption all the way, retention under my own policy, and no corporation mining my correspondence.
• DNS stays external and is not hard - though I still see a few gaps, nothing a good sysadmin can't sort out.
• Next chapter: networking, the layer that makes all of this reachable without becoming reachable to everyone.

Why email is the line I wouldn't cross

There's a category of self-hosting that is a hobby, and a category that is a statement. Email is the second. You can run Nextcloud, Jellyfin, Gitea, a wiki, half a dozen dashboards, and it's a better alternative to their cloud equivalents. But nobody mines your data out of those services with the same commercial appetite as your mailbox. Email is where your life's correspondence lives: confirmations, contracts, medical reminders, the account-reset links to everything else you own. Delegating it to an advertising company has always been the part of the "free" internet that made me the least comfortable. So email is the thing I kept for myself, through every platform this blog has covered - Proxmox, the Pi-and-NUC era, the McGyver cluster, and now this one.

That has a cost, and I want to state it honestly: self-hosted email is the single hardest service to run well, because the failure mode isn't your server going down. The failure mode is other people's servers deciding they don't trust yours. Deliverability is a social system wearing a technical costume - reputation, DNS hygiene, SPF, DKIM, DMARC - and you inherit every mistake your IP range ever made before you moved in. This is why most people's sensible advice is "don't." Mine is: don't unless the reason matters to you more than the friction. For me, the reason is the whole point.

Stalwart, and putting money where the dependency is

The mail server itself is Stalwart. I evaluated this space the way I evaluate everything that ends up on critical infrastructure, and Stalwart won on the things that matter at home: it is one binary doing SMTP, IMAP, JMAP, and the web admin, with built-in spam filtering, full-text search, and modern authentication - a comprehensive email solution with the modern bells and whistles, rather than a 2005 MTA held together with a decade of config fragments. It's fantastic, I love it, and I've watched it mature the way you watch something you depend on: closely.

One thing worth doing, because it matters more in this stack than in most: I financially support Stalwart. It's open source, it's the load-bearing wall of my correspondence, and the project is trying to give people and companies a real alternative to the big providers. When a single volunteer project carries that much weight in your life, the license being free is not the end of the transaction. It's the beginning of the bill you should be paying anyway.

Where it lives, and how the pieces connect

In the cluster, mail is a first-class tenant. It sits on the critical storage class from the storage map post - Longhorn PVCs at RF=2 - because an hour of mailbox unavailability is an inconvenience, but an hour of lost mail is a permanent wound. Post 12's war story about database restores exists in large part because mail was the workload I was protecting. Stalwart's data model treats messages and their indexes the way CNPG treats pages and WAL, and the same discipline applies: the live data is protected, and the restore path is drilled, not assumed.

DNS is the other half of the story, and it's the part people overcomplicate. My DNS management is external and not hard: the standard records - MX, SPF, DKIM keys, DMARC, autodiscover - published where the world looks them up, managed where I manage the rest of my zones. There are gaps I still see in that setup, things I want tighter than the defaults give me, but nothing a good sysadmin can't sort out with an afternoon and a checklist. I'll write that checklist up when it's done; it's exactly the kind of page this blog exists for.

The security posture is the payoff. Encryption all the way - TLS on the transport, keys rotated on my schedule, not a provider's. Retention under my own policy: I decide what archives, what expires, and what gets deleted forever, and no terms-of-service change can rewrite that decision overnight. And the quiet part underneath all of it: no corporation figuring out anything I do anymore. Not what I buy, not who I know, not when I'm awake, not which reset link proves which account is mine. That last category is the one nobody prices in until they think about it: your mailbox is the skeleton key to every other account you own, and I'd rather that skeleton key live on my own hardware.

The catch nobody mentions

Everything above only works if the world can reach my mail without me becoming reachable to the world. Deliverability is half DNS and half network posture - the IP reputation, the port 25 story, the question of how a home connection presents itself to the internet at large. That's not a mail problem. That's a networking problem. And networking is bigger than mail: it's the border of everything I've built so far. Every post in this arc eventually asks the same quiet question - how do packets find their way in? It deserves better than a passing mention.

So that's the next chapter, and it's a long one: three posts. First, Cilium - the network fabric itself, and why eBPF plus BGP support is what drew me to it. Then, ingress with Traefik - the front door in the proper sense. And finally, Tailscale - how the estate follows me out of the house without me becoming reachable to everyone.

The border of the house, drawn properly. Three times.

TL;DR

• My manifests spent the cluster's whole young life in a badly-organised monorepo, applied by hand while I was learning. That was fine as training; it's not an operating model.
• ArgoCD is now the thing that applies: it watches git and makes the cluster match it. I went in knowing the name from every CV requirement going, and came out understanding why it's on all of them.
• Setup was some work but surprisingly easy - the harder part was admitting the repo needed archaeology before a robot could be trusted with it.
• The git home is Forgejo, self-hosted and pre-dating the cluster (in a shape that wouldn't impress anyone), with SSO through Authentik. GitHub puts SSO behind a paywall; that's one more reason everything that lives there keeps migrating home.

The scaffolding post finally arrives. Posts 13 through 16 all pointed at it: the floor is mapped, the door is hung, the keys are stored - time for the thing that puts all of it together. This is that thing: GitOps, the operating model where the cluster's desired state lives in git, and a machine - not me - makes reality match the repo.

The monorepo era, an honest archaeology

First, the confession. For most of this cluster's life, my manifests lived in one big monorepo that I organised while learning, which means it was organised the way learning happens: incrementally, in whatever structure the current problem suggested. Stuff grouped by experiment, some things duplicated because I couldn't find the original, naming conventions that changed twice mid-stream. It worked, in the sense that the cluster worked. It was not, in any sense, presentable.

Applying it was manual too: edit, kubectl apply, hope. I want to defend that phase for a moment, because "just use GitOps from day one" is advice that ignores how anyone actually learns. Hand-applying manifests is how I learned what a deployment is, what a StatefulSet does when it's unhappy, and why the pending phase of a PVC can ruin an afternoon. You don't hand a robot your homework before you've done the homework.

But at some point the training wheels stop being training wheels and become just wheels. The moment that came for me was the realisation that every post in this series so far described a system whose reliability depended on me remembering things. The restore path lived in my head. The deployment procedure lived in my fingers. And a homelab whose reliability depends on one person's memory is a homelab with a single point of failure who occasionally goes to bed.

ArgoCD, and why it's on every CV

I knew of ArgoCD the way you know of a tool that appears in every job description: as a name. It took deploying it to understand why it's on so many CVs, and the reason is simpler than the marketing suggests. ArgoCD watches a git repository and continuously makes the cluster match it. That's the whole idea. The cluster's desired state is a git ref, drift is a diff, and "deploy" is a commit.

What sold me wasn't the sync, though. It's what the model does to mistakes. When I hand-apply a manifest and it breaks the cluster, the evidence about what changed lives in my shell history. When ArgoCD syncs a commit and it breaks the cluster, the evidence is the commit - reverted by a revert. The cluster's history becomes the repo's history, which is a history I can read, diff, and blame. In a cluster where I've already learned that memory is a failure mode, a machine that enforces the written-down state is not a luxury. It's the missing keystone.

The setup was some work - app-of-apps structure, sync policies, working out what gets auto-synced and what waits for a human - but surprisingly easy compared to what I'd built myself up to expect. The thing that made it easy was that the hard part wasn't ArgoCD. The hard part was the repo archaeology: you cannot point a continuous reconciler at a monorepo full of learning artifacts and half-truths, because a reconciler that enforces your desired state will faithfully enforce the broken bits too. So before the robot got the keys, the repo got the cleanup it had been owed since the day I first pointed kubectl at it. Things moved, got named consistently, and got deleted where "duplicated because I couldn't find the original" had been quietly load-bearing.

The git home, which predates the cluster

And then, in the middle of wiring all this up, I nearly forgot a vital component: where the git lives.

ArgoCD needs a git remote, and the obvious candidate is one I've had all along. Forgejo has been part of this setup longer than Kubernetes has - it even predates the cluster, running since the Proxmox days, just not in the greatest shape. It spent the whole Proxmox era as the box that hosted my repos and was therefore nobody's priority. Giving it a proper place in the picture turned out to be the last piece of the GitOps loop: code home, deployed by a robot, on the cluster that runs it. The cluster deploys from Forgejo, and Forgejo runs on the cluster. There's a certain recursion there that I find genuinely satisfying and my advisory-nature knows to be slightly circular, but git is the source and git is on the NAS-backed storage class, so the loop has a floor.

Authentik plays its part here too, of course: Forgejo signs in through the door from post 15, same policies, same second factors. And here I get to enjoy a small act of comparison shopping. GitHub puts SSO behind a paywall, which for a feature that is table stakes across the rest of the industry should embarrass them. I'm not paying a subscription to authenticate to a service I already pay for in other ways, especially when the alternative at home does it for free and lets me keep my repos on my own metal. It's one more item on the list of things that keeps migrating home from GitHub - code, CI, now this - and one more reason the migration keeps not being a sacrifice.

Not everything gets the robot

One thing I decided early and never regretted: not everything is trusted to auto-sync equally. The cluster is split by repo type, and the sync policy follows the blast radius.

Platform things - the operators and infrastructure that everything else stands on: CNPG, the storage layer, ingress, Authentik itself - do not auto-sync. When their manifests change, ArgoCD shows me the diff and waits. I read it, then I press the button. These are the pieces where a bad commit doesn't break one app, it breaks the floor everything stands on, and a diff I read before it lands is the cheapest insurance in this whole setup.

Applications - the things that merely run on the floor - auto-sync. If the repo changes, the cluster changes, and if that breaks a single app it breaks a single app. That's the trade working as intended: autonomy where the damage is bounded, a human where it isn't.

One deliberate gap: ArgoCD's Applications each point at one git ref. When I start deving my own apps, I'll use ApplicationSets to generate a dev and a live Application from the same template - one commit flowing to dev, promoted to live when dev has earned it. That's the operating model for self-written apps, and it's a story of its own rather than a paragraph here.

What this closes

With this in place, the arc's early promise is finally cashed. The cluster now runs the way I always claimed I wanted: nothing deployed by hand, nothing living only in my head, everything described in git and enforced by a machine that notices drift before I do. The boring posts about backups and restore drills had a hidden assumption - that the thing being restored has a known, written-down shape. GitOps is what makes that assumption true.

TL;DR

• Bitwarden, self-hosted because I can and recommended to anyone regardless: the human-facing vault, where I keep the credentials I type. It leans on Authentik for sign-in.
• Because of that dependency, the two systems must not trap each other: I don't keep Authentik-only credentials in the vault that needs Authentik to unlock. No circular lock-ins.
• OpenBao is the machine-facing vault: everything I don't want systems reading directly lives there. External Secrets Operator syncs from OpenBao into Kubernetes Secrets, so apps consume ESO output while source of truth stays in the vault.
• Same rule as identity: source of truth is a role, and it belongs to the service built for it.

Behind the door: keys. Two kinds of them, as it turns out, because the credentials a human types and the credentials a machine reads are two different problems wearing similar uniforms.

The one I'd recommend to anyone: Bitwarden

Let me get the disclaimer out of the way early: this one isn't a homelab Compromise, it's just a good product. Bitwarden is the password manager I'd recommend to people who have never heard of a homelab, and the fact that I can self-host it is a bonus stacked on top of a product that stands on its own. Great software that happens to be open source is rare enough that when it shows up, you use it and say thank you.

So there's a Bitwarden running in the cluster. It holds the credentials I type: logins, cards, notes, the stuff of daily life. It's an everyday essential in the most literal sense - I open it several times a day, and if it's down, I notice before any dashboard tells me. That's why it sits on the critical storage class from the fire map, same as mail and Authentik.

And it leans on Authentik to work. Single sign-on through the door from last post, second factor from the same policies. Which is convenient, and convenience creates a trap if you're not paying attention.

The circular lock-in

If my vault requires Authentik, and my Authentik credentials live in the vault, then losing access to either one locks me out of both. Forget the password to the vault: reset mail flows through Authentik, which is fine. Get locked out of Authentik somehow, or lose the hardware key that unlocks it: the vault now holds every credential except the one I need to get back in. A dependency loop, with my entire digital life inside it.

The fix is a rule, and it's boring the way good rules are: never store a system's own access credentials inside a system that depends on that system. Authentik's break-glass credentials, the vault's, the cluster's - they live outside the loop, on paper or in a place no federation can touch. The setup is self-hosted, but the escape hatch is manual, and that's on purpose. The day I need it is the day every automated path is broken anyway.

The machine-facing one: OpenBao

The other half of the problem never gets typed by a human at all. Applications need credentials: database passwords, API tokens, keys for the things that talk to other things. The naïve k8s answer is to put them in Secrets and move on, but a Secret is a config object, not a vault - it lives with the manifests, it gets copied into git-shaped places, and its access story is "whoever can read the cluster, more or less."

Here I have opinions, and one of them is historical: HashiCorp changed the license on Vault, and I don't trust a vendor that moves the goalposts mid-game. The open fork, OpenBao, is what I run instead. It's the machine-facing vault: everything I do not want systems to read directly lives there, with real access control on the way in.

The glue is External Secrets Operator. ESO watches the vault and syncs the right values into Kubernetes Secrets, so an app that speaks plain k8s Secrets keeps speaking plain k8s Secrets - it just receives its credentials from something that actually guards them. The sync lands in the cluster, but the source of truth never does. Rotate a credential in OpenBao and the cluster catches up. Delete the cluster and the vault shrugs: nothing of value was lost, because nothing of value was stored there.

The rule, again

Notice that this is the same rule as last post, wearing different clothes. Authentik is the source of truth for identity because it's built for identity. OpenBao is the source of truth for machine credentials because it's built for that. Bitwarden is where my human-scale credentials live because it's built for that too. Three systems, each doing its one job, none of them borrowing another's role, and the dependency arrows drawn so no loop closes.

The alternative - one tool doing all three jobs badly - is how you end up trusting a generalist with your rotation story and a vendor's roadmap with your license. Ask me how I know HashiCorp's roadmap looks.