Part 17: The robot that applies my git
TL;DR
- My manifests spent the cluster's whole young life in a badly-organised monorepo, applied by hand while I was learning. That was fine as training; it's not an operating model.
- ArgoCD is now the thing that applies: it watches git and makes the cluster match it. I went in knowing the name from every CV requirement going, and came out understanding why it's on all of them.
- Setup was some work but surprisingly easy - the harder part was admitting the repo needed archaeology before a robot could be trusted with it.
- The git home is Forgejo, self-hosted and pre-dating the cluster (in a shape that wouldn't impress anyone), with SSO through Authentik. GitHub puts SSO behind a paywall; that's one more reason everything that lives there keeps migrating home.
The scaffolding post finally arrives. Posts 13 through 16 all pointed at it: the floor is mapped, the door is hung, the keys are stored - time for the thing that puts all of it together. This is that thing: GitOps, the operating model where the cluster's desired state lives in git, and a machine - not me - makes reality match the repo.
The monorepo era, an honest archaeology
First, the confession. For most of this cluster's life, my manifests lived in one big monorepo that I organised while learning, which means it was organised the way learning happens: incrementally, in whatever structure the current problem suggested. Stuff grouped by experiment, some things duplicated because I couldn't find the original, naming conventions that changed twice mid-stream. It worked, in the sense that the cluster worked. It was not, in any sense, presentable.
Applying it was manual too: edit, kubectl apply, hope. I want to defend that phase for a moment, because "just use GitOps from day one" is advice that ignores how anyone actually learns. Hand-applying manifests is how I learned what a deployment is, what a StatefulSet does when it's unhappy, and why the pending phase of a PVC can ruin an afternoon. You don't hand a robot your homework before you've done the homework.
But at some point the training wheels stop being training wheels and become just wheels. The moment that came for me was the realisation that every post in this series so far described a system whose reliability depended on me remembering things. The restore path lived in my head. The deployment procedure lived in my fingers. And a homelab whose reliability depends on one person's memory is a homelab with a single point of failure who occasionally goes to bed.
ArgoCD, and why it's on every CV
I knew of ArgoCD the way you know of a tool that appears in every job description: as a name. It took deploying it to understand why it's on so many CVs, and the reason is simpler than the marketing suggests. ArgoCD watches a git repository and continuously makes the cluster match it. That's the whole idea. The cluster's desired state is a git ref, drift is a diff, and "deploy" is a commit.
What sold me wasn't the sync, though. It's what the model does to mistakes. When I hand-apply a manifest and it breaks the cluster, the evidence about what changed lives in my shell history. When ArgoCD syncs a commit and it breaks the cluster, the evidence is the commit - reverted by a revert. The cluster's history becomes the repo's history, which is a history I can read, diff, and blame. In a cluster where I've already learned that memory is a failure mode, a machine that enforces the written-down state is not a luxury. It's the missing keystone.
The setup was some work - app-of-apps structure, sync policies, working out what gets auto-synced and what waits for a human - but surprisingly easy compared to what I'd built myself up to expect. The thing that made it easy was that the hard part wasn't ArgoCD. The hard part was the repo archaeology: you cannot point a continuous reconciler at a monorepo full of learning artifacts and half-truths, because a reconciler that enforces your desired state will faithfully enforce the broken bits too. So before the robot got the keys, the repo got the cleanup it had been owed since the day I first pointed kubectl at it. Things moved, got named consistently, and got deleted where "duplicated because I couldn't find the original" had been quietly load-bearing.
The git home, which predates the cluster
And then, in the middle of wiring all this up, I nearly forgot a vital component: where the git lives.
ArgoCD needs a git remote, and the obvious candidate is one I've had all along. Forgejo has been part of this setup longer than Kubernetes has - it even predates the cluster, running since the Proxmox days, just not in the greatest shape. It spent the whole Proxmox era as the box that hosted my repos and was therefore nobody's priority. Giving it a proper place in the picture turned out to be the last piece of the GitOps loop: code home, deployed by a robot, on the cluster that runs it. The cluster deploys from Forgejo, and Forgejo runs on the cluster. There's a certain recursion there that I find genuinely satisfying and my advisory-nature knows to be slightly circular, but git is the source and git is on the NAS-backed storage class, so the loop has a floor.
Authentik plays its part here too, of course: Forgejo signs in through the door from post 15, same policies, same second factors. And here I get to enjoy a small act of comparison shopping. GitHub puts SSO behind a paywall, which for a feature that is table stakes across the rest of the industry should embarrass them. I'm not paying a subscription to authenticate to a service I already pay for in other ways, especially when the alternative at home does it for free and lets me keep my repos on my own metal. It's one more item on the list of things that keeps migrating home from GitHub - code, CI, now this - and one more reason the migration keeps not being a sacrifice.
Not everything gets the robot
One thing I decided early and never regretted: not everything is trusted to auto-sync equally. The cluster is split by repo type, and the sync policy follows the blast radius.
Platform things - the operators and infrastructure that everything else stands on: CNPG, the storage layer, ingress, Authentik itself - do not auto-sync. When their manifests change, ArgoCD shows me the diff and waits. I read it, then I press the button. These are the pieces where a bad commit doesn't break one app, it breaks the floor everything stands on, and a diff I read before it lands is the cheapest insurance in this whole setup.
Applications - the things that merely run on the floor - auto-sync. If the repo changes, the cluster changes, and if that breaks a single app it breaks a single app. That's the trade working as intended: autonomy where the damage is bounded, a human where it isn't.
One deliberate gap: ArgoCD's Applications each point at one git ref. When I start deving my own apps, I'll use ApplicationSets to generate a dev and a live Application from the same template - one commit flowing to dev, promoted to live when dev has earned it. That's the operating model for self-written apps, and it's a story of its own rather than a paragraph here.
What this closes
With this in place, the arc's early promise is finally cashed. The cluster now runs the way I always claimed I wanted: nothing deployed by hand, nothing living only in my head, everything described in git and enforced by a machine that notices drift before I do. The boring posts about backups and restore drills had a hidden assumption - that the thing being restored has a known, written-down shape. GitOps is what makes that assumption true.