Part 24: The runners, provisioned properly
TL;DR
- The last piece of the "code ownership at home" project wasn't storage or identity - it was compute for the builds. Forgejo runs on the cluster, but until now its runners didn't; a push could test itself somewhere that wasn't mine.
- Runners are the textbook scale-to-zero workload: two permanent runners burned resources around the clock on a constrained cluster, for a job that's idle most of its life. KEDA is the tool built for exactly that shape - watching the runner queue and spawning ephemeral pods on demand.
- The philosophical payoff: it makes the homelab aligned with the cloud's best idea - consume resources when you need them and no further - except the invoice is sent to nobody, because the metal is already mine.
- I didn't write the deployment from scratch. There's a helm chart on GitHub that got me 80% there, and getting the last 20% right is the part this post is about - which is the whole beauty of Open Source: somebody already walked the path, and the walking is now cheaper.
- The recursion from post 17 is now complete: the cluster deploys from Forgejo, Forgejo tests itself, and the builds run on the same metal as the code they build.
What the runners were before
Post 22 counted Forgejo's runners among the tenants - "with runners now" - but counted them the way you count a thing that exists, not the way you count a thing that's done. The honest version: runners had been registered and had executed the occasional job, but the provisioning story wasn't one I could describe without embarrassment. It was a runner or two, standing, permanent, boring, and wrong in the way permanent standing things in a cluster are always eventually wrong.
The wrongness was the shape, not the existence. I had two permanent runners, standing, allocated, always warm - two sets of resources consumed around the clock on what is, in the grand scheme of things, a constrained cluster. A CI runner is idle almost all of its life. Jobs arrive in bursts: you push, and for a few minutes you want compute, and then you don't again for hours. Standing runners pay the full price of existence for that pattern - allocated resources, warm pods, something to patch - for a fraction of the actual work. On a cluster where every node counts, two of them on permanent stand-by is the opposite of the frugality the whole estate is built on. Runners standing idle all day are paying rent without living there.
The scale-to-zero pattern was the obvious answer, and Kubernetes has a tool whose entire job is that pattern.
KEDA: the shape of the solution
KEDA - Kubernetes Event-driven Autoscaling - watches a queue, a metric, a stream, anything that can be measured, and scales a workload between zero and whatever the load asks for. When there's nothing to do, the workload doesn't exist. When work arrives, it appears, does the job, and dissolves again.
For a Forgejo runner pool, this is almost comically perfect. The runner is registered once, in the usual way, and waits - not as a pod, but as nothing at all. A push lands, Forgejo's runner queue gains a job, KEDA notices, and a runner pod spins up, picks up the job, runs the build or the tests, and either parks back at zero or waits briefly for the next one. The cluster's contribution to CI becomes: when you need a builder, here's a builder; when you don't, there isn't one.
What this buys is the same trade the whole estate runs on: cost where it's earned, absence where it isn't. The idle runners of the old shape were resources pretending to be a service. The KEDA shape is a service that exists only when used - which is also, I'd argue, the more honest Kubernetes idiom. The platform is for elasticity; using it for CI is the pattern working as intended.
There's a philosophical point under the mechanics, and it's worth spelling out: this makes a homelab philosophically aligned with the cloud's best idea. The thing the cloud actually got right was never the managed services or the dashboards - it was the economic model: consume resources when you need them and no further. That's an idea worth stealing on its merits, and Kubernetes plus KEDA lets me steal it entirely. The elasticity without the landlord, the scale-to-zero without the invoice. My cluster now bills the same way the cloud does - for work done, not for existence - except the invoice is sent to nobody, because I paid for the metal once and it's mine.
One honest wrinkle worth naming: scale-to-zero means the first job of a burst waits for a pod to schedule. It's seconds, not minutes, and in a homelab where the queue is mine and the patience is mine, that's a price I'm happy to pay. But it's a real trade - a cold start is a cold start - and anyone copying this setup should make it with eyes open.
Standing on shoulders
I didn't hand-write the deployment. There's a helm chart on GitHub - a community chart for Forgejo runners with KEDA scaling - and I started from it, the way you're supposed to.
This is the part of the story I want to defend rather than apologize for, because it highlights everything I love about Open Source. Somebody had the problem before me. They solved it, parameterized the parts that vary, and published it. I read their chart, understood their assumptions, kept the 80% that was right, and bent the 20% that wasn't toward my estate - my runner registration, my resource envelopes, the hooks into my storage and identity. The chart gave me the walk; my cluster gave the walk its own route.
The alternative, writing it from zero, buys nothing except the ability to say I did - and loses the review that comes free with a published chart, which is that dozens of people before me already hit its bugs. I'd rather inherit a solved problem and fix my deviation than invent a problem that was solved last year. Standing on shoulders isn't laziness; it's how the whole ecosystem compounds, and a homelab that pretends otherwise is just rebuilding the industry slower and worse.
And the deviations themselves are blog-sized: what I changed, why, and what the chart's defaults assumed about a cluster mine doesn't match. The mechanics of provisioning - the ScaledObject watching the queue, the ephemeral runner pods, the registration flow - are documented well enough elsewhere. What's under-documented is the judgment: which defaults to keep, which to challenge, and how to know the difference. That's the part this post is for.
The loop, closed
Here's where the recursion lands. Post 17 ended with the cluster deploying from Forgejo, and Forgejo running on the cluster - the loop I found genuinely satisfying and slightly circular. The runners close the last gap in that circle: now the CI that decides whether a commit is worth deploying runs on the same metal as the deployment it gates.
The full shape: I push to my forge. My forge's runner queue fills. A pod scales up on my cluster, builds and tests the change. If it passes, the manifests change in the repo, ArgoCD picks them up, and the cluster updates itself. Push, build, deploy - every hop on hardware I own, under policies I set, observed by the monitoring I built. Not one packet of that journey required a third party's goodwill, a paywall, or a quota.
flowchart LR
PUSH["git push"] --> FJ["Forgejo<br/>runner queue"]
FJ --> KEDA["KEDA<br/>watches the queue"]
KEDA -->|"0 to 1:<br/>spawn on demand"| RUN["Ephemeral<br/>runner pod"]
RUN -->|"build + test"| RES["Job result"]
RES -->|"pass -<br/>manifests change"| ARGO["ArgoCD"]
ARGO --> CLU["Cluster<br/>updates itself"]
CLU -.->|"same metal,<br/>end to end"| PUSH
That's what "code ownership at home" was always pointing at: hosting the whole lifecycle - the forge, the builds, the deployments, the tests - instead of storing my software's home address. It's where my software is born, proven, and shipped.
What this opens
A fully closed loop is also a loaded spring. The next thing this estate needs is something of mine to build - the first self-devved app, the model runner that's been in progress for a while now. With runners provisioned and scale-to-zero proven, the last excuse is gone: the infrastructure to build and deploy my own code is standing, waiting, elastic. The ApplicationSets dev/live story from post 17 has its stage set.
That's the runners done. The house builds houses now.