homelab-cluster

Overview

Six nodes — a mix of NUCs and Pi 5s — running k3s across amd64 and arm64. Nothing is configured by hand. The cluster's entire desired state lives in a Git repo, ArgoCD reconciles against it continuously, and if a node dies I kubectl drain it and go get a beer.

The interesting part isn't the hardware, it's that adding a service is a git push and deleting a directory genuinely deletes the service.

GitOps, three layers deep

The bootstrap is a single app-of-apps: one ArgoCD Application pointed at gitops/, which pulls in two sources — the GitOps config itself, and the sealed secrets directory, recursively.

kind: Application
metadata:
  name: gitops
spec:
  sources:
    - repoURL: [email protected]:JustBrenkman/homelab.git
      path: gitops
    - repoURL: [email protected]:JustBrenkman/homelab.git
      path: gitops/secrets
      directory:
        recurse: true
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

That Application's kustomization contains exactly one resource: an ApplicationSet with a Git directory generator.

kind: ApplicationSet
spec:
  generators:
    - git:
        repoURL: [email protected]:JustBrenkman/homelab.git
        directories:
          - path: gitops/apps/*
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      destination:
        namespace: '{{path.basename}}'
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true]

This is the whole trick. The generator globs gitops/apps/* and templates one ArgoCD Application per directory it finds, named after the folder, deployed into a namespace named after the folder, with the namespace created on demand. There is no central registry of services to update. Creating gitops/apps/whatever/ with a kustomization.yaml is the entire process for adding a service; removing the directory prunes it.

prune: true and selfHeal: true are both on deliberately. Together they mean the cluster is not merely initialized from Git — it is continuously corrected toward Git. Anything I change with kubectl gets reverted within minutes, which is exactly the behavior I want, because it makes the repo the only honest description of the cluster.

What runs on it

That last row is not a joke — the site you are reading is ghcr.io/justbrenkman/justb-dev, deployed by the same ApplicationSet as everything else, reached through a Traefik IngressRoute. The write-up you're reading is served by the cluster it's describing.

Stateful services pin their storage with explicit PersistentVolume / PersistentVolumeClaim pairs rather than relying on dynamic provisioning, which on a small cluster with known disks is easier to reason about and easier to back up.

Secrets

Secrets are committed to the repo, which is only sane because they're Sealed Secrets — encrypted against the cluster's controller key, so the ciphertext is public-safe and only this cluster can decrypt it. Registry credentials, MQTT auth, Frigate's broker credentials, and Anvil's API secrets all live in gitops/secrets/ as .sealed.yaml.

It keeps the property that makes the whole setup work: the repo is complete. There's no out-of-band step, no kubectl create secret I have to remember, and a rebuild from bare metal needs the repo and the controller key, nothing else.