NOTE / TERRAIN

Argo CD on Kubernetes: The Other GitOps Control Plane.

Argo CD is not the default GitOps engine for every part of my homelab anymore. It is still important.

The repository has moved toward a Fleet-first model, but Argo CD remains an alternate control plane, the owner of a few legacy reconciliation paths, and a very useful record of how the platform evolved. Removing it simply because another controller became the default would erase working behaviour before that behaviour had been migrated.

Git repository
      │
      ▼
Argo CD repo server
      │
      ▼
Application controller
      │
      ├──► Applications
      └──► ApplicationSets
               │
               ▼
       target clusters and namespaces

The interesting part of this setup is not installing Argo CD. It is operating two generations of GitOps without pretending they are one system.

Why keep a second reconciler?

Platform migrations rarely happen in one clean cut. Some workloads still depend on Argo CD Application resources. Some cluster relationships are captured in ApplicationSets. Fleet now owns the preferred deployment pattern, but the old path cannot be called obsolete while it is still reconciling real state.

I give the two controllers an explicit boundary:

Fleet
  └── preferred current GitOps path

Argo CD
  ├── legacy applications not yet migrated
  ├── alternate operator visibility
  └── migration and recovery reference

The dangerous version of two reconcilers is allowing both to own the same resource. The useful version is knowing exactly which controller owns which path.

Deployment layout

Argo CD lives in the argocd namespace and is assembled with Kustomize:

argocd/
├── base/
├── shared/
├── prod/
└── oci/

The base establishes the controller stack. Shared resources describe projects and cluster access. The production and OCI paths add environment-specific applications, ApplicationSets and ingress.

The current manifests track the Argo CD 3.3.x family. The exact control plane includes the API server, repository server, application controller and supporting state such as Redis.

The repository is still the desired-state source of truth, but controller secrets are part of the recovery set. Git can recreate an Application definition; it cannot recreate a private repository credential or a target-cluster credential that was never stored in Git.

Applications versus ApplicationSets

An Application is the direct mapping:

one source
→ one revision and path
→ one destination

An ApplicationSet generates several Applications from a rule. That removes repetition, but it also adds another place where a small path change can fan out across clusters.

When an ApplicationSet stops generating the expected children, I check the generator and repository layout before touching the workloads themselves. A missing application may be a templating problem rather than an application failure.

Operating Argo CD

My first health check looks at both the controller pods and the objects they reconcile:

kubectl -n argocd get pods,svc,ingressroute
kubectl -n argocd get applications,applicationsets
kubectl -n argocd logs deploy/argocd-application-controller --tail=200

If an application remains OutOfSync or Degraded, I inspect it before pressing sync repeatedly:

kubectl -n argocd describe application <name>
kubectl -n argocd describe applicationset <name>
kubectl -n argocd logs deploy/argocd-repo-server --tail=200

The failure patterns usually fall into four groups:

repository unreadable
  └── credential, network or revision problem

manifest path missing
  └── repository layout changed

application generated incorrectly
  └── ApplicationSet rule or template

desired objects rejected
  └── target-cluster API, permissions or invalid manifests

The web UI is useful, but I do not treat it as the only source of diagnostics. When the API server or ingress is down, the Kubernetes objects and controller logs still describe what Argo CD is trying to do.

Secrets and recovery

Argo CD needs two kinds of trust material: credentials for reading repositories and credentials for reaching target clusters. Restore order matters.

My disaster-recovery sequence is:

1. Restore the argocd namespace and control-plane resources
2. Restore repository credentials
3. Restore target-cluster access
4. Apply base, shared resources and the active overlay
5. Wait for the repository server and controller
6. Confirm Application generation and sync state

Git provides most of the configuration, which makes the controller recoverable. The secrets reconnect that configuration to the outside world.

The recovery objective is not to preserve every transient status value inside Argo CD. It is to restore reconciliation. Once the controllers can read Git and reach their destinations, they can rediscover most of their useful state.

Rollback and migration

For a bad Argo CD configuration change, the first rollback is the previous Kustomize revision. For a broken ApplicationSet, I revert the smallest generator or path change that caused the fan-out.

I avoid solving a GitOps incident with unrelated live edits. A manual patch can make one resource green while leaving the controller determined to change it back.

Migration from Argo CD to Fleet follows a deliberately boring sequence:

identify current owner
→ reproduce desired state in Fleet
→ verify rendered resources
→ transfer ownership without overlap
→ observe reconciliation
→ remove the old Application

The ownership-transfer step is the whole migration. YAML conversion is the easy part.

Things worth remembering

Argo CD is a good example of why “legacy” should describe an architecture, not dismiss it.

At the time of writing:

Role           alternate GitOps controller
Namespace      argocd
Targets        homelab and OCI environments
Deployment     Kustomize
Version line   3.3.x
State source   Git plus repository and cluster secrets
Recovery       restore trust, reconcile, verify ownership

It can disappear after the last dependent workload has moved and the new path has been tested. Until then, it is production infrastructure—with all the backup, access and recovery discipline that implies.