It gives me a private place to organise accounts, transactions, categories and budgeting history. The workload is small. The data is not. If the application disappears for an afternoon, nothing else in the homelab falls over. If its persistent data disappears, years of useful context can disappear with it.

That difference shapes the way I operate it.

Browser
   │
   ▼
Traefik
   │
   ▼
Actual Budget server
   │
   ▼
PVC-backed data directory

Traefik handles the HTTPS entry point. Actual Budget runs as a single Kubernetes Deployment and keeps its state on a persistent volume. The result is intentionally compact: one application pod, one Service, one route and one PVC.

Why self-host a budget?

Financial data is unusually good at revealing the shape of a life. It contains routines, places, subscriptions, mistakes and plans. I do not need a complicated threat model to prefer keeping that history on infrastructure I control.

Self-hosting also makes the boundary clear:

application can be recreated from Git
financial history must be restored from backup

The first half is a deployment problem. The second is a data-protection problem. Kubernetes helps with the first half, but it does not magically solve the second.

Deployment layout

The manifests use a reusable base and a cluster-specific overlay:

actualbudget/
├── base/
│   ├── deployment
│   ├── service
│   └── namespace
├── overlays/
│   └── local/
│       ├── ingress route
│       └── persistent volume claim
└── fleet.yaml

Kustomize builds the final resources and Rancher Fleet reconciles the overlay. The application lives in its own actualbudget namespace and currently runs as one replica.

The persistent claim is 2 GiB on local-path storage. That is modest, but capacity is not the interesting limit here. The important questions are whether the volume is mounted, whether it contains the expected database, and whether a usable copy exists somewhere outside the failure domain of that volume.

Authentication and exposure

The public request path terminates at Traefik. Authelia can sit in front of the route as an additional access-control layer, while Actual Budget still owns its application-level session and data model.

I treat those as separate controls:

Traefik and Authelia decide who reaches the application
Actual Budget decides what happens inside the application

A healthy ingress is not proof that the budget is usable. After an access-control change I open the real application, complete the authentication flow and load an account. That end-to-end check covers much more than an HTTP 200 from the proxy.

Operating Actual Budget

When the service is unavailable, I start with the complete Kubernetes path:

kubectl -n actualbudget get deploy,pod,svc,pvc,ingressroute
kubectl -n actualbudget describe deploy actualbudget
kubectl -n actualbudget logs deploy/actualbudget --tail=200

Then I check the volume:

kubectl -n actualbudget get pvc
kubectl -n actualbudget describe pvc actualbudget

The most useful failure split is simple:

pod Pending
  └── scheduling or volume binding

pod restarting
  └── application startup, permissions or damaged state

pod Ready, page unavailable
  └── Service, Traefik or authentication path

page loads, data missing
  └── wrong or empty volume mounted

That last case deserves care. Restarting a pod attached to an empty volume only produces a healthy empty application more efficiently.

Backup and recovery

The useful backup unit is the application data directory, not the Deployment manifest. Git already contains the resources needed to recreate the pod and route.

My recovery sequence is:

1. Restore the persistent data
2. Restore any required runtime secret inputs
3. Reconcile the local overlay
4. Wait for the pod and volume to become ready
5. Sign in and open real budget data

The service is a lower operational tier than ingress or authentication, so a short outage is acceptable. Data loss is not. The target is a daily snapshot and a recovery measured in hours rather than minutes.

A snapshot is only a recovery mechanism after it has been restored at least once. For this service, a good restore test is not merely seeing files on disk. It is opening the application and confirming that recent accounts, transactions and categories are coherent.

Upgrades and the latest tag

The current overlay tracks the latest application image. That is convenient, but it also means the exact application version is not fully expressed by Git.

This is technical debt I prefer to describe honestly:

latest tag
  ├── easier unattended updates
  └── weaker reproducibility and rollback

Before a meaningful upgrade, the safe order is snapshot first, change second, validate third. If the new image changes persistent state, rolling the Deployment back may not be enough; the matching data snapshot may also be required.

Moving to a pinned version would make Git history a more complete rollback record. Until then, I record the running image digest during incident work rather than assuming the tag tells the whole story.

Things worth remembering

Actual Budget is a useful reminder that criticality has two axes.

It has a small blast radius: when it stops, the cluster continues normally. It has valuable state: when its data disappears, replacing the pod is the easy part.

At the time of writing, the shape is:

Workload       single Deployment
Namespace      actualbudget
Ingress        Traefik, with optional Authelia
Storage        2 GiB local-path PVC
Deployment     Kustomize + Rancher Fleet
Image policy   latest tag
Recovery       restore data, reconcile, validate in the UI

It is deliberately a small Kubernetes application. The operational lesson is equally small: back up what cannot be rebuilt, and test the part the user actually cares about.

The repository has moved toward a Fleet-first model, but Argo CD remains an alternate control plane, the owner of a few legacy reconciliation paths, and a very useful record of how the platform evolved. Removing it simply because another controller became the default would erase working behaviour before that behaviour had been migrated.

Git repository
      │
      ▼
Argo CD repo server
      │
      ▼
Application controller
      │
      ├──► Applications
      └──► ApplicationSets
               │
               ▼
       target clusters and namespaces

The interesting part of this setup is not installing Argo CD. It is operating two generations of GitOps without pretending they are one system.

Why keep a second reconciler?

Platform migrations rarely happen in one clean cut. Some workloads still depend on Argo CD Application resources. Some cluster relationships are captured in ApplicationSets. Fleet now owns the preferred deployment pattern, but the old path cannot be called obsolete while it is still reconciling real state.

I give the two controllers an explicit boundary:

Fleet
  └── preferred current GitOps path

Argo CD
  ├── legacy applications not yet migrated
  ├── alternate operator visibility
  └── migration and recovery reference

The dangerous version of two reconcilers is allowing both to own the same resource. The useful version is knowing exactly which controller owns which path.

Deployment layout

Argo CD lives in the argocd namespace and is assembled with Kustomize:

argocd/
├── base/
├── shared/
├── prod/
└── oci/

The base establishes the controller stack. Shared resources describe projects and cluster access. The production and OCI paths add environment-specific applications, ApplicationSets and ingress.

The current manifests track the Argo CD 3.3.x family. The exact control plane includes the API server, repository server, application controller and supporting state such as Redis.

The repository is still the desired-state source of truth, but controller secrets are part of the recovery set. Git can recreate an Application definition; it cannot recreate a private repository credential or a target-cluster credential that was never stored in Git.

Applications versus ApplicationSets

An Application is the direct mapping:

one source
→ one revision and path
→ one destination

An ApplicationSet generates several Applications from a rule. That removes repetition, but it also adds another place where a small path change can fan out across clusters.

When an ApplicationSet stops generating the expected children, I check the generator and repository layout before touching the workloads themselves. A missing application may be a templating problem rather than an application failure.

Operating Argo CD

My first health check looks at both the controller pods and the objects they reconcile:

kubectl -n argocd get pods,svc,ingressroute
kubectl -n argocd get applications,applicationsets
kubectl -n argocd logs deploy/argocd-application-controller --tail=200

If an application remains OutOfSync or Degraded, I inspect it before pressing sync repeatedly:

kubectl -n argocd describe application <name>
kubectl -n argocd describe applicationset <name>
kubectl -n argocd logs deploy/argocd-repo-server --tail=200

The failure patterns usually fall into four groups:

repository unreadable
  └── credential, network or revision problem

manifest path missing
  └── repository layout changed

application generated incorrectly
  └── ApplicationSet rule or template

desired objects rejected
  └── target-cluster API, permissions or invalid manifests

The web UI is useful, but I do not treat it as the only source of diagnostics. When the API server or ingress is down, the Kubernetes objects and controller logs still describe what Argo CD is trying to do.

Secrets and recovery

Argo CD needs two kinds of trust material: credentials for reading repositories and credentials for reaching target clusters. Restore order matters.

My disaster-recovery sequence is:

1. Restore the argocd namespace and control-plane resources
2. Restore repository credentials
3. Restore target-cluster access
4. Apply base, shared resources and the active overlay
5. Wait for the repository server and controller
6. Confirm Application generation and sync state

Git provides most of the configuration, which makes the controller recoverable. The secrets reconnect that configuration to the outside world.

The recovery objective is not to preserve every transient status value inside Argo CD. It is to restore reconciliation. Once the controllers can read Git and reach their destinations, they can rediscover most of their useful state.

Rollback and migration

For a bad Argo CD configuration change, the first rollback is the previous Kustomize revision. For a broken ApplicationSet, I revert the smallest generator or path change that caused the fan-out.

I avoid solving a GitOps incident with unrelated live edits. A manual patch can make one resource green while leaving the controller determined to change it back.

Migration from Argo CD to Fleet follows a deliberately boring sequence:

identify current owner
→ reproduce desired state in Fleet
→ verify rendered resources
→ transfer ownership without overlap
→ observe reconciliation
→ remove the old Application

The ownership-transfer step is the whole migration. YAML conversion is the easy part.

Things worth remembering

Argo CD is a good example of why “legacy” should describe an architecture, not dismiss it.

At the time of writing:

Role           alternate GitOps controller
Namespace      argocd
Targets        homelab and OCI environments
Deployment     Kustomize
Version line   3.3.x
State source   Git plus repository and cluster secrets
Recovery       restore trust, reconcile, verify ownership

It can disappear after the last dependent workload has moved and the new path has been tested. Until then, it is production infrastructure—with all the backup, access and recovery discipline that implies.

A request to watch something may cross a media server, one or more catalogue managers, an indexer, a download client, a VPN boundary and several different storage areas. Each component is simple enough on its own. The operational work lives in the connections between them.

request
  │
  ├──► Sonarr / Radarr / Lidarr
  │          │
  │          ▼
  │       Prowlarr
  │          │
  │          ▼
  │   Transmission + WireGuard
  │          │
  ▼          ▼
Jellyfin ◄── shared media storage
  │
  └──► television

That is why I keep the stack together in the repository. It is not one container, but it is one operational story.

What is in the stack?

The reusable base currently defines the main applications:

Jellyfin      media playback
Sonarr        series management
Radarr        film management
Lidarr        music management
Prowlarr      indexer coordination
Bazarr        subtitle management
Transmission  download client
WireGuard     network boundary for the download path
Whisparr      an additional media catalogue

Some overlays add or alter supporting components. Prowlarr, for example, has used a Flaresolverr companion. The important design choice is that these differences remain visible in overlays rather than being copied into several complete stacks.

Deployment layout

The manifests use a base with development and homelab overlays:

arr-stack/
├── base/
│   ├── one workload per application
│   ├── services
│   └── namespace
├── overlays/
│   ├── dev/
│   └── homelab/
└── fleet.yaml

The homelab overlay runs in the media-stack namespace. Kustomize patches storage, Services, selected workloads and ingress; Rancher Fleet reconciles the result.

This is a place where the base/overlay split pays for itself. The application graph stays recognisable while the storage and network details can change per cluster.

Storage is the architecture

Media workloads make Kubernetes storage abstractions feel very concrete.

The homelab overlay separates small application configuration claims from large media volumes. Configuration occupies gigabytes. The media libraries are expressed in terabytes. Treating them as one undifferentiated PVC would make both backup policy and recovery needlessly expensive.

I divide the data by replaceability:

application configuration
  ├── indexes, histories and preferences
  └── small, valuable, worth frequent backup

download workspace
  ├── transient and partially complete data
  └── useful, but not equally valuable

media library
  ├── large payload
  └── separate retention and recovery decision

The mounts must also agree across applications. If Sonarr sees a file under one path and Transmission reports a different path for the same bytes, every pod can be healthy while imports quietly fail.

The VPN boundary

The download path is coupled to WireGuard. That is deliberate, but it creates an integration that needs its own health check.

Transmission process healthy
≠
download path safely usable

After changing network configuration I validate connectivity from the intended pod context and confirm that the download client can still reach both its external peers and the internal applications that control it.

I do not publish the provider configuration or internal routes. Those values belong to secret and environment-specific inputs, not to the public explanation of the architecture.

Operating the media stack

The first diagnostic pass asks Kubernetes for the whole namespace:

kubectl -n media-stack get deploy,pod,svc,pvc,ingressroute
kubectl -n media-stack get events --sort-by=.lastTimestamp

Then I narrow the problem to one edge of the graph:

kubectl -n media-stack logs deploy/jellyfin --tail=200
kubectl -n media-stack logs deploy/sonarr --tail=200
kubectl -n media-stack logs deploy/radarr --tail=200
kubectl -n media-stack logs deploy/prowlarr --tail=200

The most common categories are more useful than memorising every pod:

catalogue problem
  └── Sonarr, Radarr, Lidarr or Whisparr

search problem
  └── Prowlarr or a supporting resolver

download problem
  └── Transmission, WireGuard or path mapping

playback problem
  └── Jellyfin, storage throughput or media permissions

everything Ready, workflow stuck
  └── integration credentials, URLs or inconsistent mount paths

That last category is the recurring one. Kubernetes readiness proves that the processes are alive. It does not prove that an episode can travel from request to library to playback.

Backup and recovery

Backing up the entire media estate with one policy would confuse size with value.

My useful recovery set starts with:

Kustomize and Fleet configuration in Git
+
application configuration volumes
+
network and integration secret inputs
+
an explicit decision for the media library

The restore order follows the data flow:

1. Restore secrets and small configuration claims
2. Make shared storage available
3. Reconcile the overlay
4. Validate the VPN and download path
5. Validate catalogue-to-downloader path mappings
6. Play a real item through Jellyfin

The final playback test is the recovery test. A page full of green pods is only preparation for it.

Versioning and upgrades

Several images in the homelab overlay currently track moving tags. That keeps the stack fresh, but weakens reproducibility. With many coupled applications, simultaneous unplanned upgrades can make it difficult to identify which interface changed.

The safer long-term shape is pinned versions updated in small groups. Until every image is pinned, I capture running image IDs during incident work and avoid assuming that a rollback to the same YAML means a rollback to the same software.

Things worth remembering

At the time of writing, the stack is:

Namespace      media-stack
Workloads      Jellyfin plus the ARR and download services
Deployment     Kustomize + Rancher Fleet
Storage        separate config, workspace and large media volumes
Network        Traefik for selected UIs, WireGuard for download traffic
Weak point     integration boundaries and moving image tags
Recovery test  complete request-to-playback workflow

From the user side, the ideal media platform has one button. From the operator side, earning that simplicity means making every hidden hand-off explicit.

The idea is simple:

Internet
   │
   ▼
Traefik
   │
   ├──► Authelia
   │       │
   │       └──► MariaDB
   │
   ▼
Protected application

Traefik remains responsible for ingress and TLS. Before forwarding a request to a protected application, it asks Authelia whether that request should be allowed.

This gives me one authentication policy instead of implementing authentication independently in every application.

My deployment currently runs Authelia 4.39.20 on Kubernetes using Kustomize and Rancher Fleet.

Why Authelia?

A homelab eventually accumulates applications with very different authentication capabilities.

Some support OIDC properly. Some have basic local authentication. Some have no authentication worth trusting on the public Internet.

Rather than solving that problem separately in every application, I use Authelia as a shared authentication gateway.

user
 ↓
Traefik
 ↓
Authelia policy
 ↓
MFA if required
 ↓
application

For Internet-facing services, my default policy is two-factor authentication. Trusted internal or VPN networks can bypass MFA for selected applications where forcing a browser authentication flow would be impractical.

The important part is that bypasses are explicit exceptions rather than the default.

Authentication methods

My current setup supports three second-factor mechanisms:

• TOTP as the normal MFA mechanism
• Duo Push for phone-based approval
• WebAuthn for compatible authenticators

TOTP is intentionally the baseline. It is simple, portable and does not make the authentication platform entirely dependent on an external push provider. Duo remains a convenient additional option, while WebAuthn gives me a stronger path where supported.

Deployment layout

I keep a reusable base and cluster-specific overlays:

authelia/
├── base/
│   └── kustomization.yaml
└── overlays/
    ├── homelab/
    ├── local/
    └── jls/

Fleet reconciles the appropriate overlay into each target cluster. The application lives in the auth namespace and consists primarily of an Authelia Deployment and a MariaDB StatefulSet.

This keeps common configuration centralised while allowing environment-specific storage, routing and resources to remain in overlays. It is much easier to reason about than three nearly identical copies of the entire deployment.

Configuration and secrets

Authelia needs considerably more secret material than a typical stateless application. The deployment depends on values for:

session encryption
storage encryption
identity validation
database access
mail delivery
MFA providers

Those values must survive disaster recovery.

Losing the workload is mostly harmless. Losing encryption keys while retaining the database is considerably more interesting.

My recovery rule is therefore:

Configuration, database and cryptographic secrets belong to the same recovery plan.

Secrets currently come from Kubernetes Secret manifests managed with the deployment. Moving them to a dedicated secret-management system is a separate migration. Changing the storage mechanism and rotating authentication secrets at the same time would make rollback unnecessarily difficult.

Database

Authelia currently uses MariaDB. The database preserves session and service state, so it runs as a StatefulSet with persistent storage.

For a new deployment I would evaluate PostgreSQL first, but there is little value in migrating a working authentication database merely for architectural aesthetics.

A database migration deserves its own sequence:

backup
→ migration
→ validation
→ rollback plan

It should not be hidden inside an Authelia upgrade.

SMTP matters more than it looks

Mail delivery is part of the authentication system. Password resets and identity workflows depend on it, which means authentication can appear partially functional even when SMTP is broken.

After changing mail configuration I test a complete workflow rather than checking only whether the Authelia pod is Ready.

A healthy pod does not necessarily mean a healthy authentication service.

Operating Authelia

When authentication starts behaving strangely, I inspect the complete request chain instead of immediately restarting Authelia.

My first commands are usually:

kubectl -n auth get pods,svc,pvc,ingressroute
kubectl -n auth get deploy,statefulset
kubectl -n auth logs deploy/authelia --tail=200

Then I validate the active configuration:

kubectl -n auth exec deploy/authelia -- \
  authelia validate-config \
  --config /config/configuration.yaml

This catches a surprising number of problems before deeper investigation becomes necessary.

Check the database too

Because an authentication request crosses several components, inspecting only Authelia can be misleading.

kubectl -n auth describe deploy authelia
kubectl -n auth describe statefulset authelia-db
kubectl -n auth logs statefulset/authelia-db --tail=100

The failure patterns I care about most are:

Authelia
   ├── database connectivity
   ├── invalid or missing secrets
   ├── mail failures
   ├── MFA enrolment problems
   └── stale sessions after secret rotation

Traefik
   └── broken forward-auth middleware

MariaDB
   └── unavailable or damaged persistent state

Authentication redirect loops are particularly useful clues. They often indicate that each component is individually running but the integration between Traefik, Authelia and the application is wrong.

Resource sizing

Authelia itself is inexpensive for my workload:

CPU request       100m
Memory request    128 MiB
Memory limit      512 MiB

One environment receives more headroom. The extra capacity is mainly useful during memory-intensive password hashing such as Argon2.

MariaDB currently requests 50m CPU and 128 MiB memory with a 512 MiB memory limit.

I change these values in Git and let GitOps reconcile them rather than editing live resources. Actual use can be checked with:

kubectl -n auth top pod

Backup and recovery

Because Authelia sits in front of several applications, I treat it as a Tier-1 homelab service.

The useful backup set is not simply the PVC:

MariaDB data
+
Authelia secrets
+
configuration
+
Traefik integration

My recovery order is:

1. Restore secrets
2. Restore database or PVC state
3. Reconcile the GitOps overlay
4. Wait for Authelia and MariaDB
5. Validate configuration
6. Test forward authentication against a real protected application

The final step matters. Seeing 1/1 Running is not a recovery test. Successfully authenticating through Traefik is.

Rollback

Application changes are deployed through Git, so the normal rollback is equally boring:

revert Git revision
        ↓
Fleet reconciliation
        ↓
previous known-good state

If an upgrade modified persistent state, the corresponding database snapshot may also have to be restored.

Authentication policy changes need slightly different treatment. If an MFA policy unexpectedly locks users out, I revert the access-control rule rather than changing encryption secrets in an attempt to regain access.

Change the smallest thing necessary.

Things worth remembering

Authelia itself is not particularly difficult to run. The interesting part is everything around it.

It sits at the intersection of:

DNS
TLS
Traefik
sessions
database
mail
secrets
MFA
application routing

That means an “Authelia outage” frequently is not an Authelia outage.

At the time of writing:

Authelia      4.39.20
Ingress       Traefik
Database      MariaDB
Deployment    Kustomize + Rancher Fleet
MFA default   TOTP
Additional    Duo Push + WebAuthn
Backups       database + secrets + PVC snapshot

Authentication is one of the places where I prefer boring components, explicit recovery procedures and predictable Git history over clever infrastructure.

The service only becomes interesting when it disappears. Ideally, it does not.

From Kubernetes’ point of view the deployment is ordinary. One application pod talks to one PostgreSQL database. Traefik provides the web route. From my point of view the data is irreplaceable in a way that CPU time is not.

Browser
   │
   ▼
Traefik
   │
   ▼
Baby Buddy
   │
   ▼
PostgreSQL 16
   │
   ▼
5 GiB persistent volume

That is the whole architecture, and it is enough.

Why run it here?

Family tracking data is useful because it is detailed. The same detail makes it private. Self-hosting gives me control over where that history lives, when it is backed up and when it is removed.

The application does not need a large platform around it. It needs predictable availability, safe access and a database that can be restored.

The distinction I keep in mind is:

Baby Buddy container
  └── replaceable

PostgreSQL contents
  └── not replaceable

Kubernetes is good at recreating the first. My backup process has to protect the second.

Deployment layout

The repository uses a Kustomize base and two environment overlays:

babybuddy/
├── base/
│   ├── application Deployment
│   ├── PostgreSQL StatefulSet
│   ├── Services
│   └── namespace
├── overlays/
│   ├── homelab/
│   └── oci/
└── fleet.yaml

Rancher Fleet selects the appropriate overlay. Both environments use the babybuddy namespace, while each overlay owns its route and any cluster-specific changes.

The application and PostgreSQL each run as a single replica. This is not a highly available database design, and it does not pretend to be. For this workload, reliable recovery is more useful than adding distributed-database complexity.

Resources and storage

The application requests 100m CPU and 256 MiB memory, with limits of one CPU and 512 MiB memory. PostgreSQL uses the same request and limit shape.

The database has a 5 GiB persistent volume claim on local-path storage.

Those values tell me two useful things. First, the workload is inexpensive enough to remain resident. Second, the database is small enough that frequent logical backups are practical.

Small data deserves good backups too.

Configuration and secrets

The application needs a database connection and its own runtime settings. PostgreSQL needs its database name, user and password. These values are injected through Kubernetes configuration rather than written into this public description.

I treat the application secret and the database backup as a pair. A restored database without the expected connection material is not a restored service. Conversely, a perfect Secret does not help if the data volume is empty.

recovery set
  ├── PostgreSQL data
  ├── application and database secret inputs
  ├── Kustomize overlay
  └── ingress policy

Operating Baby Buddy

For an outage I start with both workloads:

kubectl -n babybuddy get deploy,statefulset,pod,svc,pvc,ingressroute
kubectl -n babybuddy logs deploy/babybuddy --tail=200
kubectl -n babybuddy logs statefulset/babybuddy-db --tail=200

Resource names can vary slightly after a manifest change, so I confirm them with kubectl get before using logs as a script.

The request path produces a useful decision tree:

route unavailable
  └── Traefik, Service or pod readiness

application starts but errors
  └── database connectivity or migrations

database pod Pending
  └── volume binding or node storage

page loads with an empty history
  └── stop and verify the mounted database before writing new data

The last situation is the one where patience matters. An empty interface can look healthy enough to invite new entries, which would make recovery and reconciliation more confusing.

Backup and recovery

For PostgreSQL I prefer a logical database backup plus an infrastructure-level volume snapshot. They answer different questions.

logical dump
  └── portable, inspectable database recovery

volume snapshot
  └── fast restoration of the exact filesystem state

The restore sequence is:

1. Restore the database secret inputs
2. Restore the PostgreSQL data or create a clean database
3. Reconcile the base and selected overlay
4. Import the logical backup if needed
5. Wait for application migrations
6. Sign in and check recent real records

The useful validation is not only a successful connection. I check that recent entries and expected history are present and that a new test entry can be written and read.

Upgrades

PostgreSQL is pinned to the 16 major line. The Baby Buddy application currently tracks a moving latest tag.

That makes the pre-upgrade database backup important. A Deployment rollback can restore an older container, but it cannot automatically reverse a schema migration already applied to PostgreSQL.

The safe order is:

backup
→ record running image
→ deploy
→ watch migrations and logs
→ validate the UI

Pinning the application to an explicit version would improve reproducibility and remains the cleaner long-term state.

Things worth remembering

At the time of writing:

Application     Baby Buddy, one Deployment
Database        PostgreSQL 16, one StatefulSet
Namespace       babybuddy
Storage         5 GiB local-path claim
Deployment      Kustomize + Rancher Fleet
Application tag latest
Recovery test   recent history plus a real write/read

This service is a good example of why resource size is a poor measure of importance. It consumes little. The data it carries deserves care.

Most Kubernetes applications are concerned with their own volumes. BackupPC is concerned with the machines around Kubernetes: workstations, servers and other systems whose useful data does not live in a pod at all.

machines on the network
         │
         ▼
     BackupPC
       │   │
       │   └──► configuration state
       ▼
   backup pool

Running a backup server as a Kubernetes Deployment is convenient, but it introduces a circular question: what backs up the backup service?

Why BackupPC?

My estate is mixed. Some data lives in Kubernetes volumes. Some lives on ordinary Linux hosts. Some machines are not always online. BackupPC provides a central schedule and history for the latter group.

It is not the only layer of protection. Storage snapshots protect Kubernetes state; Git protects manifests; application-aware dumps protect databases. BackupPC covers filesystems outside those boundaries.

Git                desired configuration
volume snapshots   Kubernetes persistent state
database dumps     application-consistent state
BackupPC           files on machines outside the cluster

The overlap is intentional. A single backup mechanism rarely matches every kind of data.

Deployment shape

The current workload is a single Deployment in the prod namespace using the adferrand/backuppc image. A Kubernetes Service exposes the web application internally, and Traefik provides the controlled browser route.

The manifest also defines separate persistent areas for BackupPC’s configuration and working state. The declared Kubernetes claims are small because the large backup pool follows its own storage path and capacity planning.

That distinction matters:

BackupPC configuration
  └── schedules, hosts and application state

backup pool
  └── the actual retained data, usually much larger

Restoring only one half produces either a server with no history or a pile of backup data the server does not understand.

Access to client machines

A backup job crosses several trust boundaries. BackupPC needs a network path to a client, an authentication method, permission to read the intended files, and enough time to complete.

I keep those concerns separate when debugging:

can the pod resolve and reach the client?
can it authenticate?
can it read the selected paths?
can it write to the backup pool?
does the job finish inside its window?

A failed backup is not always a BackupPC application problem. A sleeping workstation, rotated key, changed filesystem permission or full storage target can produce the same red result in the UI.

Credentials and client inventories are not part of this public log. They belong in the protected runtime configuration and in the recovery documentation kept with the estate.

Operating BackupPC

My first Kubernetes checks are:

kubectl -n prod get deploy,pod,svc,pvc,ingressroute
kubectl -n prod logs deploy/backuppc --tail=200
kubectl -n prod describe deploy backuppc

Then I check storage attachment and capacity. A Ready web interface is not useful if the pool is read-only or full.

kubectl -n prod get pvc
kubectl -n prod describe pvc

For a single failing client, I avoid restarting the whole service. I compare the last successful run, test network reachability from the pod and verify that the client-side access method still works.

The diagnostic split is:

all clients fail
  └── server, shared credential, network or backup-pool issue

one client fails
  └── client availability, key, permissions or path change

jobs run but retention is wrong
  └── scheduling or pool configuration

UI works but restores fail
  └── catalogue and retained data no longer agree

That final category is why restore testing matters more than a row of successful job timestamps.

Backup verification

I judge BackupPC by restores, not backups.

A lightweight verification selects a small file from a recent backup, restores it to a safe temporary location and compares the content. A deeper exercise restores a directory tree with permissions and timestamps, then records how long the process took.

backup completed
→ file visible in catalogue
→ restore requested
→ bytes recovered
→ content and metadata checked

This tests the catalogue, the pool, permissions and the operator path at once.

Recovering BackupPC itself

The useful recovery set includes:

Kubernetes manifests
+
BackupPC configuration and catalogue
+
protected access credentials
+
backup pool

My recovery order is:

1. Make the backup storage available without modifying it
2. Restore BackupPC configuration and catalogue state
3. Restore access credentials
4. Reconcile the Deployment and route
5. Confirm old backup generations are visible
6. Restore a known file
7. Only then resume scheduled writes

Mounting the pool read-only during the first inspection is a useful precaution. It prevents a misconfigured fresh instance from treating old data as disposable workspace.

Rollback and upgrades

The image is not currently pinned to an explicit version in the Deployment. That makes the running image digest part of any serious incident record.

Before an upgrade I preserve the catalogue and configuration, note the current digest and verify a recent restore. If the upgraded service cannot read its existing pool, reverting the YAML alone may fetch the same moving image again.

Pinning the application version would make this much less ambiguous.

Things worth remembering

At the time of writing:

Workload       single Deployment
Namespace      prod
Ingress        Traefik
State          configuration, catalogue and backup pool
Clients        machines outside Kubernetes
Weak point     an untested restore path
Recovery test  restore and compare a real file

A backup system is infrastructure whose output may remain unused for months. That makes routine evidence unusually important. The most reassuring green icon is a file that came back.

The piece I run in Kubernetes is centred on the local API. Agents and security components can report observations, the API keeps shared state, and consumers can ask whether an address should be allowed or blocked.

logs and security agents
          │
          ▼
     CrowdSec LAPI
       │       │
       │       └──► decision consumers
       ▼
 persistent state

There is also a Metabase container in the current pod for exploring the data. It makes the Deployment heavier than the CrowdSec API alone, but it gives the stored decisions a human-readable view.

Why centralise decisions?

Each Internet-facing component can see only part of an attack. A reverse proxy sees requests. A host agent sees authentication failures. A bouncer enforces a decision but may not know how that decision was reached.

The local API provides a meeting point:

observation
→ scenario matched
→ decision stored
→ bouncer queries decision
→ traffic allowed or denied

This is useful because enforcement remains close to the edge while detection can combine signals over time.

It is not a substitute for authentication, patching or firewall policy. CrowdSec is another layer, particularly good at turning repetitive background noise into something actionable.

Current deployment

The current Kubernetes resources are compact and somewhat legacy in layout: a persistent volume claim and a Deployment rather than a full base-and-overlay tree.

The Deployment runs:

CrowdSec       v1.4.1
Metabase       v0.44.4
init helper    prepares shared state

The persistent claim is 500 MiB on Longhorn storage. CrowdSec itself requests 50m CPU and 50 MiB memory, with limits of 150m CPU and 100 MiB memory. Metabase requests the same small baseline but has a much larger memory ceiling of roughly 1.5 GiB.

That resource difference is a useful troubleshooting clue. If the pod is killed for memory, the analytics side may be the cause even when CrowdSec appears in the pod name.

State and trust

The API stores decisions, enrolment information and operational context. Losing it does not remove the underlying log sources, but it does interrupt the shared memory of the security layer.

Clients also need credentials to report or consume decisions. I keep those credentials out of the public manifests and out of this article.

The recovery set is therefore:

LAPI persistent state
+
agent and bouncer credentials
+
Deployment configuration
+
the enforcement integration

Restoring the API without reconnecting a bouncer produces a healthy dashboard and no protection. Restoring a bouncer credential without the API state produces a client with nothing useful to ask.

Operating CrowdSec

I start with the pod, Service and persistent claim:

kubectl get deploy,pod,svc,pvc -l app=crowdsec
kubectl logs deploy/crowdsec-lapi -c crowdsec --tail=200
kubectl logs deploy/crowdsec-lapi -c metabase --tail=100

Because the two main containers share a pod, I always name the container when reading logs. Otherwise it is easy to investigate the analytics UI while the local API is the component that agents actually depend on.

The useful failure map is:

agents cannot report
  └── LAPI Service, credentials or network path

bouncers receive no decisions
  └── enrolment, API state or enforcement integration

pod restarts under memory pressure
  └── inspect each container, especially Metabase

dashboard unavailable, enforcement works
  └── analytics failure rather than security-path failure

everything healthy, attacks pass through
  └── scenario coverage or bouncer placement

The last case is the most important. CrowdSec can be perfectly healthy as software and irrelevant as a control if the enforcement point is not actually consulting it.

End-to-end validation

I prefer a safe synthetic validation over waiting for hostile traffic.

The test should prove three separate things:

1. an agent can reach the local API
2. a known test decision appears in the API
3. the intended bouncer observes that decision

I do not use a production client address for the test. Security automation is a poor place for an accidental self-lockout.

After validation, I remove the test decision and confirm normal access.

Backup and recovery

The persistent data is small enough that frequent snapshots are inexpensive. The less visible task is preserving the credential and enrolment relationships around it.

My recovery sequence is:

1. Restore the PVC state
2. Recreate the Deployment and Service
3. Restore agent and bouncer trust material
4. Confirm agents are reporting
5. Confirm consumers can query decisions
6. Run a safe end-to-end decision test

If the API state cannot be restored, re-enrolling clients is possible, but it is a rebuild rather than a transparent recovery. I document that distinction before an incident.

Upgrades and age

The pinned versions in this deployment are old enough that an upgrade deserves its own planned change. Jumping several releases can affect data formats, collections, scenarios and client compatibility.

The safe approach is:

snapshot
→ read release and migration notes
→ update CrowdSec separately from Metabase
→ validate agents
→ validate bouncers
→ keep a rollback image and data snapshot

Separating the two application upgrades keeps a CrowdSec problem from being confused with a Metabase problem.

Things worth remembering

At the time of writing:

Core            CrowdSec LAPI 1.4.1
Analytics       Metabase 0.44.4
Workload        multi-container Deployment
Storage         500 MiB Longhorn claim
Purpose         shared observations and decisions
Recovery test   agent → LAPI → bouncer

The value of CrowdSec is not the number of decisions stored. It is whether a useful signal reaches the correct enforcement point before the next request does.

That distinction is deliberate. The dashboard collects links and gives the growing application estate a readable shape. If Dashy is unavailable, the services behind it should still work. A broken map is inconvenient; it should not close the roads.

Browser
   │
   ▼
Traefik
   │
   ▼
Dashy
   │
   ├──► platform tools
   ├──► security tools
   ├──► data applications
   └──► media applications

That makes Dashy operationally simple and editorially important. It is where a collection of workloads starts to feel like one homelab.

Why a dashboard?

Bookmarks work until they do not. As services move, multiply or gain different access policies, a curated launch page becomes easier to maintain than browser state scattered across devices.

Dashy gives me one place to express:

what the service is called
what it is for
where its entry point lives
which group it belongs to

The dashboard is also a quiet inventory check. A dead link often reveals an ingress change, a retired application or an undocumented rename.

I avoid turning it into a control plane. Dashy does not own the service, the route or the authentication policy. It points to them.

Current deployment

The Kubernetes shape is compact:

single Deployment
    │
    ├── Dashy container
    ├── volume-permissions init container
    └── 500 MiB persistent claim

Service
    │
    ▼
Traefik route

Dashy runs as one replica. The main container mounts /app/public from a 500 MiB Longhorn-backed claim. A BusyBox init container fixes ownership on that volume before Dashy starts.

The pod runs with user and group ID 1000 and the Europe/Paris timezone. Those are small details, but mismatched permissions are a common reason a dashboard can read its initial state and then fail when it tries to update it.

The current Dashy image is not pinned and uses Always pull policy. The permissions helper is pinned to BusyBox 1.31.1.

Configuration is content

For this service the persistent files are not a database in the usual sense. They are the dashboard’s content and presentation state.

I treat that material as curated configuration:

links
groups
labels
icons
layout

A restore that brings the pod back but loses the catalogue produces a technically healthy blank dashboard. That is why the volume contents belong in backup, even though the application itself is disposable.

There is also a privacy boundary. The public story can name the products in the lab. The private dashboard configuration can contain routes and labels that should not be published. Backing it up does not mean copying it into Git without review.

Authentication and failure boundaries

Traefik owns the HTTPS route, and the authentication layer can protect the page before Dashy receives the request.

I test those concerns separately:

route and TLS
→ authentication flow
→ Dashy page
→ one representative destination link

A redirect loop is usually an ingress or authentication integration problem. A page with broken cards is usually dashboard content. A working card that opens a dead service is a downstream application problem.

Keeping those boundaries visible prevents a harmless dashboard issue from becoming an unnecessary restart of half the cluster.

Operating Dashy

The first Kubernetes checks are:

kubectl get deploy,pod,svc,pvc,ingressroute -l app=dashy
kubectl logs deploy/dashy -c dashy --tail=200
kubectl logs deploy/dashy -c volume-permissions --tail=100

If the pod is stuck in init, I inspect the claim and file ownership. If the pod is Ready but the page is unavailable, I inspect the Service and Traefik route. If the page loads with missing content, I stop before saving changes and verify which volume is mounted.

Init:Error
  └── volume attachment or ownership

CrashLoopBackOff
  └── application startup or incompatible persisted content

Ready, no page
  └── Service or ingress

Ready, empty page
  └── wrong, empty or unreadable persistent state

For a dashboard, an end-to-end health check opens the page and follows at least one link from each major group. That catches stale content which pod probes cannot see.

Backup and recovery

The useful backup set is small:

Dashy persistent content
+
runtime access policy
+
Kubernetes manifests

Git or the manifest repository can recreate the Deployment, Service and route. A snapshot or file backup must preserve the curated content.

My recovery order is:

1. Restore the persistent content
2. Recreate the PVC if necessary
3. Apply the workload and route
4. Confirm the init container completes
5. Open Dashy through the normal authentication path
6. Follow representative links

The final link test is important. Restoring a year-old dashboard perfectly can still produce a page full of obsolete routes.

Upgrades

The unpinned image and Always pull policy mean a pod restart can also be an application upgrade. That is convenient until it becomes surprising.

Before restarting during an incident, I record the current image ID:

kubectl get pod -l app=dashy \
  -o jsonpath='{range .items[*].status.containerStatuses[*]}{.imageID}{"\n"}{end}'

The safer future state is an explicit Dashy version updated through Git. That turns a restart back into a restart and makes rollback predictable.

Things worth remembering

At the time of writing:

Workload       one Deployment
Storage        500 MiB Longhorn claim
Init           BusyBox permissions helper
Ingress        Traefik with shared access control
Image policy   unpinned, always pull
Recovery test  page plus representative destination links

Dashy’s best feature is not that it makes the homelab look organised. It is that the organisation remains useful when the underlying estate changes.

Trivy, Gitleaks, Kubeconform and Nessus can all produce useful results. Without a common intake, those results remain scattered across CI logs and exported reports. DefectDojo gives them a shared product, engagement and test history so repeated scans can update findings instead of creating another pile of disconnected output.

CI scanners and scheduled scans
            │
            ▼
       DefectDojo API
            │
            ▼
Traefik → nginx → Django / uWSGI
                    │
                    ├──► PostgreSQL
                    ├──► Valkey
                    └──► Celery worker and beat

This is one of the more substantial application stacks in my homelab. The complexity is justified by the workflow it creates, but it deserves an equally explicit recovery plan.

One inbox for findings

My import automation uses the v2 API and re-import semantics. The first upload establishes the scan context; later pipeline runs update the same Test rather than duplicating every finding.

The current sources include:

Trivy       filesystem and dependency findings
Gitleaks    secret-scanning findings
Kubeconform Kubernetes schema findings
Nessus      scheduled infrastructure findings

The important output is not a larger finding count. It is a shorter path from observation to triage, remediation and evidence that the issue stayed fixed.

Deployment layout

DefectDojo uses a Fleet-native Helm release with a Helm-free Kustomize post-render layer. A separate Helm-enabled Kustomize entry point remains available for local rendering.

defectdojo/
├── base/
├── overlays/
│   ├── homelab/
│   │   ├── values
│   │   └── fleet post-render resources
│   └── local/
│       ├── values
│       └── fleet post-render resources
└── fleet.yaml

The homelab environment currently uses chart 1.9.29; the local, production-like environment uses chart 1.9.37.

The application lives in the defectdojo namespace. Its runtime includes Django, nginx, Celery worker and scheduler, PostgreSQL, Valkey and an initializer Job.

Why the render path is split

Fleet inflates the Helm chart itself. Its post-render Kustomize directory therefore must not contain another helmCharts block. The manual entry point does contain that block and must be rendered with Helm support enabled.

Fleet path
  Helm render by Fleet
  → Helm-free Kustomize post-render

manual path
  Kustomize with Helm enabled
  → complete local render

Mixing the two paths produces a memorable but unhelpful failure: the controller asks Kustomize to render a Helm chart without the Helm flag. Keeping the paths separate makes both uses explicit.

There is a second Helm wrinkle around the PostgreSQL password. During an upgrade, the subchart tries to recover an existing value with lookup(). Fleet renders in a dry-run context where that lookup cannot see the downstream Secret. I provide the same value through Fleet’s protected values input at render time while the pod still reads its runtime credential from the namespace Secret.

The lesson is broader than this chart: render-time configuration and runtime configuration can have different visibility, even when they represent the same credential.

Node placement and storage

The local cluster spans several cloud providers. I pin all DefectDojo components to one labelled node so Django, workers, PostgreSQL, Valkey and local-path storage remain together.

The chart does not honour one global node selector, so the overlay applies placement to every component. If the label disappears, the failure appears as a collection of Pending pods and unbound volumes.

Persistent state includes:

PostgreSQL   findings, users and workflow state
Valkey       queue and cache support
media PVC    uploaded reports and attachments

The media claim is 20 GiB. PostgreSQL and Valkey also have their own persistent claims. Database backup and media snapshot belong to the same recovery conversation.

Two authentication paths

The browser UI and automation need different ingress behaviour.

The UI passes through Authelia. The API route bypasses the browser-oriented forward-auth middleware and relies on DefectDojo API tokens. Traefik gives the API path a higher-priority match so CI receives JSON from DefectDojo instead of an HTML login page.

browser
  → Authelia
  → DefectDojo UI

CI importer
  → token-authenticated API route
  → DefectDojo API

Bypass does not mean unauthenticated. It means the API uses the authentication mechanism designed for automation.

Operating DefectDojo

I begin with the initializer and the main Django containers:

kubectl -n defectdojo get pods,job,svc,pvc,ingressroute
kubectl -n defectdojo logs job/defectdojo-initializer --tail=200
kubectl -n defectdojo logs deploy/defectdojo-django -c uwsgi --tail=200
kubectl -n defectdojo logs deploy/defectdojo-django -c nginx --tail=200

Then I inspect the background and stateful components:

kubectl -n defectdojo logs deploy/defectdojo-celery-worker --tail=100
kubectl -n defectdojo logs statefulset/defectdojo-postgresql --tail=100
kubectl -n defectdojo logs statefulset/defectdojo-valkey --tail=100

The first diagnostic split is:

first boot fails
  └── initializer, bootstrap inputs or database readiness

login or page requests restart
  └── uWSGI memory pressure or application settings

imports queue but do not finish
  └── Celery, Valkey or worker resources

CI receives a login page
  └── API route lost priority to Authelia-protected UI route

all pods Pending on local cluster
  └── dedicated node label or local volume binding

A platform with several processes needs several health checks. The Django page can load while background imports are stuck.

Backup and recovery

The useful backup set is:

PostgreSQL backup
+
media volume snapshot
+
runtime cryptographic and database inputs
+
Fleet render inputs
+
Git configuration

My recovery order is:

1. Restore placement prerequisites on the target cluster
2. Restore runtime and Fleet render inputs
3. Restore PostgreSQL and media state
4. Reconcile the correct environment
5. Confirm the initializer and all pods
6. Sign in through the UI
7. Submit or re-import a small known scan

The final import validates the API route, token, Django application, queue, database and finding workflow together.

Rollback

Chart changes, overlay changes and database migrations do not have identical rollback boundaries. A Git revert can restore the previous render. It cannot guarantee that a newer application has not changed persistent state.

For significant upgrades I keep the matching PostgreSQL backup and media snapshot until the UI and an ingestion cycle have been validated. I also change the smallest layer possible: chart version, values, post-render resources and import automation should not all move in one opaque change.

Things worth remembering

At the time of writing:

Role          central SecOps findings intake
Namespace     defectdojo
Runtime       Django, nginx, Celery, PostgreSQL, Valkey
Deployment    Fleet Helm + Kustomize post-render
Charts        1.9.29 and 1.9.37 by environment
UI auth       Authelia forward auth
API auth      DefectDojo token on a separate route
Recovery test login plus a real scan re-import

DefectDojo earns its complexity when it turns another scanner report into a finding with an owner, history and next action. The platform is healthy when that complete loop works—not merely when Django returns a page.

The monitoring path has two layers. Uptime Kuma performs the checks and keeps their history. A separate heartbeat job reads a deliberately limited view and pushes a static status snapshot to the public site.

homelab services
       │
       ▼
  Uptime Kuma
       │
       ▼
heartbeat job every 15 minutes
       │
       ▼
static public status JSON
       │
       ▼
mutana.xyz status page

The public page does not query the private monitor from a visitor’s browser. It serves the last pushed snapshot and shows when that snapshot has become stale.

That is a small architecture with a useful failure mode.

Why push the status?

Directly exposing the monitor’s API would couple the public site to the private cluster and reveal more than the site needs.

The pushed heartbeat creates a narrow contract:

service name
category
coarse state
uptime summary
generation timestamp

The public site can render that data without credentials or a live connection back into the homelab.

If the cluster, the monitor or the heartbeat job fails, the last file remains available. The timestamp eventually becomes stale and the site says so. Frozen green data is not repainted as current truth.

Current deployment

Uptime Kuma runs as a single StatefulSet in the monitoring namespace:

Traefik
   │
   ▼
Service on port 3001
   │
   ▼
Uptime Kuma 1.18.5
   │
   ▼
1 GiB persistent claim

The homelab volume uses Longhorn storage and mounts at /app/data. A development variant uses a different storage class but keeps the same workload shape.

The StatefulSet has one replica. This means the monitor has stable storage, not high availability. For a homelab monitor, a reliable history backup and an honest stale state are more valuable than pretending one in-cluster replica can observe every cluster-wide failure.

The observer problem

A monitor inside the platform it watches has a blind spot. If the whole cluster disappears, Uptime Kuma disappears with it.

The pushed status file softens that problem but does not eliminate it:

cluster healthy
  → current heartbeat arrives

cluster unhealthy
  → heartbeat stops
  → public site keeps last snapshot
  → snapshot ages into stale state

This is not equivalent to an independent external probe. It is an honest description of what the current architecture knows.

For services where independent outside-in availability matters, an external check would be the next layer. The current public status answers a narrower question: when did the homelab last report its own state?

Monitoring configuration is state

The /app/data volume contains more than a decorative history graph. It holds monitor definitions, notification relationships and application state.

If that volume is replaced with an empty claim, Kubernetes can show a healthy Uptime Kuma pod with zero useful monitors.

process health
  └── is Uptime Kuma running?

monitoring health
  └── are the expected checks running and producing recent results?

Both questions matter after a restore.

Operating Uptime Kuma

I start with the StatefulSet, route and volume:

kubectl -n monitoring get statefulset,pod,svc,pvc,ingressroute
kubectl -n monitoring logs statefulset/uptime-kuma --tail=200
kubectl -n monitoring describe statefulset uptime-kuma

Then I inspect the public heartbeat separately. If Uptime Kuma is healthy but the public timestamp is old, the failure is likely in the export or upload path rather than in monitoring itself.

The useful split is:

pod Pending
  └── Longhorn claim or scheduling

pod Ready, UI unavailable
  └── Service, Traefik or access control

UI healthy, monitors missing
  └── wrong or empty data volume

monitors current, public status stale
  └── heartbeat job, credential or upload path

many monitors fail together
  └── shared DNS, ingress, network or cluster dependency

A burst of simultaneous red checks often says more about the monitoring vantage point than about every application failing independently.

End-to-end validation

After a monitoring change, I validate the whole publication chain:

representative service responds
→ Uptime Kuma records the new check
→ heartbeat job exports a fresh snapshot
→ public timestamp advances
→ public state matches the private coarse state

I also test staleness in the site fixture. A status page that only looks correct while data is fresh has not implemented its most important failure state.

Backup and recovery

The useful recovery set is:

Uptime Kuma data volume
+
heartbeat export configuration
+
protected upload credentials
+
Kubernetes manifests
+
public stale-state behaviour

My recovery sequence is:

1. Restore the 1 GiB application data
2. Reconcile the StatefulSet, Service and route
3. Confirm the expected monitors and recent history
4. Restore the heartbeat job inputs
5. run a dry export
6. publish one fresh heartbeat
7. verify the public timestamp and service rows

The public status snapshot is replaceable. The monitor catalogue and its history are the state worth preserving.

Versioning

The deployment currently pins Uptime Kuma 1.18.5. That makes rollbacks reproducible, but the version also deserves a planned upgrade path rather than permanent stability by neglect.

Before changing it I snapshot /app/data, read the intervening migration notes, update one environment first and confirm both the UI and heartbeat contract. The public JSON should not have to change merely because the monitor’s internal schema changes.

Keeping that contract small is what decouples the site from the monitoring product.

Things worth remembering

At the time of writing:

Uptime Kuma    1.18.5
Workload       one StatefulSet
Namespace      monitoring
Storage        1 GiB Longhorn claim
Ingress        Traefik
Public status  pushed static snapshot every 15 minutes
Failure mode   explicit stale state
Recovery test  monitor → heartbeat → public page

Monitoring is not only the act of detecting failure. It is also the discipline of saying what the monitoring system cannot currently know.