BackupPC on Kubernetes: Backing Up the Machines Outside the Cluster.

BackupPC is the service that reaches beyond the cluster.

Most Kubernetes applications are concerned with their own volumes. BackupPC is concerned with the machines around Kubernetes: workstations, servers and other systems whose useful data does not live in a pod at all.

machines on the network
         │
         ▼
     BackupPC
       │   │
       │   └──► configuration state
       ▼
   backup pool

Running a backup server as a Kubernetes Deployment is convenient, but it introduces a circular question: what backs up the backup service?

Why BackupPC?

My estate is mixed. Some data lives in Kubernetes volumes. Some lives on ordinary Linux hosts. Some machines are not always online. BackupPC provides a central schedule and history for the latter group.

It is not the only layer of protection. Storage snapshots protect Kubernetes state; Git protects manifests; application-aware dumps protect databases. BackupPC covers filesystems outside those boundaries.

Git                desired configuration
volume snapshots   Kubernetes persistent state
database dumps     application-consistent state
BackupPC           files on machines outside the cluster

The overlap is intentional. A single backup mechanism rarely matches every kind of data.

Deployment shape

The current workload is a single Deployment in the prod namespace using the adferrand/backuppc image. A Kubernetes Service exposes the web application internally, and Traefik provides the controlled browser route.

The manifest also defines separate persistent areas for BackupPC’s configuration and working state. The declared Kubernetes claims are small because the large backup pool follows its own storage path and capacity planning.

That distinction matters:

BackupPC configuration
  └── schedules, hosts and application state

backup pool
  └── the actual retained data, usually much larger

Restoring only one half produces either a server with no history or a pile of backup data the server does not understand.

Access to client machines

A backup job crosses several trust boundaries. BackupPC needs a network path to a client, an authentication method, permission to read the intended files, and enough time to complete.

I keep those concerns separate when debugging:

can the pod resolve and reach the client?
can it authenticate?
can it read the selected paths?
can it write to the backup pool?
does the job finish inside its window?

A failed backup is not always a BackupPC application problem. A sleeping workstation, rotated key, changed filesystem permission or full storage target can produce the same red result in the UI.

Credentials and client inventories are not part of this public log. They belong in the protected runtime configuration and in the recovery documentation kept with the estate.

Operating BackupPC

My first Kubernetes checks are:

kubectl -n prod get deploy,pod,svc,pvc,ingressroute
kubectl -n prod logs deploy/backuppc --tail=200
kubectl -n prod describe deploy backuppc

Then I check storage attachment and capacity. A Ready web interface is not useful if the pool is read-only or full.

kubectl -n prod get pvc
kubectl -n prod describe pvc

For a single failing client, I avoid restarting the whole service. I compare the last successful run, test network reachability from the pod and verify that the client-side access method still works.

The diagnostic split is:

all clients fail
  └── server, shared credential, network or backup-pool issue

one client fails
  └── client availability, key, permissions or path change

jobs run but retention is wrong
  └── scheduling or pool configuration

UI works but restores fail
  └── catalogue and retained data no longer agree

That final category is why restore testing matters more than a row of successful job timestamps.

Backup verification

I judge BackupPC by restores, not backups.

A lightweight verification selects a small file from a recent backup, restores it to a safe temporary location and compares the content. A deeper exercise restores a directory tree with permissions and timestamps, then records how long the process took.

backup completed
→ file visible in catalogue
→ restore requested
→ bytes recovered
→ content and metadata checked

This tests the catalogue, the pool, permissions and the operator path at once.

Recovering BackupPC itself

The useful recovery set includes:

Kubernetes manifests
+
BackupPC configuration and catalogue
+
protected access credentials
+
backup pool

My recovery order is:

1. Make the backup storage available without modifying it
2. Restore BackupPC configuration and catalogue state
3. Restore access credentials
4. Reconcile the Deployment and route
5. Confirm old backup generations are visible
6. Restore a known file
7. Only then resume scheduled writes

Mounting the pool read-only during the first inspection is a useful precaution. It prevents a misconfigured fresh instance from treating old data as disposable workspace.

Rollback and upgrades

The image is not currently pinned to an explicit version in the Deployment. That makes the running image digest part of any serious incident record.

Before an upgrade I preserve the catalogue and configuration, note the current digest and verify a recent restore. If the upgraded service cannot read its existing pool, reverting the YAML alone may fetch the same moving image again.

Pinning the application version would make this much less ambiguous.

Things worth remembering

At the time of writing:

Workload       single Deployment
Namespace      prod
Ingress        Traefik
State          configuration, catalogue and backup pool
Clients        machines outside Kubernetes
Weak point     an untested restore path
Recovery test  restore and compare a real file

A backup system is infrastructure whose output may remain unused for months. That makes routine evidence unusually important. The most reassuring green icon is a file that came back.