CrowdSec on Kubernetes: Turning Edge Noise into Decisions.

CrowdSec sits near the noisy edge of my homelab and turns repeated hostile behaviour into decisions that other components can use.

The piece I run in Kubernetes is centred on the local API. Agents and security components can report observations, the API keeps shared state, and consumers can ask whether an address should be allowed or blocked.

logs and security agents
          │
          ▼
     CrowdSec LAPI
       │       │
       │       └──► decision consumers
       ▼
 persistent state

There is also a Metabase container in the current pod for exploring the data. It makes the Deployment heavier than the CrowdSec API alone, but it gives the stored decisions a human-readable view.

Why centralise decisions?

Each Internet-facing component can see only part of an attack. A reverse proxy sees requests. A host agent sees authentication failures. A bouncer enforces a decision but may not know how that decision was reached.

The local API provides a meeting point:

observation
→ scenario matched
→ decision stored
→ bouncer queries decision
→ traffic allowed or denied

This is useful because enforcement remains close to the edge while detection can combine signals over time.

It is not a substitute for authentication, patching or firewall policy. CrowdSec is another layer, particularly good at turning repetitive background noise into something actionable.

Current deployment

The current Kubernetes resources are compact and somewhat legacy in layout: a persistent volume claim and a Deployment rather than a full base-and-overlay tree.

The Deployment runs:

CrowdSec       v1.4.1
Metabase       v0.44.4
init helper    prepares shared state

The persistent claim is 500 MiB on Longhorn storage. CrowdSec itself requests 50m CPU and 50 MiB memory, with limits of 150m CPU and 100 MiB memory. Metabase requests the same small baseline but has a much larger memory ceiling of roughly 1.5 GiB.

That resource difference is a useful troubleshooting clue. If the pod is killed for memory, the analytics side may be the cause even when CrowdSec appears in the pod name.

State and trust

The API stores decisions, enrolment information and operational context. Losing it does not remove the underlying log sources, but it does interrupt the shared memory of the security layer.

Clients also need credentials to report or consume decisions. I keep those credentials out of the public manifests and out of this article.

The recovery set is therefore:

LAPI persistent state
+
agent and bouncer credentials
+
Deployment configuration
+
the enforcement integration

Restoring the API without reconnecting a bouncer produces a healthy dashboard and no protection. Restoring a bouncer credential without the API state produces a client with nothing useful to ask.

Operating CrowdSec

I start with the pod, Service and persistent claim:

kubectl get deploy,pod,svc,pvc -l app=crowdsec
kubectl logs deploy/crowdsec-lapi -c crowdsec --tail=200
kubectl logs deploy/crowdsec-lapi -c metabase --tail=100

Because the two main containers share a pod, I always name the container when reading logs. Otherwise it is easy to investigate the analytics UI while the local API is the component that agents actually depend on.

The useful failure map is:

agents cannot report
  └── LAPI Service, credentials or network path

bouncers receive no decisions
  └── enrolment, API state or enforcement integration

pod restarts under memory pressure
  └── inspect each container, especially Metabase

dashboard unavailable, enforcement works
  └── analytics failure rather than security-path failure

everything healthy, attacks pass through
  └── scenario coverage or bouncer placement

The last case is the most important. CrowdSec can be perfectly healthy as software and irrelevant as a control if the enforcement point is not actually consulting it.

End-to-end validation

I prefer a safe synthetic validation over waiting for hostile traffic.

The test should prove three separate things:

1. an agent can reach the local API
2. a known test decision appears in the API
3. the intended bouncer observes that decision

I do not use a production client address for the test. Security automation is a poor place for an accidental self-lockout.

After validation, I remove the test decision and confirm normal access.

Backup and recovery

The persistent data is small enough that frequent snapshots are inexpensive. The less visible task is preserving the credential and enrolment relationships around it.

My recovery sequence is:

1. Restore the PVC state
2. Recreate the Deployment and Service
3. Restore agent and bouncer trust material
4. Confirm agents are reporting
5. Confirm consumers can query decisions
6. Run a safe end-to-end decision test

If the API state cannot be restored, re-enrolling clients is possible, but it is a rebuild rather than a transparent recovery. I document that distinction before an incident.

Upgrades and age

The pinned versions in this deployment are old enough that an upgrade deserves its own planned change. Jumping several releases can affect data formats, collections, scenarios and client compatibility.

The safe approach is:

snapshot
→ read release and migration notes
→ update CrowdSec separately from Metabase
→ validate agents
→ validate bouncers
→ keep a rollback image and data snapshot

Separating the two application upgrades keeps a CrowdSec problem from being confused with a Metabase problem.

Things worth remembering

At the time of writing:

Core            CrowdSec LAPI 1.4.1
Analytics       Metabase 0.44.4
Workload        multi-container Deployment
Storage         500 MiB Longhorn claim
Purpose         shared observations and decisions
Recovery test   agent → LAPI → bouncer

The value of CrowdSec is not the number of decisions stored. It is whether a useful signal reaches the correct enforcement point before the next request does.