Production Hardening

Checklist for running PandoCore in production with confidence.

Docs / Production Hardening

Production Hardening

A default PandoCore install is production-safe, but a few tweaks are worth making before you turn on enforcement at scale. This page walks through the ones that actually matter.

1. Run in Monitor Mode First

The single most important hardening step is discipline, not config: run each workload in monitor for at least a week before flipping it to alert, and another week in alert before enforce. This is how you validate that the baseline captures your workload's real behavior without killing any pods.

New installs default to monitor. Don't set defaultMode=enforce on day one. See Operating Modes for the full escalation model.

2. Set Resource Limits

The sidecar ships with conservative defaults (25Mi request, 50Mi limit, 50m / 100m CPU), sized for a typical web app. High-throughput services, heavy runtimes, or workloads that churn a lot of processes may need more headroom.

# Raise sidecar limits cluster-wide
helm upgrade pando-webhook \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-webhook \
  --reuse-values \
  --set sidecar.resources.requests.memory=64Mi \
  --set sidecar.resources.limits.memory=128Mi \
  --set sidecar.resources.requests.cpu=100m \
  --set sidecar.resources.limits.cpu=200m

Check sidecar memory usage via kubectl top pod after a week in production. If you see OOM kills on the pando-sidecar container, raise the limit. If usage is steady at 30-40% of the limit, you're fine.

3. Exclude System Namespaces

By default the webhook already excludes kube-system and pando-system. Add any other namespaces you don't want the webhook touching: cert-manager, istio-system, your own infra namespaces, service meshes, CI runners, and so on.

helm upgrade pando-webhook \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-webhook \
  --reuse-values \
  --set-string excludedNamespaces="{kube-system,pando-system,cert-manager,istio-system,kube-public}"

Excluded namespaces are skipped entirely by the webhook's namespaceSelector. Pods in them can't be labeled for protection. If you need to protect a workload in one of these namespaces later, remove it from the exclusion list first.

4. Pin Sidecar Image Versions

The webhook's default sidecar tag is whatever shipped with the chart version you installed. For production, pin it explicitly so an accidental chart bump can't roll out a new sidecar without you knowing:

helm upgrade pando-webhook \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-webhook \
  --reuse-values \
  --set sidecar.image.tag=v0.5.2

Changing the tag and running helm upgrade is now a deliberate, reviewable action.

5. Choose a failurePolicy

The mutating webhook's failurePolicy decides what happens if the webhook itself is unavailable when a pod tries to start. Two options:

helm upgrade pando-webhook \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-webhook \
  --reuse-values \
  --set failurePolicy=Fail

Most teams stay on Ignore in production and rely on the portal dashboard to flag any pods that started without a sidecar. That's the right default unless you have a compliance reason to force-fail.

6. Run the Webhook with Multiple Replicas

A single webhook replica is a single point of failure for pod admission. Run at least two in production:

helm upgrade pando-webhook \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-webhook \
  --reuse-values \
  --set webhook.replicaCount=2

This is essentially free and removes a class of failure modes (webhook pod rescheduling during node drain, OOM kill, etc) from your incident surface.

7. Back Up the License Signing State

The license Secret (pando-license in pando-system) is the only thing you need to reissue from scratch after a cluster wipe. Back it up along with the rest of your cluster state:

kubectl get secret pando-license -n pando-system -o yaml > pando-license-backup.yaml

Store this alongside your other Kubernetes secret backups (encrypted, access-controlled). You can always request a fresh license from the portal if you lose this, but having a copy makes disaster recovery a one-liner.

Profiles don't need backup

Each sidecar builds its operational profile on startup, so there's nothing stateful to back up. This is intentional: the profile should reflect the currently-running process, not a stale snapshot.

8. Wire Alerts to Somewhere Humans Watch

Detections without a consumer are just log entries. Point slackWebhookURL at a channel that your oncall actually watches, or forward evidence to your SIEM via PANDO_EVIDENCE_WEBHOOK_URL. See Evidence Format.

Test the path before an incident: trigger a detection manually (drop an unusual process into a protected pod, or lower PANDO_SENSITIVITY temporarily) and confirm the alert lands where you expect.

9. Use CRD Policy for Anything Non-Trivial

Configuring a handful of deployments via annotations is fine. Managing dozens of services across multiple namespaces via kubectl annotate is not. Commit a PandoCorePolicy resource to your GitOps repo so mode, sensitivity, and response chains are version-controlled alongside everything else.

See CRD Policy for the schema and examples.

Production Hardening Checklist