Skip to main content

Production Hardening

Checklist for running PandoCore in production with confidence.

Docs / Production Hardening

Production Hardening

A default PandoCore install is production-safe, but a few tweaks are worth making before you turn on enforcement at scale. This page walks through the ones that actually matter.

1. Run in Monitor Mode First

The single most important hardening step is discipline, not config: run each workload in monitor for at least a week before flipping it to alert, and another week in alert before enforce. This is how you validate that the baseline captures your workload's real behavior without killing any pods.

New installs default to monitor. Don't set defaultMode=enforce on day one. See Operating Modes for the full escalation model.

2. Set Resource Limits

The sidecar ships with conservative defaults (25Mi request, 50Mi limit, 50m / 100m CPU), sized for a typical web app. High-throughput services, heavy runtimes, or workloads that churn a lot of processes may need more headroom.

# Raise sidecar limits cluster-wide
helm upgrade pando-webhook \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-webhook \
  --reuse-values \
  --set sidecar.resources.requests.memory=64Mi \
  --set sidecar.resources.limits.memory=128Mi \
  --set sidecar.resources.requests.cpu=100m \
  --set sidecar.resources.limits.cpu=200m

Check sidecar memory usage via kubectl top pod after a week in production. If you see OOM kills on the pando-sidecar container, raise the limit. If usage is steady at 30-40% of the limit, you're fine.

3. Exclude System Namespaces

By default the webhook already excludes kube-system and pando-system. Add any other namespaces you don't want the webhook touching: cert-manager, istio-system, your own infra namespaces, service meshes, CI runners, and so on.

helm upgrade pando-webhook \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-webhook \
  --reuse-values \
  --set-string excludedNamespaces="{kube-system,pando-system,cert-manager,istio-system,kube-public}"

Excluded namespaces are skipped entirely by the webhook's namespaceSelector. Pods in them can't be labeled for protection. If you need to protect a workload in one of these namespaces later, remove it from the exclusion list first.

4. Pin Sidecar Image Versions

The webhook's default sidecar tag is whatever shipped with the chart version you installed. For production, pin it explicitly so an accidental chart bump can't roll out a new sidecar without you knowing:

helm upgrade pando-webhook \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-webhook \
  --reuse-values \
  --set sidecar.image.tag=

Changing the tag and running helm upgrade is now a deliberate, reviewable action.

5. Understand the failurePolicy

The mutating webhook ships with failurePolicy: Ignore, fixed in the chart rather than exposed as a value. If the webhook is unavailable when a pod starts, the pod is admitted without sidecar injection and the cluster keeps running. This is deliberate: under Fail, a webhook outage would block pod creation cluster-wide, including workloads PandoCore doesn't protect and the components you'd need to recover. A monitoring component must not be able to take down the thing it monitors. The full rationale is written up in the repository's architecture docs (docs/architecture/kubernetes.md).

The cost of fail-open is that a webhook outage produces unprotected pods instead of errors. That cost is paid down with visibility, not by changing the policy:

The admission timeout is pinned explicitly to 10 seconds and tunable via webhookTimeoutSeconds (1 to 30). Because admission fails open, a timeout costs bounded pod-start latency during an outage; it never blocks a pod.

6. Run the Webhook with Multiple Replicas

A single webhook replica is a single point of failure for pod admission. The chart defaults to two; if you've overridden it down, restore it in production:

helm upgrade pando-webhook \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-webhook \
  --reuse-values \
  --set replicas=2

With more than one replica the chart also ships a PodDisruptionBudget (minAvailable: 1) and preferred pod anti-affinity across nodes, so a node drain can't evict both replicas at once and the scheduler spreads them when it can. This is essentially free and removes a class of failure modes (webhook pod rescheduling during node drain, OOM kill, etc) from your incident surface.

7. Back Up the License Signing State

The license Secret (pando-license in pando-system) is the only thing you need to reissue from scratch after a cluster wipe. Back it up along with the rest of your cluster state:

kubectl get secret pando-license -n pando-system -o yaml > pando-license-backup.yaml

Store this alongside your other Kubernetes secret backups (encrypted, access-controlled). You can always request a fresh license from the portal if you lose this, but having a copy makes disaster recovery a one-liner.

Profiles don't need backup

Each sidecar builds its operational profile on startup, so there's nothing stateful to back up. This is intentional: the profile should reflect the currently-running process, not a stale snapshot.

8. Wire Alerts to Somewhere Humans Watch

Detections without a consumer are just log entries. Point slackWebhookURL at a channel that your oncall actually watches, or forward evidence to your SIEM via PANDO_EVIDENCE_WEBHOOK_URL. See Evidence Format.

Test the path before an incident: trigger a detection manually (drop an unusual process into a protected pod, or lower PANDO_SENSITIVITY temporarily) and confirm the alert lands where you expect.

9. Use CRD Policy for Anything Non-Trivial

Configuring a handful of deployments via annotations is fine. Managing dozens of services across multiple namespaces via kubectl annotate is not. Commit a PandoCorePolicy resource to your GitOps repo so mode, sensitivity, and response chains are version-controlled alongside everything else.

See CRD Policy for the schema and examples.

10. Choose the Sidecar Security Posture

The injected sidecar runs with a hardened, least-privilege security context by default: non-root (UID 1000), all Linux capabilities dropped, no privilege escalation, and a read-only root filesystem. This is compatible with the restricted Pod Security Standard and GKE Autopilot, and it is the right default for most workloads.

Under this default, the sidecar reads a workload's behavior through the signals available to a same-user observer. When your application container runs as a different user than the sidecar (for example as root, or as a distinct service UID), a subset of the deeper behavioral signals cannot be read under least privilege and report as unavailable. Core detection still runs; the effect is reduced signal coverage on those workloads, not a blind sidecar.

To get full signal coverage on a workload whose application runs as a different user, opt that workload into one of two elevated postures with a pod annotation:

Posture Security context Trade-off
Default (no annotation) Non-root, all capabilities dropped, read-only rootfs Least privilege. Compatible with the restricted Pod Security Standard and GKE Autopilot. Full signal coverage for same-user workloads only.
pandocore.io/provenance-enforce: "true" Non-root, plus the CAP_SYS_PTRACE capability Full cross-user signal coverage without running as root. Satisfies the baseline Pod Security Standard, not restricted; not permitted on GKE Autopilot.
pandocore.io/full-proc-signals: "true" Runs as root, all other capabilities dropped Full cross-user signal coverage. Satisfies the baseline Pod Security Standard, not restricted; not permitted on GKE Autopilot. Use only where a workload's own user model requires it.

Prefer provenance-enforce when you need coverage: it stays non-root. Reach for full-proc-signals only when a workload's user model leaves no alternative.

Confirm which signals are live

Two Prometheus metrics report per-workload signal availability, so you can verify coverage before relying on it: pando_signal_unavailable (non-zero when a cross-user signal could not be read under the current posture) and pando_provenance_maps_readable. Opt a workload into an elevated posture only when these show a gap you actually need to close.

# Opt one workload into the non-root elevated posture, then restart it
kubectl annotate deployment your-app pandocore.io/provenance-enforce="true" --overwrite
kubectl rollout restart deployment/your-app

Production Hardening Checklist