Skip to main content
Docs / Production Hardening

Production Hardening

Checklist for running PandoCore in production with confidence.

A default PandoCore install is production-safe, but a few things are worth settling before you let it respond to anything at scale. This page walks through the ones that actually matter.

1. Spend Time On A Free License First

The most important hardening step is discipline rather than configuration. A free license detects and alerts and never contains, on as many nodes as you want to install it on, so the validation period is not a trial mode you have to remember to leave: it is simply the tier. Run it across the fleet, read what it reports, and decide from evidence whether the detections are ones you would have wanted acted on.

The response ceiling is a property of the license rather than a chart value, so nothing in your cluster has to change when you decide to raise it. See Operating Modes for the ladder.

2. Size The Per-Node Request

The sensor requests 250m CPU and 256Mi per node and is limited to 2 cores and 1Gi. That is one pod per node rather than one per workload, so the figure to plan against is per node and does not grow with your deployment count.

Measured warm and enforcing on the shipping build, across two runs: a 130.4 mCPU p95 with a 106.9 MiB working set over a 24-hour warm window, and 127.3 mCPU with 110.6 MiB over a 34-hour run. Both held a flat working-set slope, with no CFS throttling and no OOMKills. The request is sized for the 70-pod cap rather than for the engine counts these runs carried.

helm upgrade pando-node-sensor ... \
  --set resources.requests.cpu=250m \
  --set resources.requests.memory=256Mi

Check actual usage with kubectl top pod -n pando-system after a week. Raise the request before shortening the sampling cadence, never after: at the 250 ms floor the same build costs roughly four times as much CPU as at the 1000 ms default, and CPU is shared by request under contention.

3. Decide What Is Out Of Scope

Eleven namespaces are excluded by default, covering kube-system and the GKE managed namespaces. Add your own infrastructure namespaces to that list: cert-manager, service meshes, CI runners, and anything else whose behaviour you do not want profiled.

helm upgrade pando-node-sensor ... \
  --set 'excludedNamespaces={kube-system,kube-public,kube-node-lease,cert-manager,istio-system}'
Two sharp edges here

excludedNamespaces: [] does not mean "exclude nothing". An empty list is read as unconfigured and falls back to the built-in eleven. If you override the list, write out the full set you want, including the defaults you intend to keep.

Never list pando-system in includedNamespaces. That opt-in is read before the exclusion rules and overrides the own-component exclusion as well as the namespace list, so it puts PandoCore's own pods back under monitoring. An early soak run in that configuration took 67 false positives in six minutes, 34 of them from the controller pod.

Four classes stay excluded whatever the configuration says, and includedNamespaces does not override them: the sensor's own pod, hostPID pods, hostNetwork pods, and any pod carrying pandocore.io/protect: "false". An operator opting a namespace in has not acquired the pod owner's decision.

4. Pin The Chart Version

Both charts resolve their image tag from the chart version, so pinning the chart pins what runs. Installing an OCI chart without --version fails outright rather than drifting to the newest release, which is the behaviour you want here; keep the version explicit in whatever applies your releases.

helm upgrade pando-node-sensor \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-node-sensor \
  --version 2.0.0 \
  --reuse-values

5. Run The Controller With Two Replicas

The sensor is a DaemonSet and needs no replica planning: one pod per node is the topology. The controller is the component with an availability question, because a single replica means a node drain or a rolling update is a window in which sensors have nowhere to deliver evidence.

The chart defaults to two. Below two the PodDisruptionBudget does not render at all, so if you have overridden it down, restore it.

helm upgrade pando-controller ... --set replicas=2

6. Back Up The License Secret

The license Secret (pando-license in pando-system) is the one thing worth keeping a copy of. You can always reissue from the portal, but a copy makes recovery a one-liner.

kubectl get secret pando-license -n pando-system -o yaml > pando-license-backup.yaml
Sensor state does not need backup

Each sensor keeps working state under /var/lib/pando on its node, as a hostPath rather than a PersistentVolumeClaim, deliberately: a restored state directory is refused as older than the pod reading it. Profiles rebuild on start, which is the intent, since a profile should describe the workload running now rather than a stale snapshot. Evidence durability is the controller's job, not the node's.

7. Point Evidence Somewhere Humans Watch

Detections without a consumer are log entries. The controller owns every outbound destination, so SIEM, Slack and webhook delivery are configured once there rather than per workload, and no workload pod ever holds a third-party credential. See Controller and Evidence Format.

Test the path before you need it, and confirm an alert lands where you expect.

8. Alert On The Sensor Itself

The chart ships PrometheusRules for the conditions a sensor cannot report on its own behalf. The one to wire first is coverage: pando_node_sensor_pods_unmonitored{reason="cap"} is a gauge of pods that got no engine because the node hit its cap, and those are pods you wanted watched. It is not a capacity hint.

The rules also cover the series going absent, a refusal at startup, and a cadence that has stretched past its configured interval. Absence matters most, because it is the one condition no self-report can distinguish from healthy, which is why it is alerted on from outside.

On GKE managed Prometheus these rules do nothing

Managed Prometheus does not honour the chart's scrape annotations, so metrics.podMonitoring.enabled=true is what makes it scrape at all. It also reads neither the alert-rules ConfigMap nor a PrometheusRule, so the rules ship and sit inert until they are converted to its own Rules resource.

Port the coverage and absence rules across by hand if you run it. Those two are what tell you a node has stopped being watched, and this is the configuration in which you would least want to assume the chart had covered you.

Select on the component label

The sensor's own pod carries pandocore.io/protect: "false". A Prometheus job written against protect=true matches nothing and reports a clean scrape of nothing, which is indistinguishable from healthy on a dashboard. Select on app.kubernetes.io/name=pando-node-sensor.

9. Use CRD Policy For Anything Non-Trivial

Configuring a handful of deployments by annotation is fine. Managing dozens of services across namespaces with kubectl annotate is not. Commit a PandoCorePolicy resource to your GitOps repo so mode, sensitivity and alert destinations are version-controlled alongside everything else. See CRD Policy.

Per-namespace integrations declared in a policy reach the controller through the destinations mirror, which the sensor chart installs by default. Setting mirror.enabled=false leaves them inert.

10. Understand What The Sensor Runs As

This is the section to read before you install, and it is a real increase in node surface compared with the deprecated sidecar. The sensor runs as a hostPID DaemonSet, as uid 0, with CAP_SYS_PTRACE, mounting the node's /proc read-only and a host path for its own state. The node's PID namespace is the whole point of the topology: the sensor sits outside every pod it watches, so that namespace is the only way it sees their processes.

Not compatible with restricted Pod Security, GKE Autopilot or EKS Fargate

The deprecated sidecar ran unprivileged and satisfied the restricted Pod Security Standard. The sensor does not, and there is no install path on GKE Autopilot or EKS Fargate, both of which forbid hostPID. If you need those, the node sensor is not deployable for you today. Standard GKE is the configuration this release was validated on.

On uid 0 specifically: it is load-bearing rather than inherited. CAP_SYS_PTRACE becomes effective through the uid-0 path, and the alternative posture is a file capability raised in-process with allowPrivilegeEscalation: true, which is more escalation surface than root with escalation switched off. Of the two postures that work, this is the narrower one.

What bounds it instead is the grant. The sensor's ClusterRole holds pods get/list/watch, events create, networkpolicies create/delete/get, the three workload-owner kinds get, policies list, services list, and namespaces get scoped to kube-system. pods: patch, pods: delete and endpoints: get are withheld by name, each with its reason recorded in the chart. The cluster-wide secrets get that per-namespace integrations need does not sit on this pod at all: the destinations mirror is a separate unprivileged Deployment precisely so that grant is not held by a hostPID pod running as uid 0 on every node.

11. Three Annotations The Sensor Cannot Honour

If you are migrating from the webhook, these three do nothing now:

All three were applied at admission, by rewriting a pod before it was created: the first two mutated its securityContext and the third added a hostPath volume and mount. A sensor is outside the pod and runs after it already exists, so there is no point at which it could apply any of them. The sensor warns once per pod carrying one, and the pod is still monitored.

The signal coverage the first two used to buy is not lost, because the sensor already holds CAP_SYS_PTRACE on the node. They were a per-pod posture choice under a topology where each pod carried its own observer; there is no longer a per-pod posture to choose.

On SELinux and Bottlerocket

seLinuxLevel is empty by default and is ignored where SELinux is not the enforcing module for containers, which covers GKE's Container-Optimized OS (AppArmor) and EKS AL2023 (SELinux permissive). On Bottlerocket it is required for the file-age routes to work at all: every container runs as container_t with its own category pair, so without a clearance the sensor's pair does not dominate the target's and the path walk is refused.

Production Hardening Checklist