Troubleshooting

Common issues and how to diagnose them.

Most issues fall into four buckets: a workload is not being watched, a sensor is refusing to start, a sensor is running but producing more alerts than you expect, or a pod was contained and you need to know why. Each section starts with how to diagnose, then lists the usual causes.

# One sensor per node, all Ready?
kubectl get pods -n pando-system -l app.kubernetes.io/name=pando-node-sensor -o wide

# What does the sensor on this node think it is doing?
kubectl exec -n pando-system <sensor-pod> -- wget -qO- http://localhost:9090/status

A Workload Is Not Being Watched

Symptom: a pod is running, but it never appears in detections and no engine was started for it.

# Is there a sensor on that pod's node at all?
kubectl get pods -n pando-system -o wide \
  -l app.kubernetes.io/name=pando-node-sensor | grep <node>

# How many engines is that node's sensor running, and is it shedding?
kubectl exec -n pando-system <sensor-pod> -- wget -qO- http://localhost:9090/status

# Is the pod opted out?
kubectl get pod <pod> -o jsonpath='{.metadata.labels}'

Common causes

• The node is over the engine cap. Each sensor runs at most maxEngines monitored pods (70 by default). Pods beyond it get no engine, and they are shed in a stable order, so the same pods stay unwatched across restarts rather than rotating. Check pando_node_sensor_pods_unmonitored{reason="cap"}. This is the most likely cause on a dense node, and it is a coverage gap rather than a capacity hint.
• The pod opted out. pandocore.io/protect: "false" excludes a pod, and nothing an operator sets overrides it. Note the comparison is byte-for-byte: "False" or "0" do not opt out, so an intended opt-out may also be silently inactive.
• The namespace is excluded. Eleven are by default, covering kube-system and the GKE managed namespaces. Add the namespace to includedNamespaces to protect it anyway; emptying excludedNamespaces does not work, because an empty list is read as unconfigured and falls back to the built-in eleven.
• The pod uses hostPID or hostNetwork. Both are excluded and neither is overridable. A pod whose process view or socket table is the node's cannot have a pod-scoped baseline, so watching it on request would answer the question wrongly rather than narrowly.
• No sensor is scheduled on that node. The DaemonSet tolerates everything by default, so this usually means a nodeSelector was set, or the node cannot allocate the sensor's CPU request. The install preflight refuses the latter case at install time; if it was bypassed with preflight.nodeCapacity=false, the sensor pod will sit Pending instead.

A Sensor Is Refusing Or Crash-Looping

Symptom: the sensor pod is running but /readyz returns 503 indefinitely, or the pod is in CrashLoopBackOff.

# Current and previous logs
kubectl logs -n pando-system <sensor-pod>
kubectl logs -n pando-system <sensor-pod> --previous

# Events and exit codes
kubectl describe pod -n pando-system <sensor-pod>

A refusal names one of eight reasons, which is also the label on the refusal metric: config, licence, siem, node_name, state_dir, cluster_limit, revoked or channel. A refusal is a configuration error and will not clear on its own.

• licence: the license failed validation. A signature mismatch usually means the image and the license are from different environments. An expired license needs renewal from the portal. Note the key is not trimmed, so a Secret created from a file with a trailing newline fails validation; create it with --from-literal.
• config: a value was rejected at startup rather than silently clamped. The most common is sampleIntervalMs below its 250 ms floor, which is refused by name. There is no 100 ms option.
• channel: the sensor could not establish its evidence channel to the controller. Check the controller is installed and running, and that its Service publishes the evidence port the sensor expects. The sensor composes that URL from the controller's parts rather than taking a hand-written one, so a mismatch is usually a namespace or service-name override.
• state_dir: /var/lib/pando could not be created or written on the node.
• node_name: the sensor could not determine which node it is on, so it cannot attribute evidence or be metered correctly.
• cluster_limit or revoked: a portal-side verdict about this cluster or license. Check the portal.

Memory or CPU pressure

The sensor requests 250m CPU and 256Mi per node, and is limited to 2 cores and 1Gi. If you have shortened the sampling cadence, raise the request to match: at the 250 ms floor the same build costs roughly four times as much CPU as at the 1000 ms default, and CPU is shared by request under contention.

helm upgrade pando-node-sensor ... \
  --set resources.requests.cpu=500m \
  --set resources.limits.memory=1Gi

Too Many Alerts

Symptom: detections firing on normal workload behaviour during batch jobs, traffic bursts or other legitimate variance.

First, confirm the workload finished learning. A profile takes 30 minutes and at least 900 samples, whichever is later, and detections before that are not meaningful. Behavioural checks that rest on longer baselines keep warming for hours beyond it, so a workload that has been watched for forty minutes is not yet fully calibrated.

kubectl exec -n pando-system <sensor-pod> -- wget -qO- http://localhost:9090/readyz

Tuning options, in order of preference

1. Loosen sensitivity in a PandoCorePolicy for that namespace. low, medium or high, where a higher preset is a tighter gate. This is the right lever because it is per-namespace and version-controlled. See CRD Policy.

   kubectl patch pandocorepolicy default -n production \
     --type merge -p '{"spec":{"sensitivity":"low"}}'

2. Put the workload in monitor. It keeps being watched and keeps producing evidence, and stops alerting, which is the right move while you calibrate rather than switching anything off.

   kubectl annotate deployment your-app pandocore.io/mode=monitor --overwrite
   kubectl rollout restart deployment/your-app

3. Restart the workload to re-profile it. If its traffic pattern genuinely changed, the old profile no longer describes normal. There is no re-learn endpoint on the sensor; a workload re-profiles when its pods restart.
4. Contact support. If the preset and a re-profile do not settle it, reach out. We would rather debug it with you.

Sensor Running But No Data

Symptom: the sensor is ready, but /metrics looks empty or the portal dashboard shows no activity.

# Metrics directly off the sensor
kubectl exec -n pando-system <sensor-pod> -- wget -qO- http://localhost:9090/metrics

# Is the controller receiving anything?
kubectl logs -n pando-system -l app.kubernetes.io/name=pando-controller --tail=100

• Your Prometheus job matches nothing. This is the most common cause and the most misleading, because it reports a clean scrape of nothing, which looks identical to healthy. The sensor's own pod carries pandocore.io/protect: "false", so a job selecting on protect=true finds no targets. Select on app.kubernetes.io/name=pando-node-sensor.
• Learning has not finished. Wait for /readyz to return 200.
• Evidence is reaching the controller but not leaving it. The controller owns every outbound destination, so an empty SIEM or Slack channel with healthy sensors is a controller configuration question rather than a sensor one. See Controller.
• Egress is blocked. Your cluster's NetworkPolicy or firewall may be blocking the controller's outbound calls. Check the controller's logs for HTTP errors.
• Per-namespace integrations are inert. Integrations declared in a PandoCorePolicy reach the controller through the destinations mirror. If mirror.enabled=false, nothing writes the destinations Secret and those integrations do nothing. The controller's own destinations are unaffected.

A Pod Was Contained Unexpectedly

Symptom: a workload lost network egress and you need to understand why.

Containment is a Kubernetes NetworkPolicy, so it is visible and reversible with ordinary tooling:

# The policy PandoCore wrote
kubectl get networkpolicy -n <namespace>

# The detection that caused it
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod>

1. Read the evidence record in the portal's Events tab for the detection reason and severity.
2. Check the workload's mode. If you expected it only to alert, confirm with kubectl get deployment your-app -o jsonpath='{.spec.template.metadata.annotations}'. Remember the annotation is read when the sensor starts watching a pod, so a recent change may not have applied to pods already running.
3. If the behaviour subsides, the policy is removed automatically. You do not need to delete it by hand, and deleting it does not resolve the underlying detection.

If it was clearly a false positive, see Too Many Alerts above. To stop PandoCore responding immediately while you investigate, see Emergency Procedures.

Getting Help

Collect the following before reaching out:

• kubectl version, and the chart versions from helm list -n pando-system
• kubectl logs -n pando-system <sensor-pod> (last 200 lines)
• kubectl exec -n pando-system <sensor-pod> -- wget -qO- http://localhost:9090/status
• kubectl describe pod -n pando-system <sensor-pod>
• The value of pando_node_sensor_pods_unmonitored if coverage is the question
• The relevant portal evidence record if a pod was contained

Send to support or reply to your onboarding email thread.