Troubleshooting

Most issues fall into three buckets: the sidecar never injected, the sidecar injected but crash-looped, or the sidecar is running but producing more alerts than you expect. Each section below starts with how to diagnose, then lists the usual causes.

Sidecar Not Injecting

Symptom: you labeled a deployment and restarted it, but kubectl get pods still shows 1/1 instead of 2/2.

# Check that the webhook is registered and healthy
kubectl get mutatingwebhookconfiguration pando-webhook
kubectl get pods -n pando-system -l app=pando-webhook

# Look at webhook logs for the rejected/ignored admission request
kubectl logs -n pando-system -l app=pando-webhook --tail=100

# Confirm the label is actually on the pod template, not just the deployment
kubectl get deployment your-app -o jsonpath='{.spec.template.metadata.labels}'

Common causes

Label on deployment only, not pod template. kubectl label deployment adds the label to the deployment object. The webhook matches on pod labels. Use kubectl patch or edit the deployment directly if kubectl rollout restart didn't propagate. The label must be on spec.template.metadata.labels.
Namespace excluded by webhook. The webhook skips kube-system, pando-system, and any namespace in excludedNamespaces. Check the MutatingWebhookConfiguration's namespaceSelector.
Missing ServiceAccount or license Secret in target namespace. If you're protecting workloads outside pando-system, each target namespace needs its own pando-sidecar ServiceAccount and a copy of the license Secret. Without these, the webhook logs an injection error and admits the pod unmodified. See Getting Started for the copy commands.
Webhook cert expired. The webhook uses a self-signed cert that rotates via cert-manager or the install hook. If the webhook pod logs show TLS errors, reinstall or run helm upgrade.
failurePolicy=Ignore swallowed the error. Production installs ship with failurePolicy: Ignore so a broken webhook doesn't brick your cluster. That also means injection failures are silent. Always check webhook logs when something didn't inject.

Sidecar Crash-Looping

Symptom: the pod shows 2/2 briefly then drops to 1/2, or the pando-sidecar container reports CrashLoopBackOff.

# Logs from the previous crashed instance
kubectl logs <pod> -c pando-sidecar --previous

# Current logs
kubectl logs <pod> -c pando-sidecar

# Describe pod for events and exit codes
kubectl describe pod <pod>

License errors

The sidecar validates the license on startup. If validation fails, it exits immediately with a log line starting with license:.

license: signature invalid: the license was signed with a key that doesn't match the sidecar's embedded public key. Usually means you're running a sidecar image from a different environment (staging sidecar with prod license, or vice versa). Reinstall with the matching chart.
license: expired: the expiry field is in the past. Log in to the portal and request a renewal.
license: max_nodes exceeded: you're running more protected pods than your license allows. Either upgrade your plan in the portal or unlabel some deployments.
license: secret not found: the pando-license Secret is missing from the target namespace. See the per-namespace setup in Getting Started.

Process namespace errors

The sidecar attaches to the shared process namespace of its pod. If the pod doesn't have shareProcessNamespace: true set (the webhook adds this automatically), the sidecar can't see the target process and exits with process namespace not shared. This only happens if a PodSecurityPolicy or admission controller stripped the field after the webhook set it. Check your cluster's admission chain.

Memory pressure

The sidecar defaults to a 50Mi memory limit, sized for a typical web app. Heavy runtimes or high-throughput services can push it over. Raise the limit via Helm:

helm upgrade pando-webhook \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-webhook \
  --reuse-values \
  --set sidecar.resources.limits.memory=128Mi

Too Many Alerts (False Positives)

Symptom: the sidecar is tripping on normal workload behavior during batch jobs, traffic bursts, or other legitimate variance.

Before tuning, confirm the workload has completed its learning window. The first 30 minutes after pod start are the learning phase (tunable via PANDO_LEARNING_DURATION_SECONDS) and any detections during that window are meaningless. Check readiness:

kubectl exec <pod> -c pando-sidecar -- wget -qO- http://localhost:9090/readyz

If that returns 200, learning is complete. If 503, wait it out.

Tuning options, in order of preference

Raise PANDO_SENSITIVITY. The single knob for most workloads. Default is 2.0 (3.0 for JIT and database runtimes); try bumping by 0.5–1.0 for noisy services. Higher values make the sidecar more permissive without disabling detection entirely.
```
kubectl set env deployment/your-app -c pando-sidecar PANDO_SENSITIVITY=3.0
```
Trigger a re-learn. If the workload's traffic pattern changed (deploy, config rollout, customer onboarding), the old baseline no longer reflects normal. Re-learn rebuilds it from the current state:
```
kubectl exec <pod> -c pando-sidecar -- wget -qO- --post-data='' http://localhost:9090/relearn
```
Drop back to monitor mode. If you're still calibrating, there's no shame in running in monitor for another week. Evidence still flows to the portal; no pods get killed.
Contact support. If sensitivity and re-learning don't settle the workload, reach out. We'd rather debug it with you than let you chase it alone.

Noisy runtimes

The webhook auto-applies a higher sensitivity default for a handful of well-known runtime families whose legitimate behavior is inherently variable. If your workload runs a custom base image the detection didn't match, explicitly set PANDO_SENSITIVITY=3.0.

Sidecar Running but No Metrics

Symptom: the sidecar pod is ready, but /metrics returns nothing or the portal dashboard shows no activity.

# Hit the metrics endpoint directly
kubectl exec <pod> -c pando-sidecar -- wget -qO- http://localhost:9090/metrics

# Confirm the portal URL the sidecar is talking to
kubectl exec <pod> -c pando-sidecar -- env | grep PANDO_PORTAL_URL

Metrics endpoint empty: the sidecar hasn't completed learning yet, or the target process hasn't been sampled. Wait for /readyz to return 200.
Portal dashboard empty: the sidecar posts evidence + heartbeats to PANDO_PORTAL_URL. If this is unset or pointing at the wrong environment, the portal won't receive anything. Re-run the Helm install with the correct portal URL.
Egress blocked: your cluster's NetworkPolicy or firewall may be blocking outbound calls to the portal. Check sidecar logs for HTTP errors.

Pod Terminated Unexpectedly

Symptom: a protected pod was killed and you need to understand why.

Every termination ships with a structured evidence envelope to the portal. Start there:

Open the portal dashboard, find the pod in the Events tab, and read the detection reason and severity.
If you don't have portal access at the moment, the sidecar's own logs include the full decision path. The logs are still available on the node's container runtime for a short window after termination.
Check whether the pod was in enforce mode. If you expected it to only alert, confirm your mode setting with kubectl get deployment your-app -o jsonpath='{.spec.template.metadata.annotations}'. Deployment annotations override the cluster default.

If the termination was clearly a false positive, see the false positives section above. If you need to pause enforcement immediately while you investigate, see Emergency Procedures.

Getting Help

If none of the above fixes your issue, collect the following before reaching out:

kubectl version and the sidecar image tag
kubectl logs <pod> -c pando-sidecar (last 200 lines)
kubectl describe pod <pod>
kubectl get mutatingwebhookconfiguration pando-webhook -o yaml
Relevant portal evidence envelope if there was a termination

Send to support or reply to your onboarding email thread.