Emergency Procedures

How to quickly disable or remove PandoCore when needed.

How to stop PandoCore acting, quickly, at whatever scope the situation needs. Nothing here requires a support ticket or a license change, and nothing here restarts your workloads.

Containment is a NetworkPolicy, so it is reversible with ordinary tooling

The hardest automatic action on the supported install is a Kubernetes NetworkPolicy. Nothing deletes your pods: the sensor's ClusterRole withholds pods: delete by name and the sensor caps its own ceiling in code. So the worst case you are recovering from is a workload that lost egress, which you can see with kubectl get networkpolicy and undo in one command.

Quick Reference

Scope	Command	Effect
Release one contained pod	`kubectl delete networkpolicy <name> -n <ns>`	Egress restored at once
Stop watching one pod	`kubectl label pod <pod> pandocore.io/protect=false --overwrite`	Engine retired on the next scan. No restart
Stop one workload responding	`kubectl annotate deployment <name> pandocore.io/mode=monitor --overwrite`	Keeps watching, stops acting. Needs a restart
Stop a whole namespace	`helm upgrade` with the namespace in `excludedNamespaces`	Everything in it unwatched
Stop the whole cluster	`kubectl delete daemonset -n pando-system -l app.kubernetes.io/name=pando-node-sensor`	Every sensor gone. Helm release intact
Remove entirely	`helm uninstall pando-node-sensor pando-controller -n pando-system`	Both charts removed

Immediate: Release A Contained Pod

If a workload has lost its network and you need it back now:

# Find the policy PandoCore wrote. It carries both of these labels
kubectl get networkpolicy -n <namespace> \
  -l app.kubernetes.io/managed-by=pandocore

# Or go straight to the one for a given pod
kubectl get networkpolicy -n <namespace> \
  -l pandocore.io/isolation=<pod>

# Remove it
kubectl delete networkpolicy <name> -n <namespace>

Each policy carries an owner reference to the pod it contains, so it is garbage-collected with that pod: if you delete the pod instead, the policy goes with it and no orphan is left behind.

This treats the symptom, not the cause

Deleting the policy restores egress but does not resolve the detection, so if the behaviour is still present the pod can be contained again. Pair it with one of the scope changes below, which is what stops it recurring. If the behaviour has genuinely subsided, PandoCore removes the policy itself and you do not need this at all.

Stop Watching One Pod, With No Restart

The opt-out label is evaluated every reconcile pass, not only when a pod starts, so it takes effect on a running pod without restarting anything:

kubectl label pod <pod> pandocore.io/protect=false --overwrite

The engine for that pod is retired on the next scan. Remove the label to resume, which starts a fresh learning window for that workload.

The exact string matters

The comparison is byte-for-byte against "false", with no trimming and no case folding. False, FALSE and 0 do not opt a pod out. If you are in a hurry, this is the mistake to avoid: the command will succeed and nothing will change.

A freed slot is refilled in the same pass, so on a node that is shedding at its engine cap, opting one pod out brings a previously unwatched pod under monitoring. Useful deliberately; worth knowing about before it surprises you.

Stop One Workload Responding, But Keep Watching It

Usually the better emergency move. The workload stays watched and keeps producing evidence, and stops acting:

kubectl annotate deployment <name> pandocore.io/mode=monitor --overwrite
kubectl rollout restart deployment/<name>

The restart is required: the sensor reads a pod's annotations when it starts watching that pod. If you cannot restart the workload right now, use the opt-out label above instead, which needs no restart but also stops the evidence.

Stop A Whole Namespace

helm upgrade pando-node-sensor \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-node-sensor \
  --version 2.0.0 \
  --namespace pando-system \
  --reset-then-reuse-values \
  --set 'excludedNamespaces={kube-system,kube-public,kube-node-lease,<your-namespace>}'

Write the list out in full

This value replaces the list rather than adding to it, so include the defaults you want to keep. And setting it to [] does not mean "exclude nothing": an empty list is read as unconfigured and falls back to the built-in eleven.

Stop The Whole Cluster

A DaemonSet cannot be scaled to zero, so the fastest cluster-wide stop is to delete the DaemonSet object and leave the Helm release in place:

kubectl delete daemonset -n pando-system \
  -l app.kubernetes.io/name=pando-node-sensor

Every sensor pod goes with it and nothing is watched. The Helm release, the license Secret and the controller are untouched, so helm upgrade brings the DaemonSet back when you are ready, with a fresh learning window per workload.

If you would rather keep the DaemonSet but have it schedule nowhere, give it a label no node carries:

helm upgrade pando-node-sensor ... \
  --reset-then-reuse-values \
  --set nodeSelector.pandocore\\.io/disabled=true

Complete Uninstall

# Both charts
helm uninstall pando-node-sensor -n pando-system
helm uninstall pando-controller -n pando-system

# Any isolation policies still in place
kubectl get networkpolicy --all-namespaces \
  -l app.kubernetes.io/managed-by=pandocore

# The namespace, if nothing else lives there
kubectl delete namespace pando-system

Nothing has to be removed from your workloads, because nothing was ever added to them. There are no sidecars to strip out of deployment manifests and no pod specs to revert; the opt-out labels you may have added are inert once the sensor is gone.

Check the controller's spool before you delete its volumes

The controller's spool-* PersistentVolumeClaims hold evidence it has acknowledged and not yet delivered. helm uninstall does not remove them, which is deliberate. Delete them only once you are satisfied that evidence has reached its destinations, or you are discarding it knowingly.

Diagnosing Before Disabling

If you have a minute before acting, these three make the difference between a fix and a guess:

# What does the sensor think it is doing?
kubectl exec -n pando-system <sensor-pod> -- wget -qO- http://localhost:9090/status

# What fired, and against which pod?
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod>

# What is actually in place right now?
kubectl get networkpolicy --all-namespaces

/status reports the engine count, the license ceiling and any node clamp. A clamped node is held to Monitor by a portal-side verdict rather than by anything in your cluster, and node_clamp: none means no clamp applies.

Common Causes

Symptom	Likely cause	Action
A workload is contained repeatedly	Behaviour genuinely outside its profile, or a profile built during unusual conditions	Put it in `monitor`, then restart it to re-profile. Loosen `sensitivity` in a policy if it recurs
Alerts immediately after a deploy	The workload's behaviour changed and the profile describes the old release	Expected. A restarted pod re-profiles; wait out the learning window
Nothing is being watched on a node	Sensor refusing, or no sensor scheduled	Check `/readyz` and the logs. See Troubleshooting
Some pods on a node are never watched	The node is at its engine cap	Check `pando_node_sensor_pods_unmonitored{reason="cap"}`. Spread the workload or raise `maxEngines` with resources to match
A node alerts on nothing while others do	A portal-side node clamp holding it to Monitor	Check the clamp line in `/status` and the portal, not your license

Loosening Sensitivity

Per namespace, through a policy, rather than per workload:

kubectl patch pandocorepolicy default -n <namespace> \
  --type merge -p '{"spec":{"sensitivity":"low"}}'

PANDO_SENSITIVITY still exists but belongs to the DaemonSet on the node sensor, so it applies node-wide rather than to one workload. See CRD Policy.

Re-Profiling A Workload

There is no re-learn endpoint on the sensor. One sensor holds an engine per watched pod on its node, so a node-wide rebuild is not a meaningful operation. A workload re-profiles when its pods restart:

kubectl rollout restart deployment/<name>

Expect 30 minutes and at least 900 samples, whichever is later, before the new profile is armed.

Emergency Contacts

Reach support, or reply to your onboarding email thread. If a pod is contained and you need it back immediately, delete the NetworkPolicy first and tell us afterwards; you do not need our permission to undo a containment in your own cluster.