Emergency Procedures

This page provides procedures for quickly disabling or removing PandoCore from your workloads. Keep this information accessible for incident response scenarios.

Quick Reference

Scenario	Time	Command
Stop enforcement (keep monitoring)	30 seconds	`kubectl set env deployment/NAME -c pando-sidecar PANDO_MODE=monitor`
Remove sidecar completely	2 minutes	Edit deployment to remove sidecar container
Uninstall all PandoCore components	5 minutes	`helm uninstall pando-sidecar -n pando-system`

Immediate: Stop Pod Termination (30 seconds)

If PandoCore is terminating pods you need running, switch to monitor mode immediately:

# Stop enforcement but keep monitoring
kubectl set env deployment/YOUR_DEPLOYMENT -c pando-sidecar PANDO_MODE=monitor

✓ Safe and Reversible

This triggers a rolling restart. New pods will still run PandoCore, but it will only observe and log — it will not terminate pods. Evidence is still generated for review.

For Multiple Deployments

# List all deployments with PandoCore sidecar
kubectl get deployments --all-namespaces -o json | \
  jq -r '.items[] | select(.spec.template.spec.containers[].name=="pando-sidecar") | 
  "\(.metadata.namespace)/\(.metadata.name)"'

# Switch all to monitor mode
for deploy in $(kubectl get deployments --all-namespaces -o json | \
  jq -r '.items[] | select(.spec.template.spec.containers[].name=="pando-sidecar") | 
  "\(.metadata.namespace) \(.metadata.name)"'); do
    ns=$(echo $deploy | cut -d' ' -f1)
    name=$(echo $deploy | cut -d' ' -f2)
    kubectl set env deployment/$name -n $ns -c pando-sidecar PANDO_MODE=monitor
done

Full Removal: Remove the Sidecar (2 minutes)

To completely remove PandoCore from a specific deployment:

Step 1: Edit the Deployment

kubectl edit deployment/YOUR_DEPLOYMENT -n YOUR_NAMESPACE

Step 2: Remove These Elements

In the editor, remove:

The pando-sidecar container from spec.template.spec.containers
shareProcessNamespace: true from spec.template.spec (if no other container needs it)
The serviceAccountName: pando-sidecar (if you want to revert to default)

Alternative: Patch Command

# Remove shareProcessNamespace
kubectl patch deployment YOUR_DEPLOYMENT -n YOUR_NAMESPACE \
  --type=json \
  -p='[{"op": "remove", "path": "/spec/template/spec/shareProcessNamespace"}]'

# Note: Removing the container requires editing the deployment directly

Complete Uninstall (5 minutes)

To remove all PandoCore components from your cluster:

Step 1: Uninstall Helm Release

helm uninstall pando-sidecar -n pando-system

Step 2: Clean Up Namespace (Optional)

# Delete the PandoCore namespace and all resources
kubectl delete namespace pando-system

Step 3: Remove Sidecars from Deployments

The Helm uninstall removes RBAC and ConfigMaps, but sidecars remain in your deployments until you remove them. See the "Full Removal" section above for each deployment.

Diagnosing Before Disabling

Before disabling PandoCore, check what triggered the response:

View Recent Evidence

# View the most recent detection event
kubectl logs -n YOUR_NAMESPACE YOUR_POD -c pando-sidecar --previous | \
  grep "enforcement_triggered\|drift_detected" | tail -5

View Kubernetes Events

kubectl get events -n YOUR_NAMESPACE --sort-by='.lastTimestamp' | grep -i pando

Check Current Drift Score

kubectl exec YOUR_POD -c pando-sidecar -- \
  wget -qO- http://localhost:9090/metrics | grep pando_drift

Common Causes and Solutions

Symptom	Likely Cause	Solution
Termination during deployment	New code changes baseline behavior	Trigger re-learn: `wget --post-data='' http://localhost:9090/relearn`
Termination during autoscaling	Memory/process spike from load	Raise thresholds: `PANDO_THRESHOLD_INCREMENTAL=0.20`
Termination during CronJob	Periodic batch processing	Raise cumulative threshold: `PANDO_THRESHOLD_CUMULATIVE=0.55`
Repeated terminations	Workload is inherently variable	Run in monitor mode longer, then tune thresholds

Threshold Adjustment

If PandoCore is responding to legitimate workload behavior, adjust thresholds instead of disabling:

# Make less sensitive to sudden spikes
kubectl set env deployment/YOUR_DEPLOYMENT -c pando-sidecar \
  PANDO_THRESHOLD_INCREMENTAL=0.20

# Make less sensitive to gradual drift
kubectl set env deployment/YOUR_DEPLOYMENT -c pando-sidecar \
  PANDO_THRESHOLD_CUMULATIVE=0.55

# Apply both
kubectl set env deployment/YOUR_DEPLOYMENT -c pando-sidecar \
  PANDO_THRESHOLD_INCREMENTAL=0.20 \
  PANDO_THRESHOLD_CUMULATIVE=0.55

Trigger Re-Learning

If your application's normal behavior has changed (new deployment, config change), trigger a re-learn instead of disabling:

# Via HTTP endpoint
kubectl exec YOUR_POD -c pando-sidecar -- \
  wget -qO- --post-data='' http://localhost:9090/relearn

ℹ️ Re-Learning

This resets the sidecar to its learning phase, recalibrating the baseline to the current behavior. The sidecar won't enforce during the learning period.

Emergency Contacts

If you're experiencing issues during your pilot that aren't resolved by these procedures, contact PandoCore support directly with:

The evidence JSON from the detection event
What your workload was doing at the time
Your current threshold settings
Sidecar logs from around the time of the issue

Printable Quick Reference

📋 Emergency Commands

# STOP ENFORCEMENT (30 seconds)
kubectl set env deployment/NAME -c pando-sidecar PANDO_MODE=monitor

# TRIGGER RE-LEARN
kubectl exec POD -c pando-sidecar -- wget -qO- --post-data='' http://localhost:9090/relearn

# RAISE THRESHOLDS
kubectl set env deployment/NAME -c pando-sidecar PANDO_THRESHOLD_INCREMENTAL=0.20

# VIEW RECENT EVENTS
kubectl logs POD -c pando-sidecar --previous | grep enforcement_triggered