Operating Modes

PandoCore uses a four-step ladder — Monitor → Alert → Isolate → Enforce — controlling how it responds when anomalous behavior is detected. Modes layer additively: each step does everything the step below it does, plus one additional action. Pick the step that matches your deployment stage and risk tolerance. A separate Debug mode exists for troubleshooting.

ℹ️ Layered ladder

Everything Alert does, Isolate also does. Everything Isolate does, Enforce also does. One knob (PANDO_MODE), four values, monotonic intensity.

Mode Comparison

Mode	Detection	Evidence	Alerts (Slack + portal)	NetworkPolicy isolation	Pod Termination	Use Case
Monitor	✓	✓	✗	✗	✗	Initial deployment, tuning, evaluation
Alert	✓	✓	✓	✗	✗	Production visibility with human-in-the-loop triage
Isolate	✓	✓	✓	✓	✗	Contain suspicious pods without pod termination risk
Enforce	✓	✓	✓	✓	✓	Full production protection
Debug	✓	✓	✗	✗	✗	Troubleshooting, development (verbose logs)

Monitor Mode

Set via: PANDO_MODE=monitor

Monitor mode is the recommended starting point for all deployments. PandoCore detects anomalies and generates evidence but does not terminate pods.

ℹ️ Default Mode

Monitor mode is the default. You don't need to set any configuration to use it.

Behavior

Continuously monitors workload behavior
Logs detection events to stdout (JSON format)
Emits Kubernetes events for detections
Updates Prometheus metrics for detection activity
Does not terminate pods

When to Use

Initial deployment: Understand your workload's normal behavior patterns
Tuning: Observe detection events and adjust sensitivity
Evaluation: Test PandoCore before enabling enforcement
Compliance requirements: When you need detection without automated response

Example Output

{
  "level": "warn",
  "timestamp": "2026-03-16T14:30:00Z",
  "message": "anomaly_detected",
  "mode": "monitor",
  "severity": "high",
  "action": "logged_only"
}

Alert Mode

Set via: PANDO_MODE=alert

Alert mode adds real-time notifications to monitoring. When anomalous behavior is detected, PandoCore sends alerts to Slack (and other configured destinations) so your team is notified immediately, but pods are not terminated.

Requires Configuration

Alert mode requires a Slack incoming webhook URL. Set PANDO_SLACK_WEBHOOK_URL or configure slackWebhookURL in Helm values.

Behavior

Everything in Monitor mode, plus:
Sends color-coded Slack alerts on each detection event
Alerts include pod name, namespace, severity, and timestamp
Does not terminate pods

When to Use

Production rollout: Get notified before enabling enforcement
Compliance: When you need alerting without automated response
Team visibility: Keep security and ops teams in the loop

Example Alert

Slack alerts are color-coded by severity:

Critical (red): enforcement triggered
High (orange): significant anomaly detected
Warning (yellow): detection below enforcement threshold
Info (green): low-severity detection

Isolate Mode

Set via: PANDO_MODE=isolate

Isolate mode adds automated network containment on top of Alert mode. When anomalous behavior is detected, PandoCore generates evidence, sends alerts, and applies a NetworkPolicy to the compromised pod that blocks all egress traffic except DNS. Pods are not terminated.

Why isolate instead of terminate?

Isolation contains the blast radius of an in-progress attack (preventing lateral movement, data exfiltration, C2 callbacks) while preserving the running pod for forensic inspection. If the drift signal subsides within the recovery window, the NetworkPolicy is removed automatically and the pod is released. This is the right mode for stateful workloads where pod restart is expensive.

Behavior

Everything in Alert mode, plus:
Creates a pando-isolate-{pod-name} NetworkPolicy that denies egress except DNS (UDP 53)
Starts a recovery timer — after recoveryTimeoutSecs (default 60s) of clean samples, the NetworkPolicy is removed and the pod is released
Does NOT terminate the pod; the escalation-to-terminate path is only active in Enforce mode

When to Use

Stateful workloads: Databases, caches, message queues where pod restart causes downtime or data loss
Forensic-first security posture: Contain first, preserve the pod for inspection, then decide
Graduated rollout: A middle step between Alert (notify only) and Enforce (terminate). Confirm alerts are accurate, then confirm the NetworkPolicy doesn't disrupt legitimate traffic, then enable termination.

Example Output

{
  "level": "info",
  "timestamp": "2026-04-22T19:13:21Z",
  "message": "Pod isolated via NetworkPolicy — egress denied except DNS",
  "action": "isolated_pending_escalation",
  "policy": "pando-isolate-nginx-b8f89948c-l8spl",
  "recovery_timeout_secs": 60
}

Enforce Mode

Set via: PANDO_MODE=enforce

Enforce mode provides active protection. When anomalous behavior is detected, PandoCore runs the full graduated-response chain: alert → network isolation → escalation timer → pod termination.

⚠️ Production Use

Before enabling enforce mode, run in monitor mode for at least one week in staging to validate detection behavior for your workload. Consider stepping through Alert and Isolate first — the mode ladder is designed to let you promote gradually.

Behavior

Continuously monitors workload behavior
Generates and externalizes evidence on detection
Emits Kubernetes events before pod termination
Terminates the pod via Kubernetes API
Kubernetes automatically restarts the pod (assuming RestartPolicy allows)

Response Sequence

Detection: Anomalous behavior identified
Evidence generation: Structured evidence record created
Evidence externalization: Written to logs, K8s events, webhooks — before pod deletion, so it survives
Alert dispatch: Slack + portal notifications sent
Network isolation: NetworkPolicy applied (egress denied except DNS)
Escalation timer: Runs for escalationTimeoutSecs (default 120s). If drift subsides and recoveryTimeoutSecs clean samples (default 60s) accumulate first, the NetworkPolicy is removed and the pod is released.
Pod termination: If drift persists past the escalation window, Kubernetes API DELETE call fires; a second PandoCollapse K8s event records the escalation
Pod restart: Kubernetes scheduler creates new pod
Fresh baseline: New sidecar establishes clean baseline

Graduated response favors recovery for brief attacks

With default timers (escalationTimeoutSecs=120, recoveryTimeoutSecs=60), short attacks are contained via NetworkPolicy and then released when drift subsides — the pod is not killed. To demonstrate termination quickly in a lab, lower escalationTimeoutSecs or use a sustained attack longer than the escalation window.

When to Use

Production workloads: After validation in monitor mode
High-security environments: Where automated response is required
Workloads with rapid recovery: Stateless services that restart cleanly

Example Output

{
  "level": "error",
  "timestamp": "2026-03-16T14:30:00Z",
  "message": "enforcement_triggered",
  "mode": "enforce",
  "severity": "high",
  "action": "pod_terminated",
  "evidence_id": "abc123-def456"
}

Debug Mode

Set via: PANDO_MODE=debug

Debug mode provides verbose logging for troubleshooting and development. It outputs detailed information about every sample collected and all internal decisions.

⚠️ Log Volume

Debug mode generates significantly more log output. Use only for troubleshooting, not in production.

Behavior

Same as monitor mode (no pod termination)
Logs detailed information about detection activity and learning-window progress

When to Use

Troubleshooting: Understanding why detections are or aren't occurring
Sensitivity tuning: Seeing detection activity in context
Development: Testing PandoCore integration
Support requests: Gathering detailed logs for analysis

Switching Modes

You can change the operating mode at runtime without redeploying:

Switch to Monitor Mode

kubectl set env deployment/your-app -c pando-sidecar PANDO_MODE=monitor

Switch to Alert Mode

kubectl set env deployment/your-app -c pando-sidecar PANDO_MODE=alert

Switch to Isolate Mode

kubectl set env deployment/your-app -c pando-sidecar PANDO_MODE=isolate

Switch to Enforce Mode

kubectl set env deployment/your-app -c pando-sidecar PANDO_MODE=enforce

Switch to Debug Mode

kubectl set env deployment/your-app -c pando-sidecar PANDO_MODE=debug

ℹ️ Rolling Update

Changing the mode triggers a rolling restart of the deployment. The new pods will start with the updated mode and establish fresh baselines.

Recommended Deployment Path

Week 1-2: Monitor mode. Deploy to staging with PANDO_MODE=monitor. Observe detections, adjust sensitivity if needed.
Week 3-4: Alert mode. Switch to PANDO_MODE=alert in production. Your team gets notified on detections without any automated response.
Week 5-6: Isolate mode. Switch to PANDO_MODE=isolate. Tripped pods get a NetworkPolicy that blocks egress (except DNS); the policy is removed automatically once drift subsides. This is the right durable mode for stateful workloads.
Week 7+: Enforce mode. Enable PANDO_MODE=enforce for full active protection with alerts, network isolation, and pod termination for drift that persists past the escalation window.

Mode-Specific Metrics

PandoCore exposes the current mode via Prometheus metrics:

# Current operating mode (1 = active)
pando_mode_monitor{pod="your-pod"} 1
pando_mode_alert{pod="your-pod"} 0
pando_mode_enforce{pod="your-pod"} 0
pando_mode_debug{pod="your-pod"} 0

# Enforcement events (only in enforce mode)
pando_enforcements_total{pod="your-pod"} 3

HTTP Endpoints

PandoCore exposes several HTTP endpoints on port 9090 for health checks, monitoring, and operational control:

Endpoint	Method	Description
`/healthz`	GET	Health check: returns 200 if healthy, 503 if unhealthy
`/readyz`	GET	Readiness check: 503 while the learning window is in progress, 200 once the sidecar is actively detecting. Wire this into load balancer readiness gates to hold traffic until learning completes.
`/metrics`	GET	Prometheus metrics endpoint
`/relearn`	POST	Trigger re-learning phase to rebuild the baseline
`/status`	GET	JSON summary of sidecar state: version, mode, uptime, detection counts

Re-Learning Triggers

You can trigger re-learning in two ways:

# HTTP endpoint (within pod network)
kubectl exec -it POD_NAME -c pando-sidecar -- \
  curl -X POST http://localhost:9090/relearn

# SIGUSR1 signal (Linux only)
kubectl exec -it POD_NAME -c pando-sidecar -- \
  kill -USR1 1

Re-learning is useful when:

Application behavior has legitimately changed (new release)
Baseline was established during unusual conditions

Next Steps

Configuration: Settings reference
Emergency Procedures: How to disable quickly
Evidence Format: Understand detection output