Operating Modes

PandoCore uses a three-tier escalation model — Monitor → Alert → Enforce — controlling how it responds when anomalous behavior is detected. Choose the mode that matches your deployment stage and risk tolerance.

Mode Comparison

Monitor Mode

Set via: PANDO_MODE=monitor

Monitor mode is the recommended starting point for all deployments. PandoCore detects anomalies and generates evidence but does not terminate pods.

Behavior

• Continuously monitors workload behavior
• Logs detection events to stdout (JSON format)
• Emits Kubernetes events for detections
• Updates Prometheus metrics with drift scores
• Does not terminate pods

When to Use

• Initial deployment: Understand your workload's normal behavior patterns
• Threshold tuning: Observe detection events and adjust thresholds
• Evaluation: Test PandoCore before enabling enforcement
• Compliance requirements: When you need detection without automated response

Example Output

{
  "level": "warn",
  "timestamp": "2026-03-16T14:30:00Z",
  "message": "drift_detected",
  "mode": "monitor",
  "drift_score": 0.67,
  "threshold_exceeded": "cumulative",
  "action": "logged_only"
}

Alert Mode

Set via: PANDO_MODE=alert

Alert mode adds real-time notifications to monitoring. When anomalous behavior is detected, PandoCore sends alerts to Slack (and other configured destinations) so your team is notified immediately — but pods are not terminated.

Behavior

• Everything in Monitor mode, plus:
• Sends color-coded Slack alerts on each detection event
• Alerts include pod name, namespace, drift score, severity, and timestamp
• Does not terminate pods

When to Use

• Production rollout: Get notified before enabling enforcement
• Compliance: When you need alerting without automated response
• Team visibility: Keep security and ops teams in the loop

Example Alert

Slack alerts are color-coded by severity:

• Critical (red) — enforcement triggered
• High (orange) — drift score ≥ 0.6
• Warning (yellow) — drift score ≥ 0.3
• Info (green) — low drift detection

Enforce Mode

Set via: PANDO_MODE=enforce

Enforce mode provides active protection. When anomalous behavior is detected and exceeds configured thresholds, PandoCore generates evidence, sends alerts, and terminates the compromised pod.

Behavior

• Continuously monitors workload behavior
• Generates and externalizes evidence when thresholds are exceeded
• Emits Kubernetes events before pod termination
• Terminates the pod via Kubernetes API
• Kubernetes automatically restarts the pod (assuming RestartPolicy allows)

Response Sequence

1. Detection: Drift exceeds configured threshold
2. Evidence generation: Structured evidence record created
3. Evidence externalization: Written to logs, K8s events, webhooks
4. Alert dispatch: Slack notification sent (if configured)
5. Network isolation: NetworkPolicy applied to restrict pod egress traffic
6. Escalation: If anomalous behavior persists past configurable timer, escalate to termination
7. Pod termination: Kubernetes API DELETE call
8. Pod restart: Kubernetes scheduler creates new pod
9. Fresh baseline: New sidecar establishes clean baseline

When to Use

• Production workloads: After validation in monitor mode
• High-security environments: Where automated response is required
• Workloads with rapid recovery: Stateless services that restart cleanly

Example Output

{
  "level": "error",
  "timestamp": "2026-03-16T14:30:00Z",
  "message": "enforcement_triggered",
  "mode": "enforce",
  "drift_score": 0.67,
  "threshold_exceeded": "cumulative",
  "action": "pod_terminated",
  "evidence_id": "abc123-def456"
}

Debug Mode

Set via: PANDO_MODE=debug

Debug mode provides verbose logging for troubleshooting and development. It outputs detailed information about every sample collected and all internal decisions.

Behavior

• Same as monitor mode (no pod termination)
• Logs detailed information about every sample
• Outputs baseline establishment progress
• Shows threshold calculation details
• Provides drift component breakdowns

When to Use

• Troubleshooting: Understanding why detections are or aren't occurring
• Threshold tuning: Seeing exactly what triggers detections
• Development: Testing PandoCore integration
• Support requests: Gathering detailed logs for analysis

Switching Modes

You can change the operating mode at runtime without redeploying:

Switch to Monitor Mode

kubectl set env deployment/your-app -c pando-sidecar PANDO_MODE=monitor

Switch to Alert Mode

kubectl set env deployment/your-app -c pando-sidecar PANDO_MODE=alert

Switch to Enforce Mode

kubectl set env deployment/your-app -c pando-sidecar PANDO_MODE=enforce

Switch to Debug Mode

kubectl set env deployment/your-app -c pando-sidecar PANDO_MODE=debug

Recommended Deployment Path

1. Week 1-2: Monitor mode — Deploy to staging with PANDO_MODE=monitor. Observe detections, tune thresholds if needed.
2. Week 3-4: Alert mode — Switch to PANDO_MODE=alert in production. Your team gets notified on detections without enforcement.
3. Week 5+: Enforce mode — Enable PANDO_MODE=enforce for full active protection with alerts and pod termination.

Mode-Specific Metrics

PandoCore exposes the current mode via Prometheus metrics:

# Current operating mode (1 = active)
pando_mode_monitor{pod="your-pod"} 1
pando_mode_alert{pod="your-pod"} 0
pando_mode_enforce{pod="your-pod"} 0
pando_mode_debug{pod="your-pod"} 0

# Enforcement events (only in enforce mode)
pando_enforcements_total{pod="your-pod"} 3

HTTP Endpoints

PandoCore exposes several HTTP endpoints on port 9090 for health checks, monitoring, and operational control:

Re-Learning Triggers

You can trigger re-learning in two ways:

# HTTP endpoint (within pod network)
kubectl exec -it POD_NAME -c pando-sidecar -- \
  curl -X POST http://localhost:9090/relearn

# SIGUSR1 signal (Linux only)
kubectl exec -it POD_NAME -c pando-sidecar -- \
  kill -USR1 1

Re-learning is useful when:

• Application behavior has legitimately changed (new release)
• Thresholds need recalibration after initial tuning
• Baseline was established during unusual conditions

Next Steps

• Configuration — Tune thresholds for your workload
• Emergency Procedures — How to disable quickly
• Evidence Format — Understand detection output