Skip to main content
Docs / Operating Modes

Operating Modes

A three-step ladder (monitor, alert, isolate) plus a debug mode. Each step adds one response action to the one below it.

PandoCore responds on a ladder: Monitor, Alert, Isolate. The rungs are additive, so each one does everything the rung below it does plus one more action, and intensity is monotonic. A separate Debug mode sits outside the ladder for troubleshooting.

Two things decide where a given workload sits: the ceiling your license permits, and the mode you set beneath it. A mode can only narrow what the license allows, never widen it.

What Sets The Ceiling

The response ceiling is a license property rather than a chart value, which is why raising it changes nothing in your cluster.

License Ceiling
Free Alert. Detects and alerts on as many nodes as you install on, and never contains
Growth, Enterprise Isolate
Isolate is the ceiling of the supported install

The node sensor's ClusterRole withholds pods: delete by name, and the sensor caps its own ceiling at Isolate in code rather than relying on that grant being absent. So the hardest automatic action available is a NetworkPolicy that contains the pod. A mode or license naming something harder resolves down to Isolate rather than failing, which is deliberate: the alternative is a sensor that attempts a forbidden call on every response and learns about it as a 403 each time.

There is a second, separate limit. The portal can clamp an individual node, holding it to Monitor regardless of license or annotation. That is a commercial verdict about the node rather than anything to do with your license, and the consequence is worth being precise about: a clamped node does not alert either, where a free node does. The two limits are reported separately on purpose, so an operator reading a clamped node is not sent to upgrade a license that was never the constraint.

A free license never clamps, because there is no node cap for a clamp to be a function of. What Free does not get is containment, and that comes from the license ceiling applying equally on every node rather than from protection being withdrawn from some of them.

Mode Comparison

Mode Detection Evidence Alerts NetworkPolicy isolation Use Case
Monitor ✓ ✓ ✗ ✗ Tuning a noisy workload, or a workload you want watched silently
Alert ✓ ✓ ✓ ✗ Production visibility with a human deciding what to do
Isolate ✓ ✓ ✓ ✓ Containment without restarting the workload
Debug ✓ ✓ ✗ ✗ Troubleshooting, with verbose logs

Monitor Mode

Monitor detects and records, and does nothing outward. It is the node-wide default for a pod carrying no mode annotation.

Reach for it when a specific workload is noisy and you want it watched while you tune, rather than as a fleet-wide starting posture: on a free license the fleet-wide posture is already Alert, and alerts you can read are the point of running it.

{
  "level": "warn",
  "timestamp": "2026-03-16T14:30:00Z",
  "message": "anomaly_detected",
  "mode": "monitor",
  "severity": "high",
  "action": "logged_only"
}

Alert Mode

Alert adds notification. On a detection, the controller delivers to whichever destinations you have configured, and the workload is left alone.

Destinations live on the controller

Alert does not need a webhook URL per workload. The controller owns every outbound destination, so SIEM, Slack and webhook delivery are configured once, and no workload pod holds a third-party credential. See Controller.

Isolate Mode

Isolate is the top of the ladder on the supported install. On a trip, PandoCore writes a Kubernetes NetworkPolicy that denies the pod's egress except DNS, and removes it again if the behaviour subsides.

A repeated attack on the same pod can resolve to an alert

The first excursion's own samples are folded into the baseline that judges the next one, so a second identical attack against the same pod is scored against a baseline that has partly learned it. It can therefore stay under the gate and resolve to an alert rather than an isolation. The second attack is not quieter and not less novel: in the measured case its peak was within a few percent of the first and its peak drift score was marginally higher, and what changed was that the score stayed under the gate for the whole excursion rather than crossing it.

Detection and evidence are unaffected. In the soak runs, each attack was detected and produced a signed evidence record, and Alert is the floor the ladder never drops below. A fix for the absorption is in test and is not in this release.

Why containment rather than a restart

A restart destroys the thing you want to look at and costs you the workload. Containment holds the pod still, keeps it available for inspection, and is survivable for stateful workloads where a restart is expensive: databases, caches, queues. If the behaviour was transient, the policy comes off and nothing was lost.

The policy is tighter than the sidecar's was

The sensor applies and removes the policy from outside the pod, so the policy needs no API-server exception: the deprecated sidecar ran inside the pod it was containing and had to be able to call the API to release itself, which meant granting an isolated workload reach it did not otherwise need. Dropping that also takes endpoints: get out of the sensor's grant entirely.

For the same reason ingress can be denied as well, so containment can be in both directions rather than egress-only. That half depends on your CNI permitting host-to-pod traffic regardless of policy, which Cilium and therefore GKE Dataplane V2 guarantee. On other CNIs, treat containment as egress-only.

Debug Mode

Debug is verbose logging with no response, and sits outside the ladder at the same intensity as Monitor. Use it to understand why something did or did not fire, then set it back.

Log volume

Debug is substantially more verbose than Monitor. Leave it on a single workload while you investigate rather than node-wide, and do not leave it on.

Setting A Workload's Mode

Mode is a per-pod annotation. The node-wide setting is the fallback an unannotated pod gets.

kubectl annotate deployment your-app pandocore.io/mode=isolate --overwrite
kubectl rollout restart deployment/your-app

Accepted values are monitor, alert, isolate and debug.

Three things worth knowing about this annotation

It is read when the sensor starts watching a pod, so a change applies to pods created after it. The rollout restart above is what applies it to pods already running.

The value is matched exactly, with no trimming and no case folding. Isolate and isolate are not the same string as isolate.

A value that does not match falls back to the node-wide default rather than leaving the pod unwatched, and the sensor warns naming the pod. Refusing to watch a workload over a typo would be the worse failure, but check for that warning after a change.

Reading The Current State

Ask a sensor what it is actually doing. /status reports the license ceiling and any node clamp as separate lines.

kubectl exec -n pando-system <sensor-pod> -- wget -qO- http://localhost:9090/status
node_clamp: none is a reading, not a gap

A clamp applies only on a signed verdict from the portal. A portal outage, an unresolved node identity, a cluster with no fresh presence map and an expired verdict all report none, and all of them mean this node is fully protected. An absent clamp is never a silent restriction.

  1. Install on a free license, fleet-wide. The ceiling is Alert, so it detects and tells you, and contains nothing. There is no trial to remember to leave and no mode to set.
  2. Read what it reports. Decide from your own evidence whether these are detections you would have wanted acted on. Put a noisy workload in monitor while you tune it rather than turning anything off.
  3. Raise the ceiling when you are ready for it to act. That is a license change, so the install does not change. Workloads then reach Isolate as their annotations permit.

There is no fixed number of weeks here. The old advice to spend a week in each rung existed because the evaluation period was a mode you had to remember to leave; it is now simply the free tier, and you leave it when the evidence says so.

Metrics

# Current operating mode (0=monitor, 1=alert, 2=isolate, 4=debug)
pando_operating_mode{pod="your-pod"} 2

# The ceiling this license permits, same encoding
pando_mode_ceiling 2

# Isolation policies successfully applied
pando_isolations_total{pod="your-pod"} 2

HTTP Endpoints

The sensor serves four endpoints on port 9090.

Endpoint Method Description
/healthz GET Liveness. Returns 200 even while the sensor is refusing to monitor, deliberately, so the kubelet does not crash-loop the refusal logs away
/readyz GET Readiness. 503 during learning and while refusing, 200 once armed. This is the one that gates
/metrics GET Prometheus metrics
/status GET Plain-text summary: engine count, license ceiling, node clamp

There is no re-learning endpoint on the sensor. One sensor holds an engine per watched pod on its node, so a single node-wide rebuild is not a meaningful operation; a workload re-profiles when its pods restart.

Next Steps