Skip to main content
Docs / Upgrading

Upgrading

Version the sensor and controller, and move off the deprecated webhook.

PandoCore installs as two charts: the node sensor (a DaemonSet, one pod per node) and the controller (a StatefulSet that owns evidence delivery). They version independently. This page covers upgrading each, and moving off the deprecated admission webhook.

Check your current version first
# Installed chart versions
helm list -n pando-system

# What a sensor is actually running
kubectl get daemonset -n pando-system \
  -o jsonpath='{.items[0].spec.template.spec.containers[0].image}'

# Latest available version
curl -s https://portal.pandocore.io/api/version

Migrating Off The Admission Webhook

If you are running the webhook and injected sidecars, this is the one upgrade that is not a version bump. The webhook and the sidecar are deprecated and receive no further release.

Do not run both topologies at once

The sensor excludes its own pod, hostPID pods, hostNetwork pods and anything carrying the opt-out label. It does not exclude a pod that already has a sidecar. Install the sensor while the webhook is still injecting and those pods are watched twice, by their own sidecar and by the node sensor, which means two sets of detections and two independent responses to the same behaviour. Follow the order below.

  1. Install the controller if you do not already run one. See Getting Started. Existing webhook installs usually have one.
  2. Stop the webhook injecting. Uninstall the webhook chart, or scale it to zero if you want to keep the release for rollback. Already-running pods keep their sidecars until they restart.
  3. Restart your protected workloads so their pods come back without a sidecar. kubectl rollout restart deployment/your-app, in batches, most critical last.
  4. Install the node sensor. It picks up every workload on each node without further labelling.
  5. Remove the old labels. pandocore.io/protect: "true" no longer does anything. Leaving it is harmless, but it reads as though it still means something.
Your metered node count will change

Under the webhook, the nodes that metered were the ones running injected pods. Under the sensor, every schedulable node runs a sensor and meters. A cluster that ran sidecars on 3 of 10 nodes metered 3; a DaemonSet on 10 schedulable nodes meters 10. Check Pricing against your node count before you migrate a large cluster.

You can reuse the license Secret the webhook chart created rather than passing the key again:

helm install pando-node-sensor ... \
  --set license.existingSecret.name=pando-license

Three annotations stop working at this point: pandocore.io/full-proc-signals, pandocore.io/provenance-enforce and pandocore.io/node-enforcer. All three were applied by rewriting a pod at admission, which a sensor cannot do from outside it. See Production Hardening for why the coverage they bought is not lost.

Upgrade The Node Sensor

helm upgrade pando-node-sensor \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-node-sensor \
  --version 2.0.0 \
  --namespace pando-system \
  --reset-then-reuse-values

Use --reset-then-reuse-values (Helm 3.14 or later) rather than --reuse-values. It keeps what you set, such as your license key, and takes the new chart's defaults for everything else. --reuse-values keeps the old chart's defaults too, so a setting a newer chart introduced can render empty.

The DaemonSet rolls node by node, at most maxUnavailable at a time (one by default). While a node's sensor is restarting, the workloads on that node are not being watched, so a rollout is a brief per-node coverage gap rather than a cluster-wide one. Nothing in your workloads restarts.

The capacity preflight runs on upgrade too

If the new chart raises the CPU request and a node cannot allocate it, the upgrade fails rather than leaving that node's sensor Pending for ever. The message names the nodes. Either give them room, exclude them with a nodeSelector and accept that they go unwatched, or proceed deliberately with --set preflight.nodeCapacity=false.

Upgrade The Controller

The controller is a StatefulSet that keeps each replica's evidence spool on its own volume. The StatefulSet replaces one replica at a time, and each new pod tries to deliver what is on its spool before it accepts new evidence.

helm upgrade pando-controller \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-controller \
  --version 2.0.0 \
  --namespace pando-system \
  --reset-then-reuse-values

One-time step for controllers installed before 2026-09-28

Controller charts published before 2026-09-28 wrote the chart version into the StatefulSet's volume claim template, which Kubernetes never lets change. A controller installed from one of them cannot be upgraded in place. helm upgrade stops with a message that begins:

The live StatefulSet pando-system/pando-controller-pando-controller has a spool
volumeClaimTemplate this chart cannot upgrade in place.

and ends with the command to run. Tools that render with helm template, such as Argo CD, show the API server's message instead, spec: Forbidden: updates to statefulset spec for fields other than ..., and the fix is the same. To check whether your install needs it:

kubectl get statefulset pando-controller-pando-controller -n pando-system \
  -o jsonpath='{.spec.volumeClaimTemplates[0].metadata.labels}'

If the output includes app.kubernetes.io/version, delete the StatefulSet object once, keeping its pods and volumes, then run the upgrade:

kubectl delete statefulset pando-controller-pando-controller -n pando-system --cascade=orphan

helm upgrade pando-controller \
  oci://us-central1-docker.pkg.dev/pandocore-prod/charts/pando-controller \
  --version 2.0.0 \
  --namespace pando-system \
  --reset-then-reuse-values
Never delete the spool volumes

--cascade=orphan removes only the StatefulSet object. The controller pods keep running and the spool-* PersistentVolumeClaims are untouched. Those volumes hold evidence the controller has acknowledged and not yet delivered. The new StatefulSet finds them by name (spool-pando-controller-pando-controller-0 and -1), mounts the same volumes, and delivers what is on them. Run the upgrade straight after the delete: until it runs, a controller pod that is evicted or deleted is not recreated.

Afterwards, confirm the claims are the ones you had, bound to the same volumes:

kubectl get pvc -n pando-system -l app.kubernetes.io/instance=pando-controller

Existing claims keep their old app.kubernetes.io/version label, because a StatefulSet never relabels a claim it already created. A helm rollback to a release made before this change needs the same one-time step.

Learning After An Upgrade

A sensor builds its profiles in memory and keeps working state under /var/lib/pando on its node as a hostPath rather than a PersistentVolumeClaim, deliberately: a restored state directory is refused as older than the pod reading it. So a sensor restart re-profiles the workloads on that node.

Version Skew

Sensor and controller do not need to match exactly. Upgrade the controller first: it is what accepts evidence, and a newer sensor delivering to an older controller is the direction that can lose a field.

Scenario Supported?
Sensor and controller on the same minor version Yes
Controller newer than the sensors delivering to it Yes. This is the order to upgrade in
Sensor newer than the controller Not recommended. Upgrade the controller first
Sensors on mixed patch versions mid-rollout Yes. This is what a DaemonSet rollout looks like

Rollback

# Roll back the most recent sensor upgrade
helm rollback pando-node-sensor -n pando-system

# Or the controller
helm rollback pando-controller -n pando-system

Your workloads do not restart either way, since nothing runs inside them. A sensor rollback is a DaemonSet rollout in reverse, with the same per-node learning window on the way back. If you need to stop PandoCore responding immediately rather than roll a version back, see Emergency Procedures.

Upgrade Checklist