Talos upgrades

A TalosUpgrade declares a target Talos version for a set of nodes. tuppr health-checks the cluster, then rolls the version out node-by-node (or in parallel batches), draining and rebooting each node and verifying it returns Ready on the new version before moving on.

The only required field is the target version:

apiVersion: tuppr.home-operations.com/v1alpha1
kind: TalosUpgrade
metadata:
  name: cluster
spec:
  talos:
    # renovate: datasource=docker depName=ghcr.io/siderolabs/talos
    version: v1.14.5

Everything below is optional and layers onto that. The full set of fields and defaults lives in the CRD schema; this page covers the ones you’ll reach for.

Health checks

Health checks are CEL expressions evaluated against live cluster objects before an upgrade proceeds. They run concurrently, and (for a TalosUpgrade) before each node or batch.

spec:
  healthChecks:
    # All nodes Ready
    - apiVersion: v1
      kind: Node
      expr: status.conditions.filter(c, c.type == "Ready").all(c, c.status == "True")
      timeout: 10m

    # A named Deployment fully rolled out
    - apiVersion: apps/v1
      kind: Deployment
      name: critical-app
      namespace: production
      expr: status.readyReplicas == status.replicas

    # Objects selected by labels
    - apiVersion: apps/v1
      kind: Deployment
      namespace: production
      labelSelector:
        matchLabels:
          app.kubernetes.io/part-of: critical-platform
      expr: status.readyReplicas == status.replicas

    # A custom resource
    - apiVersion: ceph.rook.io/v1
      kind: CephCluster
      name: rook-ceph
      namespace: rook-ceph
      expr: status.ceph.health in ["HEALTH_OK"]

Each check attempt is emitted as Events on the resource (HealthChecksStarted, then HealthChecksPassed or a HealthChecksFailed warning), so a run waiting on failing checks is visible in kubectl describe. A check with skipAfterPreHooks: true is left out of every run once pre-hooks have completed; a run with no checks left emits no Events.

Health checks are also available on KubernetesUpgrade.

Policy

spec.policy tunes how the upgrade runs:

Field	Default	Effect
debug	false	Verbose upgrade logging.
force	false	Skip etcd health checks. Dangerous - can break quorum.
placement	hard	How strictly the upgrade Job avoids the target node (hard or soft).
rebootMode	default	default, or powercycle for nodes that don’t reboot cleanly.
stage	false	Stage the upgrade and apply it on the next reboot (two reboots total).
waitForVolumeDetach	false	Drain and wait for CSI volumes to detach before the reboot (see below).
drainTimeout	unset	Timeout for talosctl’s own node drain (talosctl 1.13+, default 5m). If the drain exceeds it the upgrade fails without rebooting; raise it for nodes hosting several PDB-gated workloads. Keep it below timeout.
timeout	30m	Per-node upgrade timeout, including how long a node may take to become ready after its reboot before being marked failed.

waitForVolumeDetach

Set this when a fast reboot could orphan a mount and pin a volume to the node - which surfaces as a Multi-Attach error when the pod reschedules. tuppr drains the node and waits for its CSI volumes to detach before rebooting. No effect on single-node clusters.

Node selection

Without a selector, a plan targets all nodes. Use spec.nodeSelector (a standard label selector) to scope it - useful for opt-in labels or splitting a cluster into plans:

spec:
  nodeSelector:
    matchExpressions:
      - { key: tuppr.home-operations.com/upgrade, operator: In, values: ["enabled"] }
      - { key: node-role.kubernetes.io/control-plane, operator: DoesNotExist }

Multiple plans with different selectors queue; see Upgrade coordination.

Parallel upgrades

By default nodes upgrade one at a time. spec.parallelism raises the batch size:

spec:
  talos:
    version: v1.13.7
  parallelism: 3 # up to 3 nodes at once

The webhook requires parallelism >= 1 and no greater than the number of nodes the selector matches. When parallelism > 1:

• Health checks run once before each batch, not per node.
• Drain (if configured) runs on all batch nodes before any upgrade Job starts.
• The batch waits for every node’s Job to finish before the next batch begins.
• Any failure in a batch stops further batches.
• status.currentNodes lists the active batch.

Installer pre-pull

Before the first node is cordoned, tuppr pulls each node’s resolved installer image into the node’s image store (the per-node resolution described in How the upgrade image is resolved). An unreachable registry, a missing schematic, or a wrong tag then parks the run with a clear PrePullFailed status - before anything is drained - instead of failing mid-roll with one node down. The upgrade’s own image pull becomes a no-op, since Talos skips images that are already present.

Pre-pull is on by default. It requires the node to be running Talos v1.13 or newer (the ImageService API); older nodes are skipped with an event and upgrade exactly as before. Disable it for airgapped clusters that seed images out of band, or when the extra pull is unwanted on constrained links:

spec:
  talos:
    version: v1.13.7
    prePull: false

A pull that reports no progress for 90 seconds fails fast (a registry serving errors that the node retries internally would otherwise be indistinguishable from a slow link), while a slow but progressing pull gets up to 10 minutes. A parked run retries with backoff (1 minute doubling to a 5 minute cap) and resumes on its own once the registry serves the image; partial pulls resume where they left off. The retry streak is tracked in status.prePullFailure (attempt count and last error) and surfaced in the status message, so a crash-looping pre-pull is visible in kubectl get talosupgrade rather than only in events. Pulled images are tracked per node in status.prePulledNodes, keyed by the resolved ref and the Node UID: a node that joins (or becomes eligible) mid-run, is recreated under the same name, or whose resolved image changes, is pre-pulled before its next batch.

Draining

spec:
  drain:
    enabled: true
    # disableEviction: false  # force delete instead of evicting

Deprecated

spec.drain is deprecated. talosctl 1.13+ drains the node itself as part of the upgrade (disable with policy.nodrain), and policy.waitForVolumeDetach covers the CSI detach case. Note that disableEviction: true deletes pods that a PodDisruptionBudget would protect, which can stall operators that expect an orderly eviction (e.g. a CloudNativePG primary switchover).

tuppr cordons the node, evicts its pods, and waits for them to terminate before the reboot, then uncordons the node after a verified upgrade. While it is set, talosctl’s own drain is disabled so the node isn’t drained twice. This also applies to a single-node cluster, where talosctl’s own drain is disabled: evicted pods stay Pending until the node is uncordoned. policy.waitForVolumeDetach on its own never drains a single node.

Pre/post-upgrade hooks

Run side-effecting Jobs around the whole run - for example, set and clear Ceph noout so brief reboots don’t trigger rebalancing:

spec:
  talos:
    version: v1.13.7
  hooks:
    pre:
      - name: ceph-set-noout
        image: ghcr.io/rook/rook:v1.18.7
        command: ["sh", "-c"]
        args: ["ceph osd set noout"]
        envFrom:
          - secretRef:
              name: rook-ceph-mon
    post:
      - name: ceph-unset-noout
        image: ghcr.io/rook/rook:v1.18.7
        command: ["sh", "-c"]
        args: ["ceph osd unset noout"]
        envFrom:
          - secretRef:
              name: rook-ceph-mon
  healthChecks:
    # noout puts Ceph in HEALTH_WARN, so this check only gates the initial run
    - apiVersion: ceph.rook.io/v1
      kind: CephCluster
      name: rook-ceph
      namespace: rook-ceph
      expr: status.ceph.health in ["HEALTH_OK"]
      skipAfterPreHooks: true

• Pre-hooks run sequentially after the initial health check, before any node is touched. A failed pre-hook skips the upgrade and marks the run Failed (post-hooks still run as cleanup).
• Post-hooks run sequentially once the upgrade reaches a terminal state (success or failure), if any pre-hook was attempted. Their failures are logged and recorded but don’t change the upgrade outcome.
• Health checks still run before every batch after the pre-hooks. A check the pre-hooks invalidate for the upgrade window (Ceph is HEALTH_WARN while noout is set) must set skipAfterPreHooks: true: it then gates only the initial check before the hooks, otherwise every later run fails and the upgrade stalls before its first batch. Other checks keep gating each batch, so a pre-hook that pauses CI runners doesn’t switch off the Ceph check.
• Each hook is a Job in the controller namespace with the same non-root, capabilities-dropped posture as the upgrade Job. Mount credentials via volumes / envFrom, and set serviceAccountName if you need cluster-API access.

Phase progression with hooks:

Pending → HealthChecking → PreHook → (Draining → Upgrading → Rebooting per batch) → PostHook → Completed

Maintenance windows

Restrict when upgrades may start. An in-progress upgrade always runs to completion.

spec:
  maintenance:
    windows:
      - start: "0 2 * * 0" # cron: Sunday 02:00
        duration: "4h" # max 168h; warns if < 1h
        timezone: "Europe/Paris" # IANA tz, default UTC

• Upgrades stay Pending outside every window.
• Multiple windows are a union (any open window allows a start).
• A TalosUpgrade re-checks the window between nodes.
• No windows configured → upgrades start immediately.

Alertmanager silences

Every node reboot during an upgrade fires a burst of expected alerts (CephMonDown, KubeNodeUnreachable, TargetDown, …). tuppr knows exactly when that window starts and ends, so it can hold an Alertmanager silence open for the run.

The connection is operator-level (credentials stay out of CRs - see the chart values); which alerts get silenced is per-resource and user-authored, same philosophy as healthChecks:

# Helm values
silences:
  enabled: true
  alertmanager: # where the silences are created
    address: http://alertmanager.monitoring.svc:9093
    secretName: tuppr-alertmanager # optional: Secret of HTTP headers (Authorization, X-Scope-OrgID)

# TalosUpgrade
spec:
  silences: # requires silences.* configured in the Helm chart
    - matchers:
        - name: alertname
          value: ^(CephMonDown|CephOSDDown|KubeNodeUnreachable|TargetDown)$
          matchType: "=~" # "=", "!=", "=~", "!~"
      maxDuration: 2h # default 4h

Each list entry is one Alertmanager silence, held for the whole run.

Matchers within one silence are ANDed

Alertmanager semantics: a silence matches an alert only when all its matchers do. To cover alternatives, use a regex value (as above); to silence independent scopes - say Ceph alerts only in namespace=rook-ceph plus a cluster-wide TargetDown - use separate silences entries, one per scope.

Anything speaking the Alertmanager v2 silences API works: Prometheus Alertmanager, VMAlertmanager, or Grafana-managed alerting behind its prefix.

The silence is a lease, not a timer

tuppr never guesses how long the run will take. It creates a short silence (~25m) and re-extends it on every reconcile while the run is active, then expires it down to a ~5m tail at the terminal phase (so the last reboot’s alert tail stays covered). Every failure mode converges to “the silence lapses within its TTL”:

• Controller crash or CR deleted mid-run → lapses on its own.
• Alertmanager unreachable → logged and evented, never gates the upgrade.
• Run parked (Pending, MaintenanceWindow, coordination wait) → lapses; re-established when the run resumes. A cluster parked overnight should alert.
• Suspending the CR (the suspend annotation) expires the silence immediately.

Two details worth knowing:

• The silence is created only once the run leaves its initial health check, so alerts stay live while the cluster is still being verified - and it keeps being extended through inter-batch health checks, so there is no mid-run gap.
• A run that is active but stuck (health checks failing for hours because something is genuinely broken) stops extending once the continuous hold exceeds an entry’s maxDuration, emitting a SilenceMaxDurationReached event (once) - you get paged again instead of masking a real failure. The budget re-arms when the hold is released, so a run resumed after a long park silences its remaining reboots normally.

The silence IDs are visible in .status.alertSilenceIDs (indexed like spec.silences), and their lifecycle is emitted as Events (SilenceCreated, SilenceExpired, …) on the resource. Configuring spec.silences without the operator-level Alertmanager connection warns at apply time (admission webhook) and as a SilencesNotConfigured event.

Alternative: Alertmanager inhibition

If you’d rather keep alert routing entirely in Alertmanager, you can wire an inhibition rule keyed on tuppr’s tuppr_talos_upgrade_phase metric (via an always-firing “upgrade in progress” alert) instead - zero tuppr configuration, at the cost of maintaining the alert + inhibit_rules by hand.

Per-node overrides

Annotations on a Node override the plan for that node - handy for a canary version or a different installer flavor.

Annotation	Purpose	Example
tuppr.home-operations.com/version	Target Talos version for this node.	v1.12.1
tuppr.home-operations.com/factory-url	Switch the node’s installer flavor on the next upgrade.	factory.talos.dev/aws-installer
tuppr.home-operations.com/schematic	Companion to factory-url; the schematic to build the image with.	b55fbf...

# Pin one node to a different version
kubectl annotate node worker-01 tuppr.home-operations.com/version="v1.12.1"

# Switch a node to a factory flavor at the next upgrade
kubectl annotate node hcloud-01 \
  tuppr.home-operations.com/factory-url="factory.talos.dev/hcloud-installer" \
  tuppr.home-operations.com/schematic="314b18a3f89d..."

How the upgrade image is resolved

tuppr derives the image from each node’s runtime state and .machine.install.image - no per-platform branching:

1. factory-url override → builds <factory-url>/<schematic>:<version>. The schematic comes from the schematic annotation if set, otherwise from the node’s runtime ExtensionStatus.
2. Default → version-swaps the node’s current install image. A factory install stays on its factory base + schematic, and a private-registry path is preserved. A vanilla install on ghcr.io/siderolabs/installer stays vanilla up to v1.13; from v1.14, which Talos stopped publishing that image for, it moves to the Image Factory installer for the node’s platform with the default (empty) schematic.
3. Safety net → refuses with a clear error (pointing at factory-url) when the runtime schematic isn’t in the install-image path, or when reinstalling the canonical generic installer would silently wipe installed system extensions.

Private registries

Before each batch, the controller checks that the registry serves the node’s target image. The node pulls the image with the registry credentials from its machine config, but that check runs in the controller pod, which can’t read them. For a registry that rejects anonymous pulls, give the controller its own credentials: create a kubernetes.io/dockerconfigjson Secret in tuppr’s namespace and mount it as a Docker config through the chart values. Without it, the upgrade stays Pending with reason WaitingForImage and an UNAUTHORIZED error.

env:
  - name: DOCKER_CONFIG
    value: /var/run/secrets/registry
volumes:
  - name: registry-auth
    secret:
      secretName: my-registry-secret
      items:
        - key: .dockerconfigjson
          path: config.json
volumeMounts:
  - name: registry-auth
    mountPath: /var/run/secrets/registry
    readOnly: true

talosctl image

The upgrade Job’s talosctl version is auto-detected. Pin it if needed:

spec:
  talosctl:
    image:
      repository: ghcr.io/siderolabs/talosctl
      tag: v1.11.0
      pullPolicy: IfNotPresent