文档

Migration guide

The manual steps each Self-Hosted Deployment release needs beyond the standard upgrade
阅读时间34 分钟最后更新于 13 天前

The manual steps an existing Self-Hosted Deployment needs when it upgrades past a release, beyond the standard upgrade. The newest release comes first, and a release that needs no such step has no section. For everything else each release changes, refer to the release notes.
When you upgrade across several releases, complete the section of every release after your current one, up to and including your target, oldest first. Each item says which deployments it applies to and at which step of the upgrade to complete it. In every command,
<namespace>
is your deployment's namespace, the one
configuration.kubernetes.namespace
names.

Version 2.1.0

Hand the service accounts over to Helm

If you deploy with Helm (
shdctl release deploy --format helm
), complete this item once, when you regenerate and deploy the charts during the upgrade, before
shdctl release deploy
. A first installation and a deployment that uses ArgoCD skip this item.
In this release, the platform services' service accounts are part of their Helm releases, so an upgrade doesn't re-create them. Hand the existing service accounts over to Helm. Otherwise, the deployment stops with
invalid ownership metadata
.
kubectl -n <namespace> get serviceaccounts -l creator-services-app=asset-solutions -o jsonpath='{range .items[?(@.metadata.annotations.helm\.sh/hook)]}{.metadata.name} {.metadata.labels.app\.kubernetes\.io/instance}{"\n"}{end}' > /tmp/service-accounts.txtwhile read -r sa release; do kubectl -n <namespace> annotate --overwrite serviceaccount "$sa" meta.helm.sh/release-name="$release" meta.helm.sh/release-namespace=<namespace> helm.sh/hook- helm.sh/hook-weight-; done < /tmp/service-accounts.txt
The first command lists the service accounts to hand over, each with its release. The second command hands them over without restarting anything.
If you later deploy an earlier release over this one, that deployment deletes these service accounts. The platform services can't start new pods until you deploy the earlier release a second time, which restores them.

Automation 0.0.81: snapshot its database first

This item applies to every deployment. When you deploy this release, Pipeline Automation 0.0.81 migrates its database in the bundled PostgreSQL. Before you start the upgrade, take a full backup of the bundled PostgreSQL, and wait until the backup succeeds:
backup=$(kubectl -n <namespace> create -o name -f - <<'EOF'apiVersion: pgv2.percona.com/v2kind: PerconaPGBackupmetadata: generateName: before-upgrade-spec: pgCluster: pg-db repoName: repo1 options: - --type=fullEOF)kubectl -n <namespace> wait "$backup" --for=jsonpath='{.status.state}'=Succeeded --timeout=60m
Each run creates a new backup, so if you postpone the upgrade, run the commands again right before you start. The backup goes where your PostgreSQL backups already go: to their backup volume, or to the bucket that
configuration.objectStore.buckets.postgresBackups
names.

Let running transformations finish before you deploy

This item applies to every deployment. This release replaces the shared volume that transformations use as scratch space. If a transformation is running when you deploy the charts, or starts during the deployment, its pod can stay
Pending
on the old volume,
azurefile-sync-share-default
. The pod never starts and never fails.
Upgrade while no transformation is running. The old volume held only the files of transformations in progress, so you don't need to copy anything.
To clear the pods that still wait on the old volume after the upgrade completes, follow these steps:
  1. List the pods that are still waiting and the volumes they claim:
    kubectl -n <namespace> get pods --field-selector=status.phase=Pending \ -o custom-columns='WORKFLOW:.metadata.labels.workflows\.argoproj\.io/workflow,CLAIMS:.spec.volumes[*].persistentVolumeClaim.claimName'
  2. Stop each workflow that lists
    azurefile-sync-share-default
    :
    kubectl -n <namespace> patch workflow <workflow> --type merge -p '{"spec":{"shutdown":"Terminate"}}'
    If a
    built-in-3d-data-streaming-cleanup-…
    workflow is listed, stop it too. It's the hourly scratch cleanup, and no later cleanup runs until it ends.
  3. Run the transformation of each workflow that you stopped again. Don't run the cleanup workflow again, because the next hourly run does the cleanup.

Create a volume for the renamed transformation workspace

This release renames the shared volume claim that transformations use as scratch space from
azurefile-sync-share-default
to
sync-share-default
, and removes the unused
efs-claim
.
If the storage class that
configuration.kubernetes.storage.readWriteManyStorageClass
names provisions volumes automatically, skip this item.
If you create that class's PersistentVolumes by hand, create a PersistentVolume for the new claim before you regenerate and deploy the charts during the upgrade. The volume must meet the following requirements:
  • It's pre-bound to the
    sync-share-default
    claim, through
    claimRef
    .
  • It's at least as large as the claim requests:
    100Gi
    , unless your manifest overrides it.
  • It uses new, empty storage, such as a new access point, export, or file share.
For example:
apiVersion: v1kind: PersistentVolumemetadata: name: sync-share-default-<namespace>spec: capacity: storage: 100Gi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain storageClassName: <your readWriteManyStorageClass> claimRef: namespace: <namespace> name: sync-share-default # Add the same kind of volume source as the volume of azurefile-sync-share-default, # pointing at the new storage.
If your manifest pins the claim to a volume with
configuration.overrides.cluster-config.values.common-chart.pvc.volumeName
, set that field to the new volume's name.
After the upgrade completes, clean up the old storage:
  1. If a GitOps tool manages your deployment with pruning turned off, delete the
    azurefile-sync-share-default
    and
    efs-claim
    claims by hand.
  2. Delete the volumes that you created for
    azurefile-sync-share-default
    and
    efs-claim
    . They're in the
    Released
    state.
  3. Delete the access point, export, or file share behind each of those volumes, unless another volume still uses it.

Replace the automation-manager GPU override with the manifest field

This item applies to deployments whose manifest sets
env.vpcGpuAvailable
under
configuration.overrides.automation-manager
. Every other deployment skips it.
The manifest field
configuration.transformations.gpu.enabled
turns on GPU transformations and makes pipelines offer the GPU configuration profiles. The field replaces the
env.vpcGpuAvailable
override, which
shdctl release generate
refuses at step 7.
To replace the override with the field, follow these steps when you update the manifest version during the upgrade:
  1. Note the value of
    env.vpcGpuAvailable
    .
  2. Delete
    env.vpcGpuAvailable
    from
    configuration.overrides.automation-manager
    .
  3. Set the field to match the value that you noted:
    • If the value was
      true
      , set the field to
      true
      :
      configuration: transformations: gpu: enabled: true
    • If the value was
      false
      , make sure the field is
      false
      or absent (
      false
      is the default). Check this even if the field is already set, because without the override, a field set to
      true
      turns GPU transformations on.

Set Grafana's admin password

A deployment without
configuration.monitoring.prometheus.enabled
skips this item.
Grafana's admin password comes from the
grafana-credentials
secret that
shdctl secret generate
creates. But Grafana keeps the password it already has, which is
prom-operator
unless you changed it. Until the two passwords match, Grafana's dashboard and datasource updates fail. Their failed sign-ins lock the
admin
account for 5 minutes at a time.
Choose whether to switch to the generated password (recommended) or keep your current password. Then, to set Grafana's admin password, follow these steps:
  1. When you regenerate and deploy the secrets during the upgrade, before
    shdctl secret generate
    , add the following lines to
    secrets.import.yaml
    , if they apply:
    • admin-user
      : if your Grafana admin user isn't
      admin
      , its name. Grafana keeps this user name, and its updates sign in with it.
    • admin-password
      : if you keep your current password, that password.
    grafana-credentials: admin-user: "<your-admin-user>" admin-password: "<your-current-password>"
    If neither line applies, skip this step.
  2. If you switch to the generated password, set Grafana's password to the password in the secret when you verify the upgrade, as soon as the charts are deployed:
    kubectl -n <namespace> get secret grafana-credentials -o jsonpath='{.data.admin-password}' | base64 -d | kubectl -n <namespace> exec -i statefulset/kube-prometheus-stack-grafana -c grafana -- grafana cli --config /etc/grafana/grafana.ini admin reset-admin-password --password-from-stdin
    If the account is already locked, sign in 5 minutes later.

Add missing RabbitMQ queue replicas before you upgrade

This item applies to every deployment that doesn't set
configuration.infrastructure.singleNode: true
.
In this release, RabbitMQ keeps a replica of every queue on each of its nodes. When you deploy the release, RabbitMQ restarts once, one pod at a time.
A queue that was created while one of the RabbitMQ nodes was still starting can be missing a replica. A RabbitMQ pod that holds such a queue doesn't stop. It waits up to seven days for the replica. The deployment then leaves that pod in
Terminating
, and node upgrades and drains on its node fail until the pod stops. From this release on, RabbitMQ adds missing replicas itself.
To check for missing queue replicas and add them, follow these steps before you start the upgrade. Follow them again before any Kubernetes or node upgrade of your cluster.
  1. Make sure that all the RabbitMQ pods are
    Running
    and ready.
  2. Check each RabbitMQ pod:
    replicas=$(kubectl -n <namespace> get statefulset rabbitmq-asset-solutions-server -o jsonpath='{.spec.replicas}')for i in $(seq 0 $((replicas - 1))); do kubectl -n <namespace> exec rabbitmq-asset-solutions-server-$i -c rabbitmq -- rabbitmq-queues check_if_node_is_quorum_criticaldone
    A pod that reports
    no queues/streams with minimum quorum
    is fine. If every pod reports it, skip the rest of these steps.
  3. If a pod lists a queue that
    will become unavailable
    , add the missing replicas on every node:
    replicas=$(kubectl -n <namespace> get statefulset rabbitmq-asset-solutions-server -o jsonpath='{.spec.replicas}')for i in $(seq 0 $((replicas - 1))); do kubectl -n <namespace> exec rabbitmq-asset-solutions-server-0 -c rabbitmq -- rabbitmq-queues grow rabbit@rabbitmq-asset-solutions-server-$i.rabbitmq-asset-solutions-nodes.<namespace> alldone
  4. Run the check from step 2 again.
If a RabbitMQ pod is already stuck in
Terminating
, add the missing replicas with the command from step 3. The pod finishes stopping within a minute. For more information, refer to A RabbitMQ pod is stuck in Terminating.

Check for PostgreSQL, Elasticsearch, and Garage pods that share a node before you upgrade

This item applies to every deployment that doesn't set
configuration.infrastructure.singleNode: true
.
In this release, each of the following pods must run on a different node from the other pods of the same kind:
  • Each PostgreSQL pod.
  • Each Elasticsearch master node.
  • Each Elasticsearch data node.
  • Each Garage pod, if you run the in-cluster object store (
    configuration.objectStore.provider
    isn't
    s3
    ).
When you deploy the release, each of these pods restarts once, one pod at a time. If two pods of the same kind already share a node, the restarted pod must move to another node, but its volume might not be able to follow it:
  • A local-path volume stays on its node.
  • A zonal volume needs a node with room in its zone.
If the volume can't follow, the pod stays in the
Pending
state, and the upgrade doesn't finish. To prevent this, check for pods that share a node before you start the upgrade, and move the pods that can't follow their volume.

Check whether pods share a node

To check whether pods of the same kind share a node, follow these steps:
  1. List the pods and their nodes:
    kubectl -n <namespace> get pods -l postgres-operator.crunchydata.com/data=postgres -o custom-columns='POD:.metadata.name,ROLE:.metadata.labels.postgres-operator\.crunchydata\.com/role,NODE:.spec.nodeName'kubectl -n <namespace> get pods -l elasticsearch.k8s.elastic.co/cluster-name=elastic-eck-cluster -o custom-columns='POD:.metadata.name,NODESET:.metadata.labels.elasticsearch\.k8s\.elastic\.co/statefulset-name,NODE:.spec.nodeName'kubectl -n <namespace> get pods -l app.kubernetes.io/name=garage -o custom-columns='POD:.metadata.name,NODE:.spec.nodeName'
  2. Look for two pods of the same kind on one node. Two pods are the same kind if they're both PostgreSQL pods, both Garage pods, or both Elasticsearch pods with the same
    NODESET
    . An Elasticsearch master and an Elasticsearch data node can share a node.
    If no two pods of the same kind share a node, skip the rest of this item.
  3. List where each volume can go:
    kubectl get pv -o custom-columns='CLAIM:.spec.claimRef.name,PINNED_TO:.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values'
  4. For each pod that shares a node, check whether it can follow its volume:
    • A volume pinned to nothing (
      []
      ) can follow its pod to any node.
    • A volume pinned to a zone can follow its pod to another node in that zone. That zone needs a node that runs no pod of the same kind, or your cluster must add nodes on demand.
    • A volume pinned to a node, such as a local-path volume, can't follow its pod.
    If every pod that shares a node can follow its volume, skip the rest of this item, because the upgrade moves the pods. If a pod stays
    Pending
    during the upgrade anyway, add a node in its volume's zone.
  5. Move each pod that can't follow its volume, as the section for its component describes: PostgreSQL, Elasticsearch, or Garage.
  6. List the pods and their nodes again, as in step 1. Start the upgrade only after no two pods of the same kind share a node.

Move PostgreSQL replicas

Move each PostgreSQL replica that shares a node to a node that runs no PostgreSQL pod, one replica at a time. The operator recreates the replica with a new volume. A replica's volume holds nothing that the primary lacks, because the recreated replica copies its data back from the primary or the backup.
Discard only a replica's volume, never the primary's. Don't rely on Patroni maintenance mode (
patronictl pause
) to protect the primary, because the operator turns it off again within seconds.
To move a PostgreSQL replica, follow these steps:
  1. Cordon every node that already runs a PostgreSQL pod, so that the recreated replica lands on another node.
  2. In the primary pod's
    database
    container, run
    patronictl list
    . Check that the pod you move is a replica, and that another replica shows
    streaming
    , so that the primary never holds the only live copy.
  3. Delete the replica's claim and its pod in a single command, as Percona's Reinitialize replicas guide describes.
  4. After the new pod is scheduled, uncordon the nodes. If the replica stays
    Pending
    , no other node has room, so add one.
  5. Wait until the recreated replica shows
    streaming
    with a lag of 0 in
    patronictl list
    . The replica first restores the whole database from the backup, so the wait depends on the size of the database. It takes minutes for a few GB, and close to an hour for a few hundred GB.
  6. If the storage class keeps released volumes (
    reclaimPolicy: Retain
    ), the old volume stays behind as a
    Released
    PersistentVolume, with its disk. Delete the PersistentVolume and its disk.
Repeat these steps for the next replica that shares a node.

Move Elasticsearch pods

Move pods off each shared node, one at a time, until no two pods with the same
NODESET
share a node. For example, three masters on one node take two moves. The recreated pod rejoins the cluster with an empty volume. A master gets the cluster state back from the other two masters, and a data node gets each shard back from its replica on the other data node.
The checks in the following steps query Elasticsearch as its
elastic
user. To read the password, run
kubectl -n <namespace> get secret elastic-eck-cluster-es-elastic-user -o jsonpath='{.data.elastic}' | base64 -d
. Enter the password when
curl
asks for it.
To move an Elasticsearch pod, follow these steps:
  1. Check that
    kubectl -n <namespace> get elasticsearch elastic-eck-cluster
    shows
    green
    health.
  2. Check that the following command lists all five Elasticsearch pods, with the three masters showing an
    m
    role:
    kubectl -n <namespace> exec -it elastic-eck-cluster-es-data-nodes-0 -c elasticsearch -- curl -s -u elastic 'http://localhost:9200/_cat/nodes?h=name,node.role'
    The masters hold no shards, so
    green
    alone doesn't show that a recreated master has joined. If you remove a second master before then, the cluster can be left without a master.
  3. If you move a data node, check that every index has a replica. The following command lists the indices with the fewest replicas first, and every row must start with
    1
    or more:
    kubectl -n <namespace> exec -it elastic-eck-cluster-es-data-nodes-0 -c elasticsearch -- curl -s -u elastic 'http://localhost:9200/_cat/indices?h=rep,health,index&s=rep'
    Don't move a data node while an index shows
    0
    , because the index's only copy goes with the volume.
  4. Cordon every node that runs a pod of the same
    NODESET
    , including the node that the pod shares, so that the recreated pod lands on another node.
  5. Delete the pod's claim and the pod in a single command, for example
    kubectl -n <namespace> delete pvc/elasticsearch-data-<pod> pod/<pod>
    . If the pod stays
    Pending
    , no other node has room, so add one.
  6. After the recreated pod is back, uncordon the nodes.
  7. If the storage class keeps released volumes (
    reclaimPolicy: Retain
    ), the old volume stays behind as a
    Released
    PersistentVolume, with its disk. After the cluster passes the checks in steps 1 to 3 again, delete the PersistentVolume and its disk.
Repeat these steps for the next move.

Move Garage pods

Move pods off the shared node, one at a time, until each Garage pod is on a different node. To move a pod, recreate it with new volumes, and give its place in the Garage cluster layout to the new Garage node that it starts as. By default, Garage keeps a copy of every object on each of three pods, so the recreated pod gets its objects back from the others.
For a few minutes after each recreated pod starts, some S3 requests fail while the pod catches up, so move Garage pods in a maintenance window. Run the
garage
commands in a Garage pod that you aren't moving, for example
kubectl -n <namespace> exec garage-0 -c garage -- /garage status
.
If you set
configuration.infrastructure.components.garage.replicationFactor: 1
, follow the steps in Move Garage pods with a replication factor of 1 instead.
To move a Garage pod, follow these steps:
  1. Check that
    /garage status
    lists every Garage pod as healthy, and that
    /garage layout history
    reports a single live layout version. Note the node ID shown for the pod that you move.
  2. Cordon every node that runs a Garage pod, including the node that the pod shares, so that the recreated pod lands on another node.
  3. Delete the pod's claims and the pod in a single command:
    kubectl -n <namespace> delete pvc/meta-<pod> pvc/data-<pod> pod/<pod>
    . If the pod stays
    Pending
    , no other node has room, so add one.
  4. Wait until
    /garage status
    lists the pod under a new node ID marked
    NO ROLE ASSIGNED
    , and lists the ID that you noted under
    FAILED NODES
    . This takes a minute or two after the pod is
    Running
    . Until then, the pod's name can appear twice. The new ID is the one that you didn't note.
  5. In
    /garage layout show
    , read the zone and capacity of the old ID.
  6. Give the old node's place to the new one:
    /garage layout assign <new-id> --replace <old-id> -z <zone> -c <capacity>
    .
  7. Run
    /garage layout show
    to review the change, then apply it with the command that it prints,
    /garage layout apply --version <version>
    .
  8. Uncordon the nodes, and wait until
    /garage stats -a
    shows a
    resync queue length
    of 0 on every node. The queue can stay unchanged for 10 minutes before it starts to fall.
  9. Close the previous layout version, which otherwise waits forever for the deleted node. Run
    /garage layout skip-dead-nodes --version <version>
    , with the version that you applied at step 7. Don't add
    --allow-missing-data
    .
  10. Check that
    /garage layout history
    reports a single live layout version again. Move the next pod only after this step. With two layout versions waiting for deleted nodes, Garage loses its quorum, and every S3 request fails.
  11. If the storage class keeps released volumes (
    reclaimPolicy: Retain
    ), the two old volumes stay behind as
    Released
    PersistentVolumes, with their disks. Delete the PersistentVolumes and their disks.

Move Garage pods with a replication factor of 1

If you set
configuration.infrastructure.components.garage.replicationFactor: 1
, each object has a single copy, on one pod's volume. Move that copy off the pod before you delete the pod.
To move a Garage pod with a replication factor of 1, follow the steps in Move Garage pods, with these changes:
  • Between steps 1 and 2: move the pod's objects off it, as the following steps describe. Then go on to step 2 straight away, because from then on, S3 requests that reach the pod fail.
  • Step 4: the ID that you noted isn't listed under
    FAILED NODES
    , because it's no longer in the layout. The new ID is the one marked
    NO ROLE ASSIGNED
    that you didn't note.
  • Step 5: skip this step. You note the zone and capacity before step 2.
  • Step 6: run the
    assign
    command without
    --replace <old-id>
    , with the zone and capacity that you noted.
  • Steps 9 and 10: skip these steps.
To move the pod's objects off it between steps 1 and 2, follow these steps:
  1. In
    /garage layout show
    , note the pod's zone and capacity.
  2. Run
    /garage layout remove <old-id>
    .
  3. Run
    /garage layout show
    again, and then the
    /garage layout apply --version <version>
    command that it prints.
  4. Wait until
    /garage status
    doesn't show the pod as
    draining metadata..
    ,
    /garage stats -a
    shows a
    resync queue length
    of 0 on every node, and
    /garage layout history
    reports a single live layout version.

Upgrade Traefik's CRDs in Helm deployments

This item applies to deployments that use Helm (
shdctl release deploy --format helm
). A deployment that uses ArgoCD skips this item, because ArgoCD applies the CRDs on every sync.
Helm never upgrades the CustomResourceDefinitions (CRDs) that a chart installs, so a Helm deployment still has the Traefik CRDs from its first install. Traefik v3.7.16 runs with those CRDs, but the API server silently drops any newer field from your Traefik resources.
When you update the manifest version during the upgrade, list
traefik
in
deployment.helm.upgradeCrds
:
deployment: helm: upgradeCrds: [traefik]
The deploy then applies Traefik's CRDs before each Traefik upgrade, and overwrites the cluster's copies of those CRDs.
If another controller on your cluster installs Traefik's CRDs, or only an administrator can write CRDs, don't list
traefik
. Instead, apply the CRDs yourself. Refer to Keep chart CRDs up to date.

Move the Traefik settings that the new chart renames

This item applies to deployments whose manifest sets Traefik values under
configuration.overrides.traefik.values.traefik
. Every other deployment skips it.
The new Traefik chart moves the following settings. When you update the manifest version during the upgrade, move each one that you set, even if the deploy succeeds without the move:

Before

After

If you don't move it

logs.general.<setting>
log.<setting>
The deploy fails
logs.access.<setting>
, except
filters
and
fields
accessLog.<setting>
The deploy fails
logs.access.filters.statuscodes
,
.retryattempts
,
.minduration
accessLog.filters.statusCodes
,
.retryAttempts
,
.minDuration
The deploy fails
ports.<entrypoint>.tls
,
ports.<entrypoint>.middlewares
ports.<entrypoint>.http.tls
,
ports.<entrypoint>.http.middlewares
The deploy fails
ports.web.redirections
ports.web.http.redirections
The deploy fails
globalArguments
global.checkNewVersion
and
global.sendAnonymousUsage
; any other flag goes in
additionalArguments
The deploy fails
providers.file.content
as a YAML string
the same configuration as a YAML object: drop the
|
and indent it under
content
The deploy fails with a type error
logs.access.fields.general.defaultmode
,
logs.access.fields.general.names
accessLog.fields.defaultMode
,
accessLog.fields.names
The deploy fails. Moved unchanged under
accessLog
, the setting is ignored
logs.access.fields.headers.defaultmode
accessLog.fields.headers.defaultMode
The deploy fails. Moved unchanged under
accessLog
, the setting is ignored
service.type
,
service.loadBalancerClass
,
service.loadBalancerSourceRanges
the manifest fields
type
,
loadBalancerClass
, and
loadBalancerSourceRanges
under
configuration.networking.ingress.traefik
. They're not overrides: they set only what the Service type accepts
The setting is ignored
service.externalIPs
service.spec.externalIPs
The setting is ignored
A setting that fails the deploy stops it at the Traefik chart with
additional properties '<key>' not allowed
, or with a type error for
content
. An ignored setting gives no error.

Keep the Gateway API CRDs in ArgoCD deployments

This item applies to deployments that use ArgoCD. A deployment that uses Helm skips this item, because Helm leaves the CRDs in place.
In this release, Traefik doesn't install the Gateway API CustomResourceDefinitions (
gateway.networking.k8s.io
), which this deployment never used. After the upgrade, ArgoCD's next sync deletes the copies that it applied. Deleting a CRD deletes every resource of that kind in the cluster, for example the Gateways or HTTPRoutes that your own ingress controller or service mesh serves.
To keep the Gateway API CRDs, follow these steps:
  1. Check whether anything in your cluster uses the Gateway API:
    kubectl get gatewayclasses,gateways,httproutes,grpcroutes,referencegrants -A
    If the command finds no resources, nothing uses the Gateway API, so skip the rest of this item.
  2. When you regenerate and deploy the charts during the upgrade, before you push the regenerated charts, mark the CRDs so that ArgoCD keeps them:
    kubectl annotate --overwrite crd \ gatewayclasses.gateway.networking.k8s.io gateways.gateway.networking.k8s.io \ httproutes.gateway.networking.k8s.io grpcroutes.gateway.networking.k8s.io \ referencegrants.gateway.networking.k8s.io \ argocd.argoproj.io/sync-options=Prune=false argocd.argoproj.io/compare-options=IgnoreExtraneous

Route alerts through Alertmanager

This item applies to deployments that set
configuration.monitoring.prometheus.enabled: true
and have a Grafana contact point for the alerts in Grafana's Custom folder. A deployment that never set up a contact point skips this item.
This release removes those four alerts, and sends alerts through Alertmanager instead. Alertmanager routes nothing until you add a receiver. Any receiver that Alertmanager supports works, such as email, PagerDuty, Microsoft Teams, or a webhook. The following steps use a Slack incoming webhook as an example.
To route alerts through Alertmanager, follow these steps after the upgrade completes:
  1. Create a secret in
    <namespace>
    that holds your Slack webhook URL:
    kubectl -n <namespace> create secret generic alerting-slack --from-literal=url='<your Slack webhook URL>'
  2. Create an
    AlertmanagerConfig
    that sends warning and critical alerts to the webhook:
    kubectl -n <namespace> apply -f - <<'EOF'apiVersion: monitoring.coreos.com/v1alpha1kind: AlertmanagerConfigmetadata: name: alerting-slackspec: route: receiver: slack groupBy: [alertname, namespace] matchers: - name: severity matchType: "=~" value: "warning|critical" receivers: - name: slack slackConfigs: - apiURL: name: alerting-slack key: url sendResolved: trueEOF
  3. Delete the Grafana contact point, which receives nothing after the upgrade.

Version 2.0.0

Install shdctl 1.0.0 before you pull

Every deployment needs shdctl 1.0.0 or later for this release. Install it before you start: the release isn't published where vpctl looks for releases, so your current vpctl can't pull it.
Your current release ships
install-vpctl.sh
, not
install-shdctl.sh
, so install shdctl in two passes.
install-vpctl.sh
can install shdctl 1.0.0, under the name
vpctl
, and that binary can pull this release, which carries
install-shdctl.sh
. The pull reads your manifest, so if yours sets
configuration.kubernetes.docker.namespace
while
docker.repository
is Unity's registry, delete it first, as Remove settings that shdctl 1.0.0 refuses from your manifest describes. Then, from the directory that contains
extracted-release/
, the release you deployed last, run:
./extracted-release/common/scripts/install-vpctl.sh 1.0.0vpctl release pull --version 2.0.0 --clean-output./extracted-release/common/scripts/install-shdctl.sh 1.0.0
If you no longer have the release you deployed last, pull it again with
vpctl release pull
first.
install-shdctl.sh
installs
shdctl
and, on Linux and macOS, replaces
vpctl
with a link to it, so your existing scripts keep running throughout shdctl 1.x. To install elsewhere than
/usr/local/bin
, pass the same directory to both installers, as Install shdctl describes.

Automation 0.0.77: snapshot its database first

In every deployment, Automation 0.0.77 migrates its database, which the bundled PostgreSQL holds, when you deploy this release. Before you start, take a full backup of that PostgreSQL, and wait until it succeeds:
backup=$(kubectl -n <namespace> create -o name -f - <<'EOF'apiVersion: pgv2.percona.com/v2kind: PerconaPGBackupmetadata: generateName: before-upgrade-spec: pgCluster: pg-db repoName: repo1 options: - --type=fullEOF)kubectl -n <namespace> wait "$backup" --for=jsonpath='{.status.state}'=Succeeded --timeout=60m
Each run creates a new backup, so if you postpone the upgrade, run the commands again right before you start. The backup goes where your PostgreSQL backups already go: to their backup volume, or to the bucket that
configuration.objectStore.buckets.postgresBackups
names.

Remove settings that shdctl 1.0.0 refuses from your manifest

shdctl 1.0.0 refuses two settings that a manifest written for an earlier release can hold. Each refusal names the setting it stopped on.
  • configuration.kubernetes.docker.namespace
    , when
    configuration.kubernetes.docker.repository
    is Unity's registry,
    uccmpprivatecloud.azurecr.io
    . Every shdctl command that reads the manifest refuses it, including the pull in Install shdctl 1.0.0 before you pull, so delete it before you start.
  • _arrayMerge
    under
    configuration.overrides.<chart-name>.values
    .
    shdctl release generate
    refuses it when you regenerate the charts, so when you update the manifest version during the upgrade, rewrite it in the form that Per-chart Helm value overrides describes, which patches a list element by its name.

Rebuild a secrets file that vpctl persisted

If you keep your secret values in a file that vpctl wrote with
--persist
, rebuild that file from the cluster when you regenerate and deploy the secrets during the upgrade, before you generate them. A file that vpctl wrote doesn't hold the storage key of the
asset-cloud-storage-abstraction
secret (
storage-key.pem
): generating from it creates a new key, and deploying that rotates the key the cluster runs with.
Keep the vpctl file as a backup, then rebuild it in place:
shdctl secret export
replaces an existing file only with
--force
.
cp secrets.import.yaml secrets.import.vpctl.yamlshdctl secret export --output secrets.import.yaml --force
If your file has another name, use it in both commands. The export lists any value it can't read back from the cluster: take those from the backup. Then generate and deploy the secrets as Regenerate and deploy the secrets describes.

Keep an image pull secret that something else creates

shdctl secret generate
now renders the image pull secret, the one that
configuration.kubernetes.imagePullSecret
names, from the registry credentials you supply, and
shdctl secret deploy
applies it. If something else in your cluster maintains that secret, such as a token refresher for ECR's short-lived credentials, turn generation off in your manifest before you regenerate and deploy the secrets during the upgrade. Otherwise the deployed copy overwrites the live credential, and image pulls fail once it expires:
configuration: kubernetes: imagePullSecret: name: ecr-regcred generate: false
For the details, refer to The image pull secret.

Helm deployments: hand the monitoring stack over to Helm

The monitoring stack is now installed as a Helm release, like every other component. If you deploy with Helm (
shdctl release deploy --format helm
) and the monitoring stack already runs (
configuration.monitoring.prometheus.enabled: true
), hand its resources over to Helm once, when you regenerate and deploy the charts during the upgrade: after
shdctl release generate
and before
shdctl release deploy
, from the parent directory of
generated-charts/
. Otherwise the deployment stops with
invalid ownership metadata
.
helm dependency update --skip-refresh generated-charts/kube-prometheus-stackhelm template kube-prometheus-stack generated-charts/kube-prometheus-stack -n <namespace> > /tmp/monitoring-resources.yamlkubectl annotate --overwrite -f /tmp/monitoring-resources.yaml meta.helm.sh/release-name=kube-prometheus-stack meta.helm.sh/release-namespace=<namespace>kubectl label --overwrite -f /tmp/monitoring-resources.yaml app.kubernetes.io/managed-by=Helm
NotFound
messages and a non-zero exit code are expected: the commands cover every resource the release installs, including ones your deployment doesn't have yet, which the upgrade creates. Everything already running is handed over without a restart. Resources that an earlier monitoring version installed and that this release no longer includes are left as they are; you can delete them once the upgrade succeeds. A first installation and a deployment that uses ArgoCD skip this step.

ArgoCD deployments: remove the finalizer of a self-tracking application

If you deploy with ArgoCD, check when you regenerate and deploy the charts during the upgrade, before you commit the regenerated charts, whether
<pathPrefix>/asset-solutions/templates/asset-solutions.yaml
exists in your GitOps repository. If it does, the live top-level
asset-solutions
application tracks itself, and committing the regenerated charts can make ArgoCD delete every application and its workloads. Remove the finalizer from that application first:
kubectl -n <namespace> patch application asset-solutions --type merge -p '{"metadata":{"finalizers":null}}'
After you push the regenerated charts, ArgoCD can delete the
asset-solutions
application itself, while its child applications and their workloads keep running. Once ArgoCD has synced the pushed revision, a few minutes after the push, check with
kubectl -n <namespace> get application asset-solutions
. If it reports the application not found, run
shdctl release deploy --format argocd
, again if step 7 already ran it: it creates the application again, which adopts the child applications.

Update your own mirror and scripts for the
shd
package name

The release package is renamed from "onprem" to "shd", and
shdctl release pull
needs no change. If you handle the package with your own tools, update them before you pull the new release during the upgrade:
  • If you mirror or pull the release artifact yourself, switch from
    releases/onprem-configuration
    to
    releases/shd-configuration
    . This release and later ones are published only there.
  • If your scripts reference
    extracted-release/platform/onprem/…
    , change the paths to
    platform/shd/
    .
  • If your scripts extract the tarball by file name, it's now
    unity-private-cloud-shd-<version>.tar.gz
    . Update the name, or pass
    --output
    to pin one.

Set
DISPLAY
in your own action images

Automation step containers no longer receive
DISPLAY=:0
, and GPU steps no longer mount host directories. If an action image of your own relies on the platform for
DISPLAY
, set it in the image before you regenerate and deploy the charts during the upgrade. The bundled Unity Asset Manager and Unity Asset Transformer apps aren't affected.

Update callers of the Optimize and Convert pipeline

The Optimize and Convert pipeline now always creates a new asset: its
CreateNewAsset
input is removed.
Output Formats
and
Asset Version
are now required, and the output formats no longer default to
glb
and
pxz
. Update anything that calls the pipeline to pass both and to stop passing
CreateNewAsset
, before you regenerate and deploy the charts during the upgrade.

Update references to renamed ingress resources

Several
IngressRoute
resources that the release installs are renamed, and so are most of the per-route
Middleware
resources alongside them. The shared middlewares that every route chains,
cors
,
forward-auth
,
strip-backend-prefix
,
remove-authorization-header
, and
security-headers
, keep their names.

Old name

New name

asset-solutions-public
asset-solutions-client
data-streaming-public
data-streaming-client
linksharing
app-linking-client
uns
unity-notifications
uns-internal
unity-notifications-private
dt-storage-public
Removed
If something outside the release refers to one of these resources by name, such as a custom
IngressRoute
that reuses a generated middleware, a NetworkPolicy, or a dashboard or alert keyed on a resource name, update it to the new name when you deploy at step 7. The deployment removes the resources under the old names. If a GitOps tool manages your deployment with pruning turned off, remove them by hand once the new ones serve traffic.

Version 0.16.0

Add the
basic
client scope to the
sdk
,
dashboard
and
mini-usf
clients

Without the
basic
client scope, the access tokens that the
sdk
,
dashboard
, and
mini-usf
clients receive carry no
sub
claim, and user-scoped API calls fail. A new deployment gets the scope from its realm import, but an existing
unity
realm isn't imported again, so add it by hand after you deploy:
  1. In the Keycloak Admin console, open the unity realm, then Clients.
  2. For each of sdk, dashboard, and mini-usf, open Client scopes, select Add client scope, choose basic, and add it as Default.
No redeployment is needed: new sign-ins receive the
sub
claim at once.

Automation 0.0.73: snapshot its database first

This item applies to every deployment. When you deploy this release, Pipeline Automation 0.0.73 migrates its database in the bundled PostgreSQL.
Before you start the upgrade, take a full backup of the bundled PostgreSQL, as Automation 0.0.77: snapshot its database first shows. If you upgrade past 0.16.0 and 2.0.0 at once, one backup before you start covers both.

Version 0.15.0

Import the
unity-licensing-server
Keycloak client

The Unity Licensing Server authenticates against a
unity-licensing-server
Keycloak client. A new deployment gets the client from its realm import, but an existing
unity
realm isn't imported again, so add it by hand after you deploy. From the extracted release, with
kubectl
access to the cluster, generate the import file, replacing
<namespace>
with your deployment's namespace:
python3 common/scripts/upc-cli/upc-cli.py --fqdn <your-domain> --no-auth keycloak generate-client-json \ --namespace <namespace> --output unity-licensing-server-client.json
The CLI needs Python 3 with the
requests
and
keyring
packages (
pip install requests keyring
).
Then, in the Keycloak Admin console, open the unity realm, then Clients, select Import client, browse to
unity-licensing-server-client.json
, and select Save. For managing the licensing server, refer to Licensing.

Version 0.14.0

Set
imageVariant: hardened
to keep hardened images

Releases before 0.14.0 deployed hardened images unconditionally. From 0.14.0 on, a deployment gets them only when its manifest asks for them, and the base images otherwise. To keep hardened images, add this to your manifest when you update the manifest version during the upgrade, before you generate the charts:
configuration: imageVariant: hardened

Version 0.13.0

Move from rustfs to garage

Garage replaces RustFS as the in-cluster object store, and runs as a three-node cluster. Before you generate the charts, replace
configuration.infrastructure.components.rustfs
in your manifest with
configuration.infrastructure.components.garage
, which takes
metaStorage
,
dataStorage
,
replicas
, and
replicationFactor
. Then pick one of two paths before you regenerate and deploy the secrets during the upgrade. In every command, replace
<namespace>
with your deployment's namespace.

Path A: start the object store empty

The objects in RustFS aren't kept: Argo workflow artifacts are temporary, MongoDB backups are taken again at the next scheduled run, and Asset Manager ingests again on demand. Deploy the release, and the
garage-bucket-bootstrap
job creates the buckets and imports your keys.

Path B: keep the objects

The release carries
common/scripts/migrate-rustfs-to-garage.sh
, which copies the
argo-artifacts
and
psmdb-backups
buckets out of RustFS and into Garage. It needs
kubectl
and the MinIO client,
mc
. Run the commands below from the extracted release.
  1. Before you regenerate the secrets during the upgrade, from a workstation that can reach the cluster, back the buckets up. The regenerated secrets no longer hold the rustfs keys this step reads:
    export RUSTFS_ACCESS_KEY=$(kubectl -n <namespace> get secret s3-api-storage-credentials -o jsonpath='{.data.RUSTFS_ACCESS_KEY}' | base64 -d)export RUSTFS_SECRET_KEY=$(kubectl -n <namespace> get secret s3-api-storage-credentials -o jsonpath='{.data.RUSTFS_SECRET_KEY}' | base64 -d)NAMESPACE=<namespace> BACKUP_DIR=./rustfs-backup ./common/scripts/migrate-rustfs-to-garage.sh backup
  2. Deploy, then wait for the garage StatefulSet to be ready and for the
    garage-bucket-bootstrap
    job to complete.
  3. Restore the buckets into Garage:
    export GARAGE_ACCESS_KEY=$(kubectl -n <namespace> get secret s3-api-storage-credentials -o jsonpath='{.data.GARAGE_ACCESS_KEY}' | base64 -d)export GARAGE_SECRET_KEY=$(kubectl -n <namespace> get secret s3-api-storage-credentials -o jsonpath='{.data.GARAGE_SECRET_KEY}' | base64 -d)NAMESPACE=<namespace> BACKUP_DIR=./rustfs-backup ./common/scripts/migrate-rustfs-to-garage.sh restore

Reclaim the rustfs volumes

On either path, removing RustFS doesn't remove its volumes: they stay bound, unused, and billed. Once the upgrade has finished and no rustfs pod remains, delete them; on path B, check the restored data first.
kubectl -n <namespace> get pvc | grep rustfskubectl -n <namespace> get pvc -o name | grep rustfs | xargs -r kubectl -n <namespace> delete
If you deploy with Helm rather than ArgoCD, also run
helm -n <namespace> uninstall rustfs
.