Migration guide
The manual steps each Self-Hosted Deployment release needs beyond the standard upgrade
阅读时间34 分钟最后更新于 13 天前
The manual steps an existing Self-Hosted Deployment needs when it upgrades past a release, beyond the standard upgrade. The newest release comes first, and a release that needs no such step has no section. For everything else each release changes, refer to the release notes.
When you upgrade across several releases, complete the section of every release after your current one, up to and including your target, oldest first. Each item says which deployments it applies to and at which step of the upgrade to complete it. In every command, is your deployment's namespace, the one names.
<namespace>configuration.kubernetes.namespaceVersion 2.1.0
Hand the service accounts over to Helm
If you deploy with Helm (), complete this item once, when you regenerate and deploy the charts during the upgrade, before . A first installation and a deployment that uses ArgoCD skip this item.
shdctl release deploy --format helmshdctl release deployIn this release, the platform services' service accounts are part of their Helm releases, so an upgrade doesn't re-create them. Hand the existing service accounts over to Helm. Otherwise, the deployment stops with .
invalid ownership metadatakubectl -n <namespace> get serviceaccounts -l creator-services-app=asset-solutions -o jsonpath='{range .items[?(@.metadata.annotations.helm\.sh/hook)]}{.metadata.name} {.metadata.labels.app\.kubernetes\.io/instance}{"\n"}{end}' > /tmp/service-accounts.txtwhile read -r sa release; do kubectl -n <namespace> annotate --overwrite serviceaccount "$sa" meta.helm.sh/release-name="$release" meta.helm.sh/release-namespace=<namespace> helm.sh/hook- helm.sh/hook-weight-; done < /tmp/service-accounts.txt
The first command lists the service accounts to hand over, each with its release. The second command hands them over without restarting anything.
If you later deploy an earlier release over this one, that deployment deletes these service accounts. The platform services can't start new pods until you deploy the earlier release a second time, which restores them.
Automation 0.0.81: snapshot its database first
This item applies to every deployment. When you deploy this release, Pipeline Automation 0.0.81 migrates its database in the bundled PostgreSQL. Before you start the upgrade, take a full backup of the bundled PostgreSQL, and wait until the backup succeeds:
backup=$(kubectl -n <namespace> create -o name -f - <<'EOF'apiVersion: pgv2.percona.com/v2kind: PerconaPGBackupmetadata: generateName: before-upgrade-spec: pgCluster: pg-db repoName: repo1 options: - --type=fullEOF)kubectl -n <namespace> wait "$backup" --for=jsonpath='{.status.state}'=Succeeded --timeout=60m
Each run creates a new backup, so if you postpone the upgrade, run the commands again right before you start. The backup goes where your PostgreSQL backups already go: to their backup volume, or to the bucket that names.
configuration.objectStore.buckets.postgresBackupsLet running transformations finish before you deploy
This item applies to every deployment. This release replaces the shared volume that transformations use as scratch space. If a transformation is running when you deploy the charts, or starts during the deployment, its pod can stay on the old volume, . The pod never starts and never fails.
Pendingazurefile-sync-share-defaultUpgrade while no transformation is running. The old volume held only the files of transformations in progress, so you don't need to copy anything.
To clear the pods that still wait on the old volume after the upgrade completes, follow these steps:
-
List the pods that are still waiting and the volumes they claim:kubectl -n <namespace> get pods --field-selector=status.phase=Pending \ -o custom-columns='WORKFLOW:.metadata.labels.workflows\.argoproj\.io/workflow,CLAIMS:.spec.volumes[*].persistentVolumeClaim.claimName'
-
Stop each workflow that lists:
azurefile-sync-share-defaultkubectl -n <namespace> patch workflow <workflow> --type merge -p '{"spec":{"shutdown":"Terminate"}}'If aworkflow is listed, stop it too. It's the hourly scratch cleanup, and no later cleanup runs until it ends.built-in-3d-data-streaming-cleanup-… -
Run the transformation of each workflow that you stopped again. Don't run the cleanup workflow again, because the next hourly run does the cleanup.
Create a volume for the renamed transformation workspace
This release renames the shared volume claim that transformations use as scratch space from to , and removes the unused .
azurefile-sync-share-defaultsync-share-defaultefs-claimIf the storage class that names provisions volumes automatically, skip this item.
configuration.kubernetes.storage.readWriteManyStorageClassIf you create that class's PersistentVolumes by hand, create a PersistentVolume for the new claim before you regenerate and deploy the charts during the upgrade. The volume must meet the following requirements:
- It's pre-bound to the claim, through
sync-share-default.claimRef - It's at least as large as the claim requests: , unless your manifest overrides it.
100Gi - It uses new, empty storage, such as a new access point, export, or file share.
For example:
apiVersion: v1kind: PersistentVolumemetadata: name: sync-share-default-<namespace>spec: capacity: storage: 100Gi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain storageClassName: <your readWriteManyStorageClass> claimRef: namespace: <namespace> name: sync-share-default # Add the same kind of volume source as the volume of azurefile-sync-share-default, # pointing at the new storage.
If your manifest pins the claim to a volume with , set that field to the new volume's name.
configuration.overrides.cluster-config.values.common-chart.pvc.volumeNameAfter the upgrade completes, clean up the old storage:
- If a GitOps tool manages your deployment with pruning turned off, delete the and
azurefile-sync-share-defaultclaims by hand.efs-claim - Delete the volumes that you created for and
azurefile-sync-share-default. They're in theefs-claimstate.Released - Delete the access point, export, or file share behind each of those volumes, unless another volume still uses it.
Replace the automation-manager GPU override with the manifest field
This item applies to deployments whose manifest sets under . Every other deployment skips it.
env.vpcGpuAvailableconfiguration.overrides.automation-managerThe manifest field turns on GPU transformations and makes pipelines offer the GPU configuration profiles. The field replaces the override, which refuses at step 7.
configuration.transformations.gpu.enabledenv.vpcGpuAvailableshdctl release generateTo replace the override with the field, follow these steps when you update the manifest version during the upgrade:
-
Note the value of.
env.vpcGpuAvailable -
Deletefrom
env.vpcGpuAvailable.configuration.overrides.automation-manager -
Set the field to match the value that you noted:
-
If the value was, set the field to
true:trueconfiguration: transformations: gpu: enabled: true -
If the value was, make sure the field is
falseor absent (falseis the default). Check this even if the field is already set, because without the override, a field set tofalseturns GPU transformations on.true
-
Set Grafana's admin password
A deployment without skips this item.
configuration.monitoring.prometheus.enabledGrafana's admin password comes from the secret that creates. But Grafana keeps the password it already has, which is unless you changed it. Until the two passwords match, Grafana's dashboard and datasource updates fail. Their failed sign-ins lock the account for 5 minutes at a time.
grafana-credentialsshdctl secret generateprom-operatoradminChoose whether to switch to the generated password (recommended) or keep your current password. Then, to set Grafana's admin password, follow these steps:
-
When you regenerate and deploy the secrets during the upgrade, before, add the following lines to
shdctl secret generate, if they apply:secrets.import.yaml- : if your Grafana admin user isn't
admin-user, its name. Grafana keeps this user name, and its updates sign in with it.admin - : if you keep your current password, that password.
admin-password
grafana-credentials: admin-user: "<your-admin-user>" admin-password: "<your-current-password>"If neither line applies, skip this step. -
If you switch to the generated password, set Grafana's password to the password in the secret when you verify the upgrade, as soon as the charts are deployed:kubectl -n <namespace> get secret grafana-credentials -o jsonpath='{.data.admin-password}' | base64 -d | kubectl -n <namespace> exec -i statefulset/kube-prometheus-stack-grafana -c grafana -- grafana cli --config /etc/grafana/grafana.ini admin reset-admin-password --password-from-stdinIf the account is already locked, sign in 5 minutes later.
Add missing RabbitMQ queue replicas before you upgrade
This item applies to every deployment that doesn't set .
configuration.infrastructure.singleNode: trueIn this release, RabbitMQ keeps a replica of every queue on each of its nodes. When you deploy the release, RabbitMQ restarts once, one pod at a time.
A queue that was created while one of the RabbitMQ nodes was still starting can be missing a replica. A RabbitMQ pod that holds such a queue doesn't stop. It waits up to seven days for the replica. The deployment then leaves that pod in , and node upgrades and drains on its node fail until the pod stops. From this release on, RabbitMQ adds missing replicas itself.
TerminatingTo check for missing queue replicas and add them, follow these steps before you start the upgrade. Follow them again before any Kubernetes or node upgrade of your cluster.
-
Make sure that all the RabbitMQ pods areand ready.
Running -
Check each RabbitMQ pod:replicas=$(kubectl -n <namespace> get statefulset rabbitmq-asset-solutions-server -o jsonpath='{.spec.replicas}')for i in $(seq 0 $((replicas - 1))); do kubectl -n <namespace> exec rabbitmq-asset-solutions-server-$i -c rabbitmq -- rabbitmq-queues check_if_node_is_quorum_criticaldoneA pod that reportsis fine. If every pod reports it, skip the rest of these steps.
no queues/streams with minimum quorum -
If a pod lists a queue that, add the missing replicas on every node:
will become unavailablereplicas=$(kubectl -n <namespace> get statefulset rabbitmq-asset-solutions-server -o jsonpath='{.spec.replicas}')for i in $(seq 0 $((replicas - 1))); do kubectl -n <namespace> exec rabbitmq-asset-solutions-server-0 -c rabbitmq -- rabbitmq-queues grow rabbit@rabbitmq-asset-solutions-server-$i.rabbitmq-asset-solutions-nodes.<namespace> alldone -
Run the check from step 2 again.
If a RabbitMQ pod is already stuck in , add the missing replicas with the command from step 3. The pod finishes stopping within a minute. For more information, refer to A RabbitMQ pod is stuck in Terminating.
TerminatingCheck for PostgreSQL, Elasticsearch, and Garage pods that share a node before you upgrade
This item applies to every deployment that doesn't set .
configuration.infrastructure.singleNode: trueIn this release, each of the following pods must run on a different node from the other pods of the same kind:
- Each PostgreSQL pod.
- Each Elasticsearch master node.
- Each Elasticsearch data node.
- Each Garage pod, if you run the in-cluster object store (isn't
configuration.objectStore.provider).s3
When you deploy the release, each of these pods restarts once, one pod at a time. If two pods of the same kind already share a node, the restarted pod must move to another node, but its volume might not be able to follow it:
- A local-path volume stays on its node.
- A zonal volume needs a node with room in its zone.
If the volume can't follow, the pod stays in the state, and the upgrade doesn't finish. To prevent this, check for pods that share a node before you start the upgrade, and move the pods that can't follow their volume.
PendingCheck whether pods share a node
To check whether pods of the same kind share a node, follow these steps:
-
List the pods and their nodes:kubectl -n <namespace> get pods -l postgres-operator.crunchydata.com/data=postgres -o custom-columns='POD:.metadata.name,ROLE:.metadata.labels.postgres-operator\.crunchydata\.com/role,NODE:.spec.nodeName'kubectl -n <namespace> get pods -l elasticsearch.k8s.elastic.co/cluster-name=elastic-eck-cluster -o custom-columns='POD:.metadata.name,NODESET:.metadata.labels.elasticsearch\.k8s\.elastic\.co/statefulset-name,NODE:.spec.nodeName'kubectl -n <namespace> get pods -l app.kubernetes.io/name=garage -o custom-columns='POD:.metadata.name,NODE:.spec.nodeName'
-
Look for two pods of the same kind on one node. Two pods are the same kind if they're both PostgreSQL pods, both Garage pods, or both Elasticsearch pods with the same. An Elasticsearch master and an Elasticsearch data node can share a node.
NODESETIf no two pods of the same kind share a node, skip the rest of this item. -
List where each volume can go:kubectl get pv -o custom-columns='CLAIM:.spec.claimRef.name,PINNED_TO:.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values'
-
For each pod that shares a node, check whether it can follow its volume:
- A volume pinned to nothing () can follow its pod to any node.
[] - A volume pinned to a zone can follow its pod to another node in that zone. That zone needs a node that runs no pod of the same kind, or your cluster must add nodes on demand.
- A volume pinned to a node, such as a local-path volume, can't follow its pod.
If every pod that shares a node can follow its volume, skip the rest of this item, because the upgrade moves the pods. If a pod staysduring the upgrade anyway, add a node in its volume's zone.Pending - A volume pinned to nothing (
-
Move each pod that can't follow its volume, as the section for its component describes: PostgreSQL, Elasticsearch, or Garage.
-
List the pods and their nodes again, as in step 1. Start the upgrade only after no two pods of the same kind share a node.
Move PostgreSQL replicas
Move each PostgreSQL replica that shares a node to a node that runs no PostgreSQL pod, one replica at a time. The operator recreates the replica with a new volume. A replica's volume holds nothing that the primary lacks, because the recreated replica copies its data back from the primary or the backup.
Discard only a replica's volume, never the primary's. Don't rely on Patroni maintenance mode () to protect the primary, because the operator turns it off again within seconds.
patronictl pauseTo move a PostgreSQL replica, follow these steps:
- Cordon every node that already runs a PostgreSQL pod, so that the recreated replica lands on another node.
- In the primary pod's container, run
database. Check that the pod you move is a replica, and that another replica showspatronictl list, so that the primary never holds the only live copy.streaming - Delete the replica's claim and its pod in a single command, as Percona's Reinitialize replicas guide describes.
- After the new pod is scheduled, uncordon the nodes. If the replica stays , no other node has room, so add one.
Pending - Wait until the recreated replica shows with a lag of 0 in
streaming. The replica first restores the whole database from the backup, so the wait depends on the size of the database. It takes minutes for a few GB, and close to an hour for a few hundred GB.patronictl list - If the storage class keeps released volumes (), the old volume stays behind as a
reclaimPolicy: RetainPersistentVolume, with its disk. Delete the PersistentVolume and its disk.Released
Repeat these steps for the next replica that shares a node.
Move Elasticsearch pods
Move pods off each shared node, one at a time, until no two pods with the same share a node. For example, three masters on one node take two moves. The recreated pod rejoins the cluster with an empty volume. A master gets the cluster state back from the other two masters, and a data node gets each shard back from its replica on the other data node.
NODESETThe checks in the following steps query Elasticsearch as its user. To read the password, run . Enter the password when asks for it.
elastickubectl -n <namespace> get secret elastic-eck-cluster-es-elastic-user -o jsonpath='{.data.elastic}' | base64 -dcurlTo move an Elasticsearch pod, follow these steps:
-
Check thatshows
kubectl -n <namespace> get elasticsearch elastic-eck-clusterhealth.green -
Check that the following command lists all five Elasticsearch pods, with the three masters showing anrole:
mkubectl -n <namespace> exec -it elastic-eck-cluster-es-data-nodes-0 -c elasticsearch -- curl -s -u elastic 'http://localhost:9200/_cat/nodes?h=name,node.role'The masters hold no shards, soalone doesn't show that a recreated master has joined. If you remove a second master before then, the cluster can be left without a master.green -
If you move a data node, check that every index has a replica. The following command lists the indices with the fewest replicas first, and every row must start withor more:
1kubectl -n <namespace> exec -it elastic-eck-cluster-es-data-nodes-0 -c elasticsearch -- curl -s -u elastic 'http://localhost:9200/_cat/indices?h=rep,health,index&s=rep'Don't move a data node while an index shows, because the index's only copy goes with the volume.0 -
Cordon every node that runs a pod of the same, including the node that the pod shares, so that the recreated pod lands on another node.
NODESET -
Delete the pod's claim and the pod in a single command, for example. If the pod stays
kubectl -n <namespace> delete pvc/elasticsearch-data-<pod> pod/<pod>, no other node has room, so add one.Pending -
After the recreated pod is back, uncordon the nodes.
-
If the storage class keeps released volumes (), the old volume stays behind as a
reclaimPolicy: RetainPersistentVolume, with its disk. After the cluster passes the checks in steps 1 to 3 again, delete the PersistentVolume and its disk.Released
Repeat these steps for the next move.
Move Garage pods
Move pods off the shared node, one at a time, until each Garage pod is on a different node. To move a pod, recreate it with new volumes, and give its place in the Garage cluster layout to the new Garage node that it starts as. By default, Garage keeps a copy of every object on each of three pods, so the recreated pod gets its objects back from the others.
For a few minutes after each recreated pod starts, some S3 requests fail while the pod catches up, so move Garage pods in a maintenance window. Run the commands in a Garage pod that you aren't moving, for example .
garagekubectl -n <namespace> exec garage-0 -c garage -- /garage statusIf you set , follow the steps in Move Garage pods with a replication factor of 1 instead.
configuration.infrastructure.components.garage.replicationFactor: 1To move a Garage pod, follow these steps:
- Check that lists every Garage pod as healthy, and that
/garage statusreports a single live layout version. Note the node ID shown for the pod that you move./garage layout history - Cordon every node that runs a Garage pod, including the node that the pod shares, so that the recreated pod lands on another node.
- Delete the pod's claims and the pod in a single command: . If the pod stays
kubectl -n <namespace> delete pvc/meta-<pod> pvc/data-<pod> pod/<pod>, no other node has room, so add one.Pending - Wait until lists the pod under a new node ID marked
/garage status, and lists the ID that you noted underNO ROLE ASSIGNED. This takes a minute or two after the pod isFAILED NODES. Until then, the pod's name can appear twice. The new ID is the one that you didn't note.Running - In , read the zone and capacity of the old ID.
/garage layout show - Give the old node's place to the new one: .
/garage layout assign <new-id> --replace <old-id> -z <zone> -c <capacity> - Run to review the change, then apply it with the command that it prints,
/garage layout show./garage layout apply --version <version> - Uncordon the nodes, and wait until shows a
/garage stats -aof 0 on every node. The queue can stay unchanged for 10 minutes before it starts to fall.resync queue length - Close the previous layout version, which otherwise waits forever for the deleted node. Run , with the version that you applied at step 7. Don't add
/garage layout skip-dead-nodes --version <version>.--allow-missing-data - Check that reports a single live layout version again. Move the next pod only after this step. With two layout versions waiting for deleted nodes, Garage loses its quorum, and every S3 request fails.
/garage layout history - If the storage class keeps released volumes (), the two old volumes stay behind as
reclaimPolicy: RetainPersistentVolumes, with their disks. Delete the PersistentVolumes and their disks.Released
Move Garage pods with a replication factor of 1
If you set , each object has a single copy, on one pod's volume. Move that copy off the pod before you delete the pod.
configuration.infrastructure.components.garage.replicationFactor: 1To move a Garage pod with a replication factor of 1, follow the steps in Move Garage pods, with these changes:
- Between steps 1 and 2: move the pod's objects off it, as the following steps describe. Then go on to step 2 straight away, because from then on, S3 requests that reach the pod fail.
- Step 4: the ID that you noted isn't listed under , because it's no longer in the layout. The new ID is the one marked
FAILED NODESthat you didn't note.NO ROLE ASSIGNED - Step 5: skip this step. You note the zone and capacity before step 2.
- Step 6: run the command without
assign, with the zone and capacity that you noted.--replace <old-id> - Steps 9 and 10: skip these steps.
To move the pod's objects off it between steps 1 and 2, follow these steps:
- In , note the pod's zone and capacity.
/garage layout show - Run .
/garage layout remove <old-id> - Run again, and then the
/garage layout showcommand that it prints./garage layout apply --version <version> - Wait until doesn't show the pod as
/garage status,draining metadata..shows a/garage stats -aof 0 on every node, andresync queue lengthreports a single live layout version./garage layout history
Upgrade Traefik's CRDs in Helm deployments
This item applies to deployments that use Helm (). A deployment that uses ArgoCD skips this item, because ArgoCD applies the CRDs on every sync.
shdctl release deploy --format helmHelm never upgrades the CustomResourceDefinitions (CRDs) that a chart installs, so a Helm deployment still has the Traefik CRDs from its first install. Traefik v3.7.16 runs with those CRDs, but the API server silently drops any newer field from your Traefik resources.
When you update the manifest version during the upgrade, list in :
traefikdeployment.helm.upgradeCrdsdeployment: helm: upgradeCrds: [traefik]
The deploy then applies Traefik's CRDs before each Traefik upgrade, and overwrites the cluster's copies of those CRDs.
If another controller on your cluster installs Traefik's CRDs, or only an administrator can write CRDs, don't list . Instead, apply the CRDs yourself. Refer to Keep chart CRDs up to date.
traefikMove the Traefik settings that the new chart renames
This item applies to deployments whose manifest sets Traefik values under . Every other deployment skips it.
configuration.overrides.traefik.values.traefikThe new Traefik chart moves the following settings. When you update the manifest version during the upgrade, move each one that you set, even if the deploy succeeds without the move:
Before | After | If you don't move it |
|---|---|---|
| | The deploy fails |
| | The deploy fails |
| | The deploy fails |
| | The deploy fails |
| | The deploy fails |
| | The deploy fails |
| the same configuration as a YAML object: drop the | The deploy fails with a type error |
| | The deploy fails. Moved unchanged under |
| | The deploy fails. Moved unchanged under |
| the manifest fields | The setting is ignored |
| | The setting is ignored |
A setting that fails the deploy stops it at the Traefik chart with , or with a type error for . An ignored setting gives no error.
additional properties '<key>' not allowedcontentKeep the Gateway API CRDs in ArgoCD deployments
This item applies to deployments that use ArgoCD. A deployment that uses Helm skips this item, because Helm leaves the CRDs in place.
In this release, Traefik doesn't install the Gateway API CustomResourceDefinitions (), which this deployment never used. After the upgrade, ArgoCD's next sync deletes the copies that it applied. Deleting a CRD deletes every resource of that kind in the cluster, for example the Gateways or HTTPRoutes that your own ingress controller or service mesh serves.
gateway.networking.k8s.ioTo keep the Gateway API CRDs, follow these steps:
-
Check whether anything in your cluster uses the Gateway API:kubectl get gatewayclasses,gateways,httproutes,grpcroutes,referencegrants -AIf the command finds no resources, nothing uses the Gateway API, so skip the rest of this item.
-
When you regenerate and deploy the charts during the upgrade, before you push the regenerated charts, mark the CRDs so that ArgoCD keeps them:kubectl annotate --overwrite crd \ gatewayclasses.gateway.networking.k8s.io gateways.gateway.networking.k8s.io \ httproutes.gateway.networking.k8s.io grpcroutes.gateway.networking.k8s.io \ referencegrants.gateway.networking.k8s.io \ argocd.argoproj.io/sync-options=Prune=false argocd.argoproj.io/compare-options=IgnoreExtraneous
Route alerts through Alertmanager
This item applies to deployments that set and have a Grafana contact point for the alerts in Grafana's Custom folder. A deployment that never set up a contact point skips this item.
configuration.monitoring.prometheus.enabled: trueThis release removes those four alerts, and sends alerts through Alertmanager instead. Alertmanager routes nothing until you add a receiver. Any receiver that Alertmanager supports works, such as email, PagerDuty, Microsoft Teams, or a webhook. The following steps use a Slack incoming webhook as an example.
To route alerts through Alertmanager, follow these steps after the upgrade completes:
-
Create a secret inthat holds your Slack webhook URL:
<namespace>kubectl -n <namespace> create secret generic alerting-slack --from-literal=url='<your Slack webhook URL>' -
Create anthat sends warning and critical alerts to the webhook:
AlertmanagerConfigkubectl -n <namespace> apply -f - <<'EOF'apiVersion: monitoring.coreos.com/v1alpha1kind: AlertmanagerConfigmetadata: name: alerting-slackspec: route: receiver: slack groupBy: [alertname, namespace] matchers: - name: severity matchType: "=~" value: "warning|critical" receivers: - name: slack slackConfigs: - apiURL: name: alerting-slack key: url sendResolved: trueEOF -
Delete the Grafana contact point, which receives nothing after the upgrade.
Version 2.0.0
Install shdctl 1.0.0 before you pull
Every deployment needs shdctl 1.0.0 or later for this release. Install it before you start: the release isn't published where vpctl looks for releases, so your current vpctl can't pull it.
Your current release ships , not , so install shdctl in two passes. can install shdctl 1.0.0, under the name , and that binary can pull this release, which carries . The pull reads your manifest, so if yours sets while is Unity's registry, delete it first, as Remove settings that shdctl 1.0.0 refuses from your manifest describes. Then, from the directory that contains , the release you deployed last, run:
install-vpctl.shinstall-shdctl.shinstall-vpctl.shvpctlinstall-shdctl.shconfiguration.kubernetes.docker.namespacedocker.repositoryextracted-release/./extracted-release/common/scripts/install-vpctl.sh 1.0.0vpctl release pull --version 2.0.0 --clean-output./extracted-release/common/scripts/install-shdctl.sh 1.0.0
If you no longer have the release you deployed last, pull it again with first. installs and, on Linux and macOS, replaces with a link to it, so your existing scripts keep running throughout shdctl 1.x. To install elsewhere than , pass the same directory to both installers, as Install shdctl describes.
vpctl release pullinstall-shdctl.shshdctlvpctl/usr/local/binAutomation 0.0.77: snapshot its database first
In every deployment, Automation 0.0.77 migrates its database, which the bundled PostgreSQL holds, when you deploy this release. Before you start, take a full backup of that PostgreSQL, and wait until it succeeds:
backup=$(kubectl -n <namespace> create -o name -f - <<'EOF'apiVersion: pgv2.percona.com/v2kind: PerconaPGBackupmetadata: generateName: before-upgrade-spec: pgCluster: pg-db repoName: repo1 options: - --type=fullEOF)kubectl -n <namespace> wait "$backup" --for=jsonpath='{.status.state}'=Succeeded --timeout=60m
Each run creates a new backup, so if you postpone the upgrade, run the commands again right before you start. The backup goes where your PostgreSQL backups already go: to their backup volume, or to the bucket that names.
configuration.objectStore.buckets.postgresBackupsRemove settings that shdctl 1.0.0 refuses from your manifest
shdctl 1.0.0 refuses two settings that a manifest written for an earlier release can hold. Each refusal names the setting it stopped on.
- , when
configuration.kubernetes.docker.namespaceis Unity's registry,configuration.kubernetes.docker.repository. Every shdctl command that reads the manifest refuses it, including the pull in Install shdctl 1.0.0 before you pull, so delete it before you start.uccmpprivatecloud.azurecr.io - under
_arrayMerge.configuration.overrides.<chart-name>.valuesrefuses it when you regenerate the charts, so when you update the manifest version during the upgrade, rewrite it in the form that Per-chart Helm value overrides describes, which patches a list element by its name.shdctl release generate
Rebuild a secrets file that vpctl persisted
If you keep your secret values in a file that vpctl wrote with , rebuild that file from the cluster when you regenerate and deploy the secrets during the upgrade, before you generate them. A file that vpctl wrote doesn't hold the storage key of the secret (): generating from it creates a new key, and deploying that rotates the key the cluster runs with.
--persistasset-cloud-storage-abstractionstorage-key.pemKeep the vpctl file as a backup, then rebuild it in place: replaces an existing file only with .
shdctl secret export--forcecp secrets.import.yaml secrets.import.vpctl.yamlshdctl secret export --output secrets.import.yaml --force
If your file has another name, use it in both commands. The export lists any value it can't read back from the cluster: take those from the backup. Then generate and deploy the secrets as Regenerate and deploy the secrets describes.
Keep an image pull secret that something else creates
shdctl secret generateconfiguration.kubernetes.imagePullSecretshdctl secret deployconfiguration: kubernetes: imagePullSecret: name: ecr-regcred generate: false
For the details, refer to The image pull secret.
Helm deployments: hand the monitoring stack over to Helm
The monitoring stack is now installed as a Helm release, like every other component. If you deploy with Helm () and the monitoring stack already runs (), hand its resources over to Helm once, when you regenerate and deploy the charts during the upgrade: after and before , from the parent directory of . Otherwise the deployment stops with .
shdctl release deploy --format helmconfiguration.monitoring.prometheus.enabled: trueshdctl release generateshdctl release deploygenerated-charts/invalid ownership metadatahelm dependency update --skip-refresh generated-charts/kube-prometheus-stackhelm template kube-prometheus-stack generated-charts/kube-prometheus-stack -n <namespace> > /tmp/monitoring-resources.yamlkubectl annotate --overwrite -f /tmp/monitoring-resources.yaml meta.helm.sh/release-name=kube-prometheus-stack meta.helm.sh/release-namespace=<namespace>kubectl label --overwrite -f /tmp/monitoring-resources.yaml app.kubernetes.io/managed-by=Helm
NotFoundArgoCD deployments: remove the finalizer of a self-tracking application
If you deploy with ArgoCD, check when you regenerate and deploy the charts during the upgrade, before you commit the regenerated charts, whether exists in your GitOps repository. If it does, the live top-level application tracks itself, and committing the regenerated charts can make ArgoCD delete every application and its workloads. Remove the finalizer from that application first:
<pathPrefix>/asset-solutions/templates/asset-solutions.yamlasset-solutionskubectl -n <namespace> patch application asset-solutions --type merge -p '{"metadata":{"finalizers":null}}'
After you push the regenerated charts, ArgoCD can delete the application itself, while its child applications and their workloads keep running. Once ArgoCD has synced the pushed revision, a few minutes after the push, check with . If it reports the application not found, run , again if step 7 already ran it: it creates the application again, which adopts the child applications.
asset-solutionskubectl -n <namespace> get application asset-solutionsshdctl release deploy --format argocdUpdate your own mirror and scripts for the shd
package name
shdThe release package is renamed from "onprem" to "shd", and needs no change. If you handle the package with your own tools, update them before you pull the new release during the upgrade:
shdctl release pull- If you mirror or pull the release artifact yourself, switch from to
releases/onprem-configuration. This release and later ones are published only there.releases/shd-configuration - If your scripts reference , change the paths to
extracted-release/platform/onprem/….platform/shd/ - If your scripts extract the tarball by file name, it's now . Update the name, or pass
unity-private-cloud-shd-<version>.tar.gzto pin one.--output
Set DISPLAY
in your own action images
DISPLAYAutomation step containers no longer receive , and GPU steps no longer mount host directories. If an action image of your own relies on the platform for , set it in the image before you regenerate and deploy the charts during the upgrade. The bundled Unity Asset Manager and Unity Asset Transformer apps aren't affected.
DISPLAY=:0DISPLAYUpdate callers of the Optimize and Convert pipeline
The Optimize and Convert pipeline now always creates a new asset: its input is removed. and are now required, and the output formats no longer default to and . Update anything that calls the pipeline to pass both and to stop passing , before you regenerate and deploy the charts during the upgrade.
CreateNewAssetOutput FormatsAsset VersionglbpxzCreateNewAssetUpdate references to renamed ingress resources
Several resources that the release installs are renamed, and so are most of the per-route resources alongside them. The shared middlewares that every route chains, , , , , and , keep their names.
IngressRouteMiddlewarecorsforward-authstrip-backend-prefixremove-authorization-headersecurity-headersOld name | New name |
|---|---|
| |
| |
| |
| |
| |
| Removed |
If something outside the release refers to one of these resources by name, such as a custom that reuses a generated middleware, a NetworkPolicy, or a dashboard or alert keyed on a resource name, update it to the new name when you deploy at step 7. The deployment removes the resources under the old names. If a GitOps tool manages your deployment with pruning turned off, remove them by hand once the new ones serve traffic.
IngressRouteVersion 0.16.0
Add the basic
client scope to the sdk
, dashboard
and mini-usf
clients
basicsdkdashboardmini-usfWithout the client scope, the access tokens that the , , and clients receive carry no claim, and user-scoped API calls fail. A new deployment gets the scope from its realm import, but an existing realm isn't imported again, so add it by hand after you deploy:
basicsdkdashboardmini-usfsubunity- In the Keycloak Admin console, open the unity realm, then Clients.
- For each of sdk, dashboard, and mini-usf, open Client scopes, select Add client scope, choose basic, and add it as Default.
No redeployment is needed: new sign-ins receive the claim at once.
subAutomation 0.0.73: snapshot its database first
This item applies to every deployment. When you deploy this release, Pipeline Automation 0.0.73 migrates its database in the bundled PostgreSQL.
Before you start the upgrade, take a full backup of the bundled PostgreSQL, as Automation 0.0.77: snapshot its database first shows. If you upgrade past 0.16.0 and 2.0.0 at once, one backup before you start covers both.
Version 0.15.0
Import the unity-licensing-server
Keycloak client
unity-licensing-serverThe Unity Licensing Server authenticates against a Keycloak client. A new deployment gets the client from its realm import, but an existing realm isn't imported again, so add it by hand after you deploy. From the extracted release, with access to the cluster, generate the import file, replacing with your deployment's namespace:
unity-licensing-serverunitykubectl<namespace>python3 common/scripts/upc-cli/upc-cli.py --fqdn <your-domain> --no-auth keycloak generate-client-json \ --namespace <namespace> --output unity-licensing-server-client.json
The CLI needs Python 3 with the and packages ().
requestskeyringpip install requests keyringThen, in the Keycloak Admin console, open the unity realm, then Clients, select Import client, browse to , and select Save. For managing the licensing server, refer to Licensing.
unity-licensing-server-client.jsonVersion 0.14.0
Set imageVariant: hardened
to keep hardened images
imageVariant: hardenedReleases before 0.14.0 deployed hardened images unconditionally. From 0.14.0 on, a deployment gets them only when its manifest asks for them, and the base images otherwise. To keep hardened images, add this to your manifest when you update the manifest version during the upgrade, before you generate the charts:
configuration: imageVariant: hardened
Version 0.13.0
Move from rustfs to garage
Garage replaces RustFS as the in-cluster object store, and runs as a three-node cluster. Before you generate the charts, replace in your manifest with , which takes , , , and . Then pick one of two paths before you regenerate and deploy the secrets during the upgrade. In every command, replace with your deployment's namespace.
configuration.infrastructure.components.rustfsconfiguration.infrastructure.components.garagemetaStoragedataStoragereplicasreplicationFactor<namespace>Path A: start the object store empty
The objects in RustFS aren't kept: Argo workflow artifacts are temporary, MongoDB backups are taken again at the next scheduled run, and Asset Manager ingests again on demand. Deploy the release, and the job creates the buckets and imports your keys.
garage-bucket-bootstrapPath B: keep the objects
The release carries , which copies the and buckets out of RustFS and into Garage. It needs and the MinIO client, . Run the commands below from the extracted release.
common/scripts/migrate-rustfs-to-garage.shargo-artifactspsmdb-backupskubectlmc-
Before you regenerate the secrets during the upgrade, from a workstation that can reach the cluster, back the buckets up. The regenerated secrets no longer hold the rustfs keys this step reads:export RUSTFS_ACCESS_KEY=$(kubectl -n <namespace> get secret s3-api-storage-credentials -o jsonpath='{.data.RUSTFS_ACCESS_KEY}' | base64 -d)export RUSTFS_SECRET_KEY=$(kubectl -n <namespace> get secret s3-api-storage-credentials -o jsonpath='{.data.RUSTFS_SECRET_KEY}' | base64 -d)NAMESPACE=<namespace> BACKUP_DIR=./rustfs-backup ./common/scripts/migrate-rustfs-to-garage.sh backup
-
Deploy, then wait for the garage StatefulSet to be ready and for thejob to complete.
garage-bucket-bootstrap -
Restore the buckets into Garage:export GARAGE_ACCESS_KEY=$(kubectl -n <namespace> get secret s3-api-storage-credentials -o jsonpath='{.data.GARAGE_ACCESS_KEY}' | base64 -d)export GARAGE_SECRET_KEY=$(kubectl -n <namespace> get secret s3-api-storage-credentials -o jsonpath='{.data.GARAGE_SECRET_KEY}' | base64 -d)NAMESPACE=<namespace> BACKUP_DIR=./rustfs-backup ./common/scripts/migrate-rustfs-to-garage.sh restore
Reclaim the rustfs volumes
On either path, removing RustFS doesn't remove its volumes: they stay bound, unused, and billed. Once the upgrade has finished and no rustfs pod remains, delete them; on path B, check the restored data first.
kubectl -n <namespace> get pvc | grep rustfskubectl -n <namespace> get pvc -o name | grep rustfs | xargs -r kubectl -n <namespace> delete
If you deploy with Helm rather than ArgoCD, also run .
helm -n <namespace> uninstall rustfs