# Migration guide > The manual steps each Self-Hosted Deployment release needs beyond the standard upgrade The manual steps an existing Self-Hosted Deployment needs when it upgrades past a release, beyond the [standard upgrade](./upgrade.md). The newest release comes first, and a release that needs no such step has no section. For everything else each release changes, refer to the [release notes](../release-notes). When you upgrade across several releases, complete the section of every release after your current one, up to and including your target, oldest first. Each item says which deployments it applies to and at which step of the upgrade to complete it. In every command, `` is your deployment's namespace, the one `configuration.kubernetes.namespace` names. ## Version 2.0.0 ### Install shdctl 1.0.0 before you pull Every deployment needs shdctl 1.0.0 or later for this release. Install it before you start: the release isn't published where vpctl looks for releases, so your current vpctl can't pull it. Your current release ships `install-vpctl.sh`, not `install-shdctl.sh`, so install shdctl in two passes. `install-vpctl.sh` can install shdctl 1.0.0, under the name `vpctl`, and that binary can pull this release, which carries `install-shdctl.sh`. The pull reads your manifest, so if yours sets `configuration.kubernetes.docker.namespace` while `docker.repository` is Unity's registry, delete it first, as [Remove settings that shdctl 1.0.0 refuses from your manifest](#remove-settings-that-shdctl-1.0.0-refuses-from-your-manifest) describes. Then, from the directory that contains `extracted-release/`, the release you deployed last, run: ```sh ./extracted-release/common/scripts/install-vpctl.sh 1.0.0 vpctl release pull --version 2.0.0 --clean-output ./extracted-release/common/scripts/install-shdctl.sh 1.0.0 ``` If you no longer have the release you deployed last, pull it again with `vpctl release pull` first. `install-shdctl.sh` installs `shdctl` and, on Linux and macOS, replaces `vpctl` with a link to it, so your existing scripts keep running throughout shdctl 1.x. To install elsewhere than `/usr/local/bin`, pass the same directory to both installers, as [Install shdctl](../../shdctl/install) describes. ### Automation 0.0.77: snapshot its database first In every deployment, Automation 0.0.77 migrates its database, which the bundled PostgreSQL holds, when you deploy this release. Before you start, take a full backup of that PostgreSQL, and wait until it succeeds: ```sh backup=$(kubectl -n create -o name -f - <<'EOF' apiVersion: pgv2.percona.com/v2 kind: PerconaPGBackup metadata: generateName: before-upgrade- spec: pgCluster: pg-db repoName: repo1 options: - --type=full EOF ) kubectl -n wait "$backup" --for=jsonpath='{.status.state}'=Succeeded --timeout=60m ``` Each run creates a new backup, so if you postpone the upgrade, run the commands again right before you start. The backup goes where your PostgreSQL backups already go: to their backup volume, or to the bucket that `configuration.objectStore.buckets.postgresBackups` names. ### Remove settings that shdctl 1.0.0 refuses from your manifest shdctl 1.0.0 refuses two settings that a manifest written for an earlier release can hold. Each refusal names the setting it stopped on. * `configuration.kubernetes.docker.namespace`, when `configuration.kubernetes.docker.repository` is Unity's registry, `uccmpprivatecloud.azurecr.io`. Every shdctl command that reads the manifest refuses it, including the pull in [Install shdctl 1.0.0 before you pull](#install-shdctl-1.0.0-before-you-pull), so delete it before you start. * `_arrayMerge` under `configuration.overrides..values`. `shdctl release generate` refuses it at step 7, so at step 2 of the upgrade, rewrite it in the form that [Per-chart Helm value overrides](../../shdctl/manifest#per-chart-helm-value-overrides) describes, which patches a list element by its name. ### Rebuild a secrets file that vpctl persisted If you keep your secret values in a file that vpctl wrote with `--persist`, rebuild that file from the cluster at step 6 of the upgrade, before you generate the secrets. A file that vpctl wrote doesn't hold the storage key of the `asset-cloud-storage-abstraction` secret (`storage-key.pem`): generating from it creates a new key, and deploying that rotates the key the cluster runs with. Keep the vpctl file as a backup, then rebuild it in place: `shdctl secret export` replaces an existing file only with `--force`. ```sh cp secrets.import.yaml secrets.import.vpctl.yaml shdctl secret export --output secrets.import.yaml --force ``` If your file has another name, use it in both commands. The export lists any value it can't read back from the cluster: take those from the backup. Then generate and deploy the secrets as [step 6](./upgrade.md#6.-regenerate-and-deploy-the-secrets) describes. ### Keep an image pull secret that something else creates `shdctl secret generate` now renders the image pull secret, the one that `configuration.kubernetes.imagePullSecret` names, from the registry credentials you supply, and `shdctl secret deploy` applies it. If something else in your cluster maintains that secret, such as a token refresher for ECR's short-lived credentials, turn generation off in your manifest before step 6 of the upgrade. Otherwise the deployed copy overwrites the live credential, and image pulls fail once it expires: ```yaml configuration: kubernetes: imagePullSecret: name: ecr-regcred generate: false ``` For the details, refer to [The image pull secret](../../shdctl/commands/secret#the-image-pull-secret). ### Helm deployments: hand the monitoring stack over to Helm The monitoring stack is now installed as a Helm release, like every other component. If you deploy with Helm (`shdctl release deploy --format helm`) and the monitoring stack already runs (`configuration.monitoring.prometheus.enabled: true`), hand its resources over to Helm once, at step 7 of the upgrade: after `shdctl release generate` and before `shdctl release deploy`, from the parent directory of `generated-charts/`. Otherwise the deployment stops with `invalid ownership metadata`. ```sh helm dependency update --skip-refresh generated-charts/kube-prometheus-stack helm template kube-prometheus-stack generated-charts/kube-prometheus-stack -n > /tmp/monitoring-resources.yaml kubectl annotate --overwrite -f /tmp/monitoring-resources.yaml meta.helm.sh/release-name=kube-prometheus-stack meta.helm.sh/release-namespace= kubectl label --overwrite -f /tmp/monitoring-resources.yaml app.kubernetes.io/managed-by=Helm ``` `NotFound` messages and a non-zero exit code are expected: the commands cover every resource the release installs, including ones your deployment doesn't have yet, which the upgrade creates. Everything already running is handed over without a restart. Resources that an earlier monitoring version installed and that this release no longer includes are left as they are; you can delete them once the upgrade succeeds. A first installation and a deployment that uses ArgoCD skip this step. ### ArgoCD deployments: remove the finalizer of a self-tracking application If you deploy with ArgoCD, check at step 7 of the upgrade, before you commit the regenerated charts, whether `/asset-solutions/templates/asset-solutions.yaml` exists in your GitOps repository. If it does, the live top-level `asset-solutions` application tracks itself, and committing the regenerated charts can make ArgoCD delete every application and its workloads. Remove the finalizer from that application first: ```sh kubectl -n patch application asset-solutions --type merge -p '{"metadata":{"finalizers":null}}' ``` After you push the regenerated charts, ArgoCD can delete the `asset-solutions` application itself, while its child applications and their workloads keep running. Once ArgoCD has synced the pushed revision, a few minutes after the push, check with `kubectl -n get application asset-solutions`. If it reports the application not found, run `shdctl release deploy --format argocd`, again if step 7 already ran it: it creates the application again, which adopts the child applications. ### Update your own mirror and scripts for the `shd` package name The release package is renamed from "onprem" to "shd", and `shdctl release pull` needs no change. If you handle the package with your own tools, update them before step 4 of the upgrade: * If you mirror or pull the release artifact yourself, switch from `releases/onprem-configuration` to `releases/shd-configuration`. This release and later ones are published only there. * If your scripts reference `extracted-release/platform/onprem/…`, change the paths to `platform/shd/`. * If your scripts extract the tarball by file name, it's now `unity-private-cloud-shd-.tar.gz`. Update the name, or pass `--output` to pin one. ### Set `DISPLAY` in your own action images Automation step containers no longer receive `DISPLAY=:0`, and GPU steps no longer mount host directories. If an action image of your own relies on the platform for `DISPLAY`, set it in the image before step 7 of the upgrade. The bundled Unity Asset Manager and Unity Asset Transformer apps aren't affected. ### Update callers of the Optimize and Convert pipeline The Optimize and Convert pipeline now always creates a new asset: its `CreateNewAsset` input is removed. `Output Formats` and `Asset Version` are now required, and the output formats no longer default to `glb` and `pxz`. Update anything that calls the pipeline to pass both and to stop passing `CreateNewAsset`, in time for step 7 of the upgrade. ### Update references to renamed ingress resources Several `IngressRoute` resources that the release installs are renamed, and so are most of the per-route `Middleware` resources alongside them. The shared middlewares that every route chains, `cors`, `forward-auth`, `strip-backend-prefix`, `remove-authorization-header`, and `security-headers`, keep their names. | Old name | New name | | ------------------------ | ----------------------------- | | `asset-solutions-public` | `asset-solutions-client` | | `data-streaming-public` | `data-streaming-client` | | `linksharing` | `app-linking-client` | | `uns` | `unity-notifications` | | `uns-internal` | `unity-notifications-private` | | `dt-storage-public` | Removed | If something outside the release refers to one of these resources by name, such as a custom `IngressRoute` that reuses a generated middleware, a NetworkPolicy, or a dashboard or alert keyed on a resource name, update it to the new name when you deploy at step 7. The deployment removes the resources under the old names. If a GitOps tool manages your deployment with pruning turned off, remove them by hand once the new ones serve traffic. ## Version 0.16.0 ### Add the `basic` client scope to the `sdk`, `dashboard` and `mini-usf` clients Without the `basic` client scope, the access tokens that the `sdk`, `dashboard`, and `mini-usf` clients receive carry no `sub` claim, and user-scoped API calls fail. A new deployment gets the scope from its realm import, but an existing `unity` realm isn't imported again, so add it by hand after you deploy: 1. In the [Keycloak Admin console](/self-hosted-deployment/admin/keycloak-admin-console.md), open the **unity** realm, then **Clients**. 2. For each of **sdk**, **dashboard**, and **mini-usf**, open **Client scopes**, select **Add client scope**, choose **basic**, and add it as **Default**. No redeployment is needed: new sign-ins receive the `sub` claim at once. ## Version 0.15.0 ### Import the `unity-licensing-server` Keycloak client The Unity Licensing Server authenticates against a `unity-licensing-server` Keycloak client. A new deployment gets the client from its realm import, but an existing `unity` realm isn't imported again, so add it by hand after you deploy. From the extracted release, with `kubectl` access to the cluster, generate the import file, replacing `` with your deployment's namespace: ```sh python3 common/scripts/upc-cli/upc-cli.py --fqdn --no-auth keycloak generate-client-json \ --namespace --output unity-licensing-server-client.json ``` The CLI needs Python 3 with the `requests` and `keyring` packages (`pip install requests keyring`). Then, in the [Keycloak Admin console](/self-hosted-deployment/admin/keycloak-admin-console.md), open the **unity** realm, then **Clients**, select **Import client**, browse to `unity-licensing-server-client.json`, and select **Save**. For managing the licensing server, refer to [Licensing](/self-hosted-deployment/admin/licensing.md). ### Automation 0.0.72: snapshot its database first In every deployment, Automation 0.0.72 migrates its database, which the bundled PostgreSQL holds, when you deploy this release. Before you start, take a full backup of that PostgreSQL as [Automation 0.0.77: snapshot its database first](#automation-0.0.77:-snapshot-its-database-first) shows. If you upgrade past 0.15.0 and 2.0.0 at once, one backup before you start covers both. ## Version 0.14.0 ### Set `imageVariant: hardened` to keep hardened images Releases before 0.14.0 deployed hardened images unconditionally. From 0.14.0 on, a deployment gets them only when its manifest asks for them, and the base images otherwise. To keep hardened images, add this to your manifest at step 2 of the upgrade, before you generate the charts: ```yaml configuration: imageVariant: hardened ``` ## Version 0.13.0 ### Move from rustfs to garage Garage replaces RustFS as the in-cluster object store, and runs as a three-node cluster. Before you generate the charts, replace `configuration.infrastructure.components.rustfs` in your manifest with `configuration.infrastructure.components.garage`, which takes `metaStorage`, `dataStorage`, `replicas`, and `replicationFactor`. Then pick one of two paths before step 6 of the upgrade. In every command, replace `` with your deployment's namespace. #### Path A: start the object store empty The objects in RustFS aren't kept: Argo workflow artifacts are temporary, MongoDB backups are taken again at the next scheduled run, and Asset Manager ingests again on demand. Deploy the release, and the `garage-bucket-bootstrap` job creates the buckets and imports your keys. #### Path B: keep the objects The release carries `common/scripts/migrate-rustfs-to-garage.sh`, which copies the `argo-artifacts` and `psmdb-backups` buckets out of RustFS and into Garage. It needs `kubectl` and the MinIO client, `mc`. Run the commands below from the extracted release. 1. Before you regenerate the secrets at step 6, from a workstation that can reach the cluster, back the buckets up. The regenerated secrets no longer hold the rustfs keys this step reads: ```sh export RUSTFS_ACCESS_KEY=$(kubectl -n get secret s3-api-storage-credentials -o jsonpath='{.data.RUSTFS_ACCESS_KEY}' | base64 -d) export RUSTFS_SECRET_KEY=$(kubectl -n get secret s3-api-storage-credentials -o jsonpath='{.data.RUSTFS_SECRET_KEY}' | base64 -d) NAMESPACE= BACKUP_DIR=./rustfs-backup ./common/scripts/migrate-rustfs-to-garage.sh backup ``` 2. Deploy, then wait for the garage StatefulSet to be ready and for the `garage-bucket-bootstrap` job to complete. 3. Restore the buckets into Garage: ```sh export GARAGE_ACCESS_KEY=$(kubectl -n get secret s3-api-storage-credentials -o jsonpath='{.data.GARAGE_ACCESS_KEY}' | base64 -d) export GARAGE_SECRET_KEY=$(kubectl -n get secret s3-api-storage-credentials -o jsonpath='{.data.GARAGE_SECRET_KEY}' | base64 -d) NAMESPACE= BACKUP_DIR=./rustfs-backup ./common/scripts/migrate-rustfs-to-garage.sh restore ``` #### Reclaim the rustfs volumes On either path, removing RustFS doesn't remove its volumes: they stay bound, unused, and billed. Once the upgrade has finished and no rustfs pod remains, delete them; on path B, check the restored data first. ```sh kubectl -n get pvc | grep rustfs kubectl -n get pvc -o name | grep rustfs | xargs -r kubectl -n delete ``` If you deploy with Helm rather than ArgoCD, also run `helm -n uninstall rustfs`.