Customize Pipeline Automation workloads
Change the resources and the node placement of Pipeline Automation's built-in configuration profiles.
Read time 5 minutesLast updated 13 days ago
Unity Pipeline Automation runs each step of a pipeline in its own Kubernetes pod. The step's configuration profile sets the CPU and memory that the pod gets, and which nodes the pod runs on. You can change these settings for the built-in configuration profiles, for example, to run large steps on a dedicated node pool. You can't add configuration profiles. For the list of built-in profiles and where their steps run by default, refer to Pipeline Automation configuration profiles.
Change a configuration profile
To change a configuration profile:
- Find the profile's index in Built-in configuration profiles.
- In your manifest (), add the settings you want to change under
manifest.yaml. For the setting names, refer to Setting names.configuration.overrides.automation.values.common-chart.configMaps.automation-config.data - Run to generate the release's Helm charts again, then deploy them. Your settings go into the values of the
shdctl release generatechart, which deploys Pipeline Automation. For more information, refer to step 7 of Deploy the solution.automation
The settings stay in the manifest, so every later keeps them.
shdctl release generateSetting names
Every setting name starts with , where is the profile's index. The following table lists the rest of each name.
UCAUTOMATION_ConfigurationProfileOptions__Specs__<index>__<index>Setting | Rest of the name | Description |
|---|---|---|
| CPU request | | The CPU that the step's pod requests, in cores, for example |
| CPU limit | | The most CPU that the pod can use, in cores. |
| Memory request | | The memory that the pod requests, for example |
| Memory limit | | The most memory that the pod can use. |
| Toleration | | One Kubernetes toleration for each |
| Node affinity | | A required node affinity rule. |
This example moves the steps of the profile (index 4) to a node pool whose nodes are labeled and tainted :
premiumworkload=highmemworkload=highmem:NoSchedule- Toleration adds back the profile's default toleration,
0, because a toleration setting replaces the defaults.argo-workflows=cpu:NoSchedule - Toleration tolerates the node pool's taint,
1.workload=highmem:NoSchedule - The node affinity rule requires nodes labeled .
workload=highmem
configuration: overrides: automation: values: common-chart: configMaps: automation-config: data: UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__0__Key: "argo-workflows" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__0__Operator: "Equal" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__0__Value: "cpu" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__0__Effect: "NoSchedule" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__1__Key: "workload" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__1__Operator: "Equal" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__1__Value: "highmem" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__1__Effect: "NoSchedule" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__AffinityOverride__NodeAffinity__RequiredDuringSchedulingIgnoredDuringExecution__0__MatchExpressions__0__Key: "workload" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__AffinityOverride__NodeAffinity__RequiredDuringSchedulingIgnoredDuringExecution__0__MatchExpressions__0__Operator: "In" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__AffinityOverride__NodeAffinity__RequiredDuringSchedulingIgnoredDuringExecution__0__MatchExpressions__0__Values__0: "highmem"
Steps on a CPU profile still run only on nodes labeled , because the affinity adds to the default placement and doesn't replace it. So label the nodes of the pool too.
aks-node-pool=argocpuworkload=highmemaks-node-pool=argocpuKeep these rules in mind:
- A toleration setting replaces all of the profile's default tolerations, so list every taint that the target nodes carry. For more information, refer to Taints and Tolerations in the Kubernetes documentation. The default tolerations are:
- CPU profiles: the taint .
argo-workflows=cpu:NoSchedule - GPU profiles: with the
nvidia.com/gpuoperator and theExistseffect,NoSchedule, andsku=gpu:NoSchedulewith theCriticalAddonsOnlyoperator. Keep the ones your GPU nodes carry.Exists
- CPU profiles: the taint
- Pipeline Automation ignores a setting name it doesn't recognize and doesn't report an error, so a misspelled name has no effect. To avoid typos, copy the setting names from this page, then check the configuration profile.
- To set node placement, use the node affinity settings. For more information, refer to Node affinity in the Kubernetes documentation. Don't use a profile's node selector settings for this: they carry the label in the setting's name, and a Kubernetes ConfigMap key can't contain the in labels such as
/.node.kubernetes.io/instance-type - The index of a profile belongs to the release. Before each upgrade, compare your settings with the new release's built-in configuration profiles.
Check a configuration profile
To check that a configuration profile applies:
-
Check that the settings reached the automation service's configuration:kubectl get configmap automation-config -n <namespace> -o yaml | grep ConfigurationProfileOptions
-
Run a pipeline step that uses the profile.
-
Find the step's pod, then check its tolerations and affinity:kubectl get pods -n <namespace> -l workflows.argoproj.io/workflowkubectl get pod <step-pod> -n <namespace> -o jsonpath='{.spec.tolerations}{"\n"}{.spec.affinity}{"\n"}'
If the configuration profile applies, step 1 lists your settings, and the tolerations and affinity that step 3 shows include the ones you set.
Change a configuration profile in the AWS Terraform package
The settings on this page also apply to a deployment from the AWS Terraform package. In your copy of the package, add them under in :
common-chart.configMaps.automation-config.dataplatform/aws/applications/automation/values.yamlcommon-chart: configMaps: automation-config: data: UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__0__Key: "argo-workflows"
The next chart generation merges this file into the chart's values, and ArgoCD applies them. The configuration profiles, the rules, and the checks on this page apply unchanged.
automation