Documentation

Customize Pipeline Automation workloads

Change the resources and the node placement of Pipeline Automation's built-in configuration profiles.
Read time 5 minutesLast updated 13 days ago

Unity Pipeline Automation runs each step of a pipeline in its own Kubernetes pod. The step's configuration profile sets the CPU and memory that the pod gets, and which nodes the pod runs on. You can change these settings for the built-in configuration profiles, for example, to run large steps on a dedicated node pool. You can't add configuration profiles. For the list of built-in profiles and where their steps run by default, refer to Pipeline Automation configuration profiles.

Change a configuration profile

To change a configuration profile:
  1. Find the profile's index in Built-in configuration profiles.
  2. In your manifest (
    manifest.yaml
    ), add the settings you want to change under
    configuration.overrides.automation.values.common-chart.configMaps.automation-config.data
    . For the setting names, refer to Setting names.
  3. Run
    shdctl release generate
    to generate the release's Helm charts again, then deploy them. Your settings go into the values of the
    automation
    chart, which deploys Pipeline Automation. For more information, refer to step 7 of Deploy the solution.
The settings stay in the manifest, so every later
shdctl release generate
keeps them.

Setting names

Every setting name starts with
UCAUTOMATION_ConfigurationProfileOptions__Specs__<index>__
, where
<index>
is the profile's index. The following table lists the rest of each name.

Setting

Rest of the name

Description

CPU request
Hardware__CpuRequest
The CPU that the step's pod requests, in cores, for example
2
.
CPU limit
Hardware__CpuLimit
The most CPU that the pod can use, in cores.
Memory request
Hardware__MemoryRequest
The memory that the pod requests, for example
16Gi
.
Memory limit
Hardware__MemoryLimit
The most memory that the pod can use.
Toleration
TolerationOverride__<n>__Key
,
TolerationOverride__<n>__Operator
,
TolerationOverride__<n>__Value
, and
TolerationOverride__<n>__Effect
One Kubernetes toleration for each
<n>
, numbered from
0
. Replaces all of the profile's default tolerations.
Node affinity
AffinityOverride__NodeAffinity__RequiredDuringSchedulingIgnoredDuringExecution__<n>__MatchExpressions__<m>__Key
, the same name ending in
__Operator
, and the same name ending in
__Values__<k>
A required node affinity rule.
<n>
numbers the node selector terms,
<m>
the match expressions in a term, and
<k>
the values in an expression, each from
0
. Adds to the default placement.
This example moves the steps of the
premium
profile (index 4) to a node pool whose nodes are labeled
workload=highmem
and tainted
workload=highmem:NoSchedule
:
  • Toleration
    0
    adds back the profile's default toleration,
    argo-workflows=cpu:NoSchedule
    , because a toleration setting replaces the defaults.
  • Toleration
    1
    tolerates the node pool's taint,
    workload=highmem:NoSchedule
    .
  • The node affinity rule requires nodes labeled
    workload=highmem
    .
configuration: overrides: automation: values: common-chart: configMaps: automation-config: data: UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__0__Key: "argo-workflows" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__0__Operator: "Equal" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__0__Value: "cpu" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__0__Effect: "NoSchedule" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__1__Key: "workload" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__1__Operator: "Equal" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__1__Value: "highmem" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__1__Effect: "NoSchedule" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__AffinityOverride__NodeAffinity__RequiredDuringSchedulingIgnoredDuringExecution__0__MatchExpressions__0__Key: "workload" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__AffinityOverride__NodeAffinity__RequiredDuringSchedulingIgnoredDuringExecution__0__MatchExpressions__0__Operator: "In" UCAUTOMATION_ConfigurationProfileOptions__Specs__4__AffinityOverride__NodeAffinity__RequiredDuringSchedulingIgnoredDuringExecution__0__MatchExpressions__0__Values__0: "highmem"
Steps on a CPU profile still run only on nodes labeled
aks-node-pool=argocpu
, because the affinity adds to the default placement and doesn't replace it. So label the nodes of the
workload=highmem
pool
aks-node-pool=argocpu
too.
Keep these rules in mind:
  • A toleration setting replaces all of the profile's default tolerations, so list every taint that the target nodes carry. For more information, refer to Taints and Tolerations in the Kubernetes documentation. The default tolerations are:
    • CPU profiles: the taint
      argo-workflows=cpu:NoSchedule
      .
    • GPU profiles:
      nvidia.com/gpu
      with the
      Exists
      operator and the
      NoSchedule
      effect,
      sku=gpu:NoSchedule
      , and
      CriticalAddonsOnly
      with the
      Exists
      operator. Keep the ones your GPU nodes carry.
  • Pipeline Automation ignores a setting name it doesn't recognize and doesn't report an error, so a misspelled name has no effect. To avoid typos, copy the setting names from this page, then check the configuration profile.
  • To set node placement, use the node affinity settings. For more information, refer to Node affinity in the Kubernetes documentation. Don't use a profile's node selector settings for this: they carry the label in the setting's name, and a Kubernetes ConfigMap key can't contain the
    /
    in labels such as
    node.kubernetes.io/instance-type
    .
  • The index of a profile belongs to the release. Before each upgrade, compare your settings with the new release's built-in configuration profiles.

Check a configuration profile

To check that a configuration profile applies:
  1. Check that the settings reached the automation service's configuration:
    kubectl get configmap automation-config -n <namespace> -o yaml | grep ConfigurationProfileOptions
  2. Run a pipeline step that uses the profile.
  3. Find the step's pod, then check its tolerations and affinity:
    kubectl get pods -n <namespace> -l workflows.argoproj.io/workflowkubectl get pod <step-pod> -n <namespace> -o jsonpath='{.spec.tolerations}{"\n"}{.spec.affinity}{"\n"}'
If the configuration profile applies, step 1 lists your settings, and the tolerations and affinity that step 3 shows include the ones you set.

Change a configuration profile in the AWS Terraform package

The settings on this page also apply to a deployment from the AWS Terraform package. In your copy of the package, add them under
common-chart.configMaps.automation-config.data
in
platform/aws/applications/automation/values.yaml
:
common-chart: configMaps: automation-config: data: UCAUTOMATION_ConfigurationProfileOptions__Specs__4__TolerationOverride__0__Key: "argo-workflows"
The next chart generation merges this file into the
automation
chart's values, and ArgoCD applies them. The configuration profiles, the rules, and the checks on this page apply unchanged.

Additional resources