omnia_config.yml¶

This file controls the deployment of Slurm and Kubernetes across cluster nodes.

Parameter reference¶

Slurm Configuration Parameters¶

Parameter	Details
slurm_cluster list / elements=dict / required	Slurm cluster configuration list. When Slurm is enabled, supply one entry. The current implementation reads only the first entry and does not process additional entries.
cluster_name string / required	Unique name for the slurm cluster.
nfs_storage_name string / required	Indicates the NFS storage name for the NFS storage to be used by this slurm cluster. This is defined in storage_config.yml as name. Refer - Slurm Storage Architecture
vast_storage_name string	Optional storage name for a separate Slurm VAST mount in `storage_config.yml`. Use the standard value `vast_storage`, which must exactly match the mount `name`. When omitted or empty, Orchestrator skips the standard `vast_storage` entry and reuses `nfs_storage_name` for shared-data and HPC-tools paths. When selected, VAST storage supplies /hpc_tools, /apps, /scratch, /tmp, and /projects. It also supplies /home when OpenLDAP support is enabled. Refer - Slurm Storage Architecture
skip_merge boolean	Variable indicates whether a specific configuration file path under config_sources should be used as-is without merging
config_sources dict/str	Config can be a file path or inline mapping Value can be: str (File path string) or dict (Inline configuration mapping).
node_discovery_mode string	Node hardware discovery mode. 'homogeneous' for group-based discovery, 'heterogeneous' for individual node discovery. Default: heterogeneous
node_hardware_defaults dict
<node_hardware_defaults_key> dict	Hardware specifications for homogeneous node groups. Key is group name (grp0-grp100)
sockets integer / required	Number of CPU sockets per node Minimum: 1.
cores_per_socket integer / required	Number of CPU cores per socket Minimum: 1.
threads_per_core integer / required	Number of CPU threads per core Minimum: 1.
real_memory integer / required	Memory in MB (exact value to use in Slurm) Minimum: 1.
gres string	GPU resources in format 'gpu:N' (optional)

This release supports one Slurm cluster configuration. When Slurm is enabled, supply one item in slurm_cluster; the current implementation reads only the first item and does not process additional items. vast_storage_name is optional. When it is empty or omitted, Orchestrator reuses nfs_storage_name for Slurm shared-data and HPC-tools paths and skips the standard mount whose name is vast_storage. When set, vast_storage_name must exactly match one mount name in storage_config.yml.

Custom Slurm configuration sources¶

The optional config_sources mapping supplies custom Slurm configuration values. Each child accepts either an inline key-value mapping or an absolute path to a configuration file that is readable on the OIM. Relative paths and environment-variable expressions are not supported. Unknown child names are rejected.

Child	Generated configuration file	Accepted value
`slurm`	`slurm.conf`	Inline mapping or absolute file path
`slurmdbd`	`slurmdbd.conf`	Inline mapping or absolute file path
`cgroup`	`cgroup.conf`	Inline mapping or absolute file path
`gres`	`gres.conf`	Inline mapping or absolute file path
`acct_gather`	`acct_gather.conf`	Inline mapping or absolute file path
`helpers`	`helpers.conf`	Inline mapping or absolute file path
`job_container`	`job_container.conf`	Inline mapping or absolute file path
`mpi`	`mpi.conf`	Inline mapping or absolute file path
`oci`	`oci.conf`	Inline mapping or absolute file path
`topology`	`topology.conf`	Inline mapping or absolute file path
`burst_buffer`	`burst_buffer.conf`	Inline mapping or absolute file path

By default, file-based configuration is merged with generated defaults. Set skip_merge: true in the same slurm_cluster entry when file-path values must replace the generated defaults. Inline mappings continue to merge with the defaults, and one config_sources mapping can contain both value forms.

Kubernetes Configuration Parameters¶

Parameter	Details
service_k8s_cluster list / elements=dict / required	List of service Kubernetes cluster configurations.
cluster_name string / required	Non-empty, case-sensitive name of the cluster on which you want to deploy Kubernetes. Cluster names must be unique within this list.
deployment boolean / required	Indicates if Kubernetes will be deployed or not. Accepted values: `true` or `false` Set this value to `true` on exactly one service Kubernetes cluster entry. Input validation rejects configurations with no selected cluster or with multiple selected clusters.
enable_powerscale_csi boolean	Enables PowerScale CSI on the cluster selected with `deployment: true`. When enabled, both PowerScale driver file paths are required. The default is `false`.
etcd_on_local_disk boolean / required	Determines whether ETCD is deployed on local disk or NFS storage. Accepted values: `true` or `false` When set to `true`, ETCD is deployed on local disk on all master nodes. The system prioritizes BOSS card if available, and falls back to SSD/SATA disks if BOSS is not present. The `/var/lib/etcd` directory is mounted on the selected local disk. When set to `false`, ETCD storage is provisioned using NFS, and no local disk configuration is performed for ETCD. The field cannot be omitted. Important Migration from NFS to local disk is not supported during upgrades. This configuration is only applicable for fresh installations.
k8s_cni string / required	Kubernetes SDN network. Schema-accepted values: `calico` or `flannel` Supported provisioning value: `calico`. The current path always stages and applies Calico; `flannel` passes input validation but does not select a Flannel manifest.
pod_external_ip_range string / required	Ordered IPv4 range or IPv4 CIDR used by the load balancer for assigning external IPs to Kubernetes services. The range must not overlap the service CIDR, pod CIDR, or any DHCP pool. It must not contain an OIM admin or BMC address, or any `ADMIN_IP`, `BMC_IP`, or `IB_IP` from the PXE mapping file. Ensure that the `pod_external_ip_range` defined in the `omnia_config.yml` file is reachable from the OpenManage Enterprise appliance and the SFM network. Sample values: `172.16.107.170-172.16.107.200`
k8s_service_addresses string / required	Canonical IPv4 network used internally for Kubernetes services. It must not overlap the pod CIDR, external pool, or any physical network in `network_spec.yml`. Sample value: `"10.233.0.0/18"`
k8s_pod_network_cidr string / required	Canonical IPv4 network used to assign addresses to individual pods. It must not overlap the service CIDR, external pool, or any physical network in `network_spec.yml`. Sample value: `"10.233.64.0/18"`
nfs_storage_name string / conditional required	When Kubernetes functional groups are selected, the entry selected with `deployment: true` must define a non-empty value that exists in `storage_config.yml`. Its matching mount must also set `mount_on_oim: true`. Entries with `deployment: false` are not deployed and need not define this value.
csi_powerscale_driver_secret_file_path string / conditional required	Required when `enable_powerscale_csi` is `true`. Specify an absolute path to an existing regular `secret.yaml` file that satisfies the PowerScale CSI input contract.
csi_powerscale_driver_values_file_path string / conditional required	Required when `enable_powerscale_csi` is `true`. Specify an absolute path to an existing regular, plaintext `values.yaml` file that satisfies the PowerScale CSI input contract.
k8s_crio_storage_size string / required	Specifies the disk size allocated for CRI-O container storage.

PowerScale CSI is controlled by enable_powerscale_csi on the service_k8s_cluster selected with deployment: true. The optional Boolean defaults to false. When set to true, both csi_powerscale_driver_secret_file_path and csi_powerscale_driver_values_file_path are required and must identify existing regular files by absolute path. Catalog membership and populated file paths do not enable CSI when the flag is false or omitted.

When service Kubernetes is configured, set deployment: true on exactly one service_k8s_cluster item. Other entries may remain in the list with deployment: false, but Orchestrator deploys only the selected item. Input validation rejects configurations with no selected cluster or with multiple entries marked true. When the PXE mapping selects Kubernetes functional groups, the item selected with deployment: true must define a nonempty nfs_storage_name that exists in storage_config.yml. Retained entries with deployment: false are not deployed and need not define that storage name.

For the deployed cluster, pod_external_ip_range, k8s_service_addresses, and k8s_pod_network_cidr must be valid, mutually non-overlapping IPv4 ranges. The service and pod CIDRs must be canonical networks and must not overlap a physical network in network_spec.yml. The external pool must not overlap a DHCP pool or contain an OIM address or any ADMIN_IP, BMC_IP, or IB_IP from the PXE mapping file.

Provisioning bolt-ons¶

The optional orchestrator.bolt_ons mapping overrides the bolt-on list for a workload category. Supplying a category list replaces that category's default; it does not extend it. Values must be unique and may contain only the names shown below:

Category	Default list	Accepted values
`kubernetes`	`mount_config`, `k8s_config`, `telemetry`	`mount_config`, `k8s_config`, `telemetry`, `openldap`
`slurm`	`mount_config`, `slurm_config`, `openldap`	`mount_config`, `slurm_config`, `openldap`

Adding openldap to the Kubernetes list enables LDAP client configuration only when the selected catalog also enables OpenLDAP. The current Kubernetes provisioning play does not invoke a telemetry role from the telemetry list item; deploy Telemetry through its domain workflow.

Usage example¶

File: $OMNIA_DATA_PATH/orchestrator/input/$OMNIA_PROJECT_NAME/omnia_config.yml

---
slurm_cluster:
  - cluster_name: slurm_cluster
    nfs_storage_name: nfs_slurm
    vast_storage_name: vast_storage
    node_discovery_mode: "homogeneous"
    # Optional: Override Slurm and cgroup configuration
    config_sources:
      slurm: /path/to/custom/slurm.conf
      cgroup: /path/to/custom/cgroup.conf
      slurmdbd: /path/to/custom/slurmdbd.conf
      acct_gather:
        ProfileHDF5Dir: /var/log/slurm/profile
      # slurm:
      #   SlurmctldTimeout: 60
      #   SlurmdTimeout: 150
    # Optional: Override hardware specs for specific node groups
    node_hardware_defaults:
      grp1:
        sockets: 2
        cores_per_socket: 64
        threads_per_core: 2
        real_memory: 512000
        gres: "gpu:4"
      grp2:
        sockets: 2
        cores_per_socket: 32
        threads_per_core: 2
        real_memory: 256000

service_k8s_cluster:
  - cluster_name: service_cluster
    deployment: true
    enable_powerscale_csi: false
    etcd_on_local_disk: false
    k8s_cni: "calico"
    pod_external_ip_range: "172.16.107.170-172.16.107.200"
    k8s_service_addresses: "10.233.0.0/18"
    k8s_pod_network_cidr: "10.233.64.0/18"
    nfs_storage_name: "nfs_k8s"
    k8s_crio_storage_size: "20G"
    csi_powerscale_driver_secret_file_path: ""
    csi_powerscale_driver_values_file_path: ""

# Optional: replace the default bolt-on lists for either category.
orchestrator:
  bolt_ons:
    kubernetes:
      - mount_config
      - k8s_config
      - openldap
    slurm:
      - mount_config
      - slurm_config
      - openldap

Info

Orchestrator Config -- Provisioning, catalog path, and upstream output settings.
Slurm Conf -- Custom Slurm configuration.
HA Config -- Kubernetes high-availability settings.
Slurm Storage Architecture -- How NFS and VAST mounts are used by Slurm.
K8s Storage Architecture -- How NFS mounts are used by service K8s.