omnia_config.yml¶

This file controls the deployment of Slurm and Kubernetes across cluster nodes.

Parameter reference¶

Slurm Configuration Parameters¶

Parameter	Details
slurm_cluster list / elements=dict / required	List of slurm cluster configurations.
cluster_name string / required	Unique name for the slurm cluster.
nfs_storage_name string / required	Indicates the NFS storage name for the NFS storage to be used by this slurm cluster. This is defined in storage_config.yml as name.
vast_storage_name string	Storage name corresponding to the VAST storage to be used by slurm cluster. This should match exactly with an entry in storage_config.yml. The following directories will be mounted on the VAST storage: - /scratch - /tmp - /home - /apps - /projects
skip_merge boolean	Variable indicates whether a specific configuration file path under config_sources should be used as-is without merging
config_sources dict/str	Config can be a file path or inline mapping Value can be: str (File path string) or dict (Inline configuration mapping).
node_discovery_mode string	Node hardware discovery mode. 'homogeneous' for group-based discovery, 'heterogeneous' for individual node discovery. Default: heterogeneous
node_hardware_defaults dict
<node_hardware_defaults_key> dict	Hardware specifications for homogeneous node groups. Key is group name (grp0-grp100)
sockets integer / required	Number of CPU sockets per node Minimum: 1.
cores_per_socket integer / required	Number of CPU cores per socket Minimum: 1.
threads_per_core integer / required	Number of CPU threads per core Minimum: 1.
real_memory integer / required	Memory in MB (exact value to use in Slurm) Minimum: 1.
gres string	GPU resources in format 'gpu:N' (optional)

Kubernetes Configuration Parameters¶

Parameter	Details
service_k8s_cluster list / elements=dict / required	List of service Kubernetes cluster configurations.
cluster_name string / required	Name of the cluster on which you want to deploy Kubernetes. This input is case-sensitive. Do not add any special characters except `_` (underscore) in the cluster name.
deployment boolean	Indicates if Kubernetes will be deployed or not. Accepted values: `true` or `false`
etcd_on_local_disk boolean / required	Determines whether ETCD is deployed on local disk or NFS storage. Accepted values: `true` or `false` Default value: `false` When set to `true`, ETCD is deployed on local disk on all master nodes. The system prioritizes BOSS card if available, and falls back to SSD/SATA disks if BOSS is not present. The `/var/lib/etcd` directory is mounted on the selected local disk. When set to `false` or omitted, ETCD storage is provisioned using NFS, and no local disk configuration is performed for ETCD. Important Migration from NFS to local disk is not supported during upgrades. This configuration is only applicable for fresh installations.
k8s_cni string / required	Kubernetes SDN network. Accepted values: `calico` Default value: `calico`
pod_external_ip_range string	These addresses will be used by the loadbalancer for assigning external IPs to Kubernetes services. Ensure that the IP range provided is not assigned to any node in the cluster. Ensure that the `pod_external_ip_range` defined in the `omnia_config.yml` file is reachable from the OpenManage Enterprise appliance and the SFM network. Sample values: `172.16.107.170-172.16.107.200`
k8s_service_addresses string / required	Kubernetes internal network for services. This network must be unused in your network infrastructure. Default value: `"10.233.0.0/18"`
k8s_pod_network_cidr string	Kubernetes pod network CIDR for internal network. When used, it will assign IP addresses from this range to individual pods. This network must be unused in your network infrastructure. Default value: `"10.233.64.0/18"`
nfs_storage_name string	Use same name as mentioned in each of the `name` available in `storage_config.yml`.
csi_powerscale_driver_secret_file_path string	If you want to deploy the CSI driver for PowerScale on your service cluster, add the file path of the `secrets.yaml` file to this variable.
csi_powerscale_driver_values_file_path string	If you want to deploy the CSI driver for PowerScale on your service cluster, add the file path of the `values.yaml` file to this variable.
k8s_crio_storage_size string / required	Specifies the disk size allocated for CRI-O container storage.

Usage example¶

File: /opt/omnia/input/project_default/omnia_config.yml

---
slurm_cluster:
  - cluster_name: slurm_cluster
    nfs_storage_name: nfs_slurm
    vast_storage_name: vast_storage
    # Optional: Override Slurm and cgroup configuration
    config_sources:
      slurm: /path/to/custom/slurm.conf
      cgroup: /path/to/custom/cgroup.conf
      # slurm:
      #   SlurmctldTimeout: 60
      #   SlurmdTimeout: 150  
    # Optional: Override hardware specs for specific node groups
    node_hardware_defaults:
      grp1:
        sockets: 2
        cores_per_socket: 64
        threads_per_core: 2
        real_memory: 512000
        gres: "gpu:4"
      grp2:
        sockets: 2
        cores_per_socket: 32
        threads_per_core: 2
        real_memory: 256000


service_k8s_cluster:
  - cluster_name: service_cluster
    deployment: true
    etcd_on_local_disk: false
    k8s_cni: "calico"
    pod_external_ip_range: "172.16.107.170-172.16.107.200"
    k8s_service_addresses: "10.233.0.0/18"
    k8s_pod_network_cidr: "10.233.64.0/18"
    nfs_storage_name: "nfs_k8s"
    k8s_crio_storage_size: "20G"
    csi_powerscale_driver_secret_file_path: ""
    csi_powerscale_driver_values_file_path: ""

Info

Software Config -- Package-level software selection.
Slurm Conf -- Custom Slurm configuration.
HA Config -- Kubernetes high-availability settings.