Skip to content

Slurm Configuration Files

Overview

Omnia provides flexible mechanisms to manage Slurm configuration files (slurm.conf, slurmdbd.conf, cgroup.conf, gres.conf). You can use the default configurations or supply custom configurations through the config_sources parameter in omnia_config.yml.

Prerequisites

  • The Orchestrator project inputs are staged in the active project's input directory.
  • The omnia_config.yml file is configured with the desired slurm_cluster settings.

Procedure

Configure Slurm using either default or custom configuration as described below.

Default configuration

Omnia applies a default configuration optimized for HPC clusters. These defaults are used unless overridden via config_sources.

Default partition and node settings

Setting Default value
Partition name normal (all compute nodes from PXE mapping file)
Partition config PartitionName=normal Nodes=<compute_nodes> Default=YES MaxTime=INFINITE State=UP
Node config (iDRAC unreachable) NodeName=<nodename> Sockets=2 CoresPerSocket=72 ThreadsPerCore=1 RealMemory=864

Default slurm.conf

Note

The parameters ClusterName, SlurmctldHost, and AccountingStorageHost are auto-detected and cannot be modified.

File: /etc/slurm/slurm.conf (defaults)
# Authentication and Security
AuthType=auth/munge
CredType=cred/munge
SlurmUser=slurm

# Controller Configuration
ClusterName=cluster
SlurmctldHost=<auto-detected>
SlurmctldPort=6817
SlurmctldTimeout=120
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldParameters=enable_configless
StateSaveLocation=/var/spool/slurmctld

# Compute Node Configuration
SlurmdPort=6818
SlurmdTimeout=300
SlurmdLogFile=/var/log/slurm/slurmd.log
SlurmdPidFile=/var/run/slurmd.pid
SlurmdSpoolDir=/var/spool/slurmd

# Accounting
AccountingStorageHost=<auto-detected>
AccountingStoragePort=6819
AccountingStorageType=accounting_storage/slurmdbd

# Job Execution
SrunPortRange=60001-63000
ReturnToService=2
Epilog=/etc/slurm/epilog.d/logout_user.sh
PrologFlags=contain

# Scheduling
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory
GresTypes=gpu
SlurmdParameters=l3cache_as_socket
MaxNodeCount=65000

# Resource Tracking
TaskPlugin=task/cgroup
ProctrackType=proctrack/cgroup
JobAcctGatherType=jobacct_gather/linux
JobAcctGatherFrequency=30

# MPI Configuration
MpiDefault=none

# Plugin Directory
PluginDir=/usr/lib64/slurm

# Default Node Configuration
NodeSet=normal Feature=normal
NodeName=DEFAULT State=UNKNOWN

# Default Partition Configuration
PartitionName=DEFAULT Nodes=ALL MaxTime=INFINITE State=UP
PartitionName=normal Nodes=<compute_nodes> Default=YES MaxTime=INFINITE State=UP

Default slurmdbd.conf

Note

The parameters DbdHost and StorageHost are auto-detected and cannot be modified.

File: /etc/slurm/slurmdbd.conf (defaults)
# Authentication
AuthType=auth/munge
SlurmUser=slurm

# Database Daemon Configuration
DbdHost=<auto-detected>
DbdPort=6819
LogFile=/var/log/slurm/slurmdbd.log
PidFile=/var/run/slurmdbd.pid
PluginDir=/usr/lib64/slurm

# Database Connection
StorageType=accounting_storage/mysql
StorageHost=<auto-detected>
StoragePort=3306
StorageLoc=slurm_acct_db
StorageUser=slurm
StoragePass=<storage_password>

Default cgroup.conf

File: /etc/slurm/cgroup.conf (defaults)
CgroupPlugin=autodetect
ConstrainCores=yes
ConstrainDevices=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes

Default gres.conf

File: /etc/slurm/gres.conf (defaults)
AutoDetect=nvml

Custom configuration sources

Custom configuration files are supplied through the config_sources parameter in omnia_config.yml. Two methods are available:

Parameter-based configuration

Specify individual parameters directly. Omnia merges these values with the defaults:

File: omnia_config.yml
slurm_cluster:
  - cluster_name: slurm_cluster
    nfs_storage_name: nfs_slurm
    vast_storage_name: vast_storage
    config_sources:
      slurm:
        SlurmctldTimeout: 60
        SlurmdTimeout: 150
      cgroup:
        CgroupPlugin: autodetect
        AllowedRAMSpace: 100

File-based configuration

Provide complete custom configuration files:

File: omnia_config.yml
slurm_cluster:
  - cluster_name: slurm_cluster
    nfs_storage_name: nfs_slurm
    config_sources:
      slurm: /path/to/custom_slurm.conf
      cgroup: /path/to/custom_cgroup.conf
      slurmdbd: /path/to/custom_slurmdbd.conf
      gres: /path/to/custom_gres.conf

Merge behavior (skip_merge)

By default, Omnia merges user-provided configurations with defaults to produce a complete configuration. Set skip_merge: true when file-path entries in config_sources must replace the generated defaults:

File: omnia_config.yml
slurm_cluster:
  - cluster_name: slurm_cluster
    nfs_storage_name: nfs_slurm
    skip_merge: true
    config_sources:
      slurm: /path/to/custom_slurm.conf
      cgroup: /path/to/custom_cgroup.conf
      slurmdbd: /path/to/custom_slurmdbd.conf
      gres: /path/to/custom_gres.conf

Warning

When skip_merge: true is set:

  • A file-path value is processed without the generated defaults. The supplied file must therefore be complete and valid.
  • Inline parameter mappings remain supported and continue to merge with generated defaults.
  • A single config_sources mapping may contain both forms. Use a file path for each configuration that must replace its defaults.

Configuration validation

Omnia validates Slurm configuration files (slurm.conf, slurmdbd.conf, cgroup.conf, gres.conf) before deployment. The validator checks that:

  • Parameters are recognized by the supported Slurm version.
  • Parameter values match expected data types.
  • Common configuration errors are detected before deployment.

Info

Verification

After deploying Slurm, verify that the configuration files are applied:

Run on: Slurm control node
scontrol show config | head -30

Confirm that the parameter values match your custom or default configuration.

Next steps

Troubleshooting

  • Configuration validation fails: Check that parameter names match the supported Slurm version. Review the error output for specific invalid parameters.
  • Custom configuration not applied: Verify the file path in config_sources is correct and accessible from the OIM.