Slurm Issues¶

Issues related to the Slurm job scheduler, including controller failures, node state problems, job submission errors, and GPU detection.

`slurmctld` Not Starting¶

Symptom

The Slurm controller daemon fails to start. Running systemctl status slurmctld shows the service as failed or inactive.

Cause

The slurm.conf file has syntax errors or references non-existent nodes.
The munge authentication service is not running.
File permissions on /var/spool/slurmctld/ are incorrect.
The Slurm database daemon (slurmdbd) is unreachable and AccountingStorageEnforce is set.

Resolution

Check the slurmctld log for specific errors:
Run on: Slurm controller node
```
tail -100 /var/log/slurm/slurmctld.log
```
Verify munge is running:
Run on: Slurm controller node
```
systemctl status munge
```
If munge is not running, start it:
Run on: Slurm controller node
```
systemctl start munge
```

Validate the Slurm configuration:

Run on: Slurm controller node

slurmd -C    # Show computed configuration
slurmctld -Dvvv    # Run in foreground with verbose logging

Fix spool directory permissions:

Run on: Slurm controller node

chown -R slurm:slurm /var/spool/slurmctld/
chmod 755 /var/spool/slurmctld/

If slurmdbd is the issue, see the slurmdbd connection issues section below.

Nodes Entering DRAINED State¶

Symptom

Slurm nodes enter DRAINED state unexpectedly. Error messages include:

Expected output

State=IDLE+DRAIN Reason=Kill task failed

or:

Expected output

State=DOWN+DRAIN Reason=Not responding

Cause

To identify the root cause, first check the drain reason:

Run on: Slurm controller node

scontrol show node <node_name> | grep -i reason

Drain Reason	Root Cause
Kill task failed	Epilog/prolog script error
Not responding	slurmd lost connection to slurmctld (network, firewall, or slurmd crash)
Low RealMemory	Node has less memory than configured in slurm.conf
Node unexpectedly rebooted	Hardware issue or kernel panic
(blank/manual)	Administrator manually drained the node

Resolution

Resolution steps vary by root cause:

1. Epilog script error

Run on: Slurm controller node

chmod 0755 /etc/slurm/epilog.d/logout_user.sh
scontrol update nodename=<node> state=resume
scontrol reconfigure

2. Not responding

Check the slurmd service status on the compute node:

Run on: compute node

systemctl status slurmd
systemctl restart slurmd

Then resume the node from the Slurm controller:

Run on: Slurm controller node

scontrol update nodename=<node> state=resume

3. Low RealMemory

Verify the actual memory available on the node:

Run on: compute node

free -m

Check the configured RealMemory in slurm.conf:

Run on: Slurm controller node

grep <node> /etc/slurm/slurm.conf

Update the RealMemory value in slurm.conf to match the actual available memory, then run:

Run on: Slurm controller node

scontrol reconfigure

Warning

slurm.conf is managed by the slurm_config role. Manual edits will be overwritten on the next provision.yml run. Update the source configuration instead to make permanent changes.

4. Invalid State (Resource Mismatch)

Nodes enter an invalid state when the hardware resources reported by Slurm do not match the actual node configuration. This typically occurs when incorrect iDRAC credentials cause the provisioning system to apply default resource values that do not reflect the actual hardware capabilities.

Resolution

Identify nodes in invalid state:
Run on: Slurm controller node
```
scontrol show node | grep -i invalid
```
SSH to the affected compute node:
Run on: Slurm controller node
```
ssh <node_name>
```
Retrieve actual hardware configuration:
Run on: compute node
```
slurmd -C
```
The slurmd -C command outputs comprehensive hardware information including CPU architecture, core count, threads per core, sockets, RealMemory, GPU presence and model, and other resource specifications.
Document the actual hardware values from the slurmd -C output for comparison with the Slurm configuration.
SSH to the Slurm control node:
Run on: compute node
```
ssh <slurm_controller_host>
```
Update slurm.conf to match actual hardware:
Run on: Slurm controller node
```
sudo nano /etc/slurm/slurm.conf
```
Locate the node configuration section and update the resource values (CPUs, RealMemory, GPUs, etc.) to match the actual hardware from step 3.
Apply the configuration changes:
Run on: Slurm controller node
```
sudo scontrol reconfigure
```

Resume the node:

Run on: Slurm controller node

sudo scontrol update nodename=<node_name> state=resume

Verify the node state:
Run on: Slurm controller node
```
sudo scontrol show node <node_name>
```
Confirm that the node no longer shows an invalid state and that the resource values are correct.

Note

When using the slurm_config role to manage slurm.conf, update the source configuration (inventory variables or configuration files) rather than manually editing /etc/slurm/slurm.conf. Manual edits are overwritten on the next provision.yml execution.

Prevention

To prevent resource mismatch issues: - Verify iDRAC credentials are correct before provisioning to ensure accurate hardware discovery

Nodes Stuck in DOWN State¶

Symptom

sinfo shows one or more nodes in down or down* state:

PARTITION  AVAIL  TIMELIMIT  NODES  STATE  NODELIST
normal*       up   infinite      1  down*  compute-03

Cause

The slurmd service on the compute node is not running.
Network connectivity between the control node and the compute node is broken.
The node was manually set to DOWN and not resumed.
Hardware issues (memory errors, disk failures) triggered an automatic drain.

Resolution

Check why the node is down:
Run on: Slurm controller node
```
scontrol show node compute-03 | grep -i reason
```
Verify slurmd is running on the compute node:
Run on: Slurm controller node
```
ssh compute-03 systemctl status slurmd
```
If not running:
Run on: Slurm controller node
```
ssh compute-03 systemctl start slurmd
```
Test network connectivity:
Run on: Slurm controller node
```
ping compute-03
ssh compute-03 hostname
```
Resume the node after fixing the underlying issue:
Run on: Slurm controller node
```
scontrol update NodeName=compute-03 State=RESUME
```
Verify the node returns to idle:
Run on: Slurm controller node
```
sinfo -n compute-03
```

Job Submission Failures¶

Symptom

Submitting a job with sbatch or srun fails with errors such as:

sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified
srun: error: Unable to allocate resources: No partition specified or system default partition

Cause

The user's account is not configured in Slurm accounting.
No default partition is defined in slurm.conf.

Resolution

Verify account associations:

Run on: Slurm controller node

sacctmgr show assoc user=$USER format=User,Account,Partition

Check available accounts:
Run on: Slurm controller node
```
sacctmgr show user $USER
```
Submit with a valid account:
Run on: Slurm controller node
```
sbatch --account=myaccount job.sh
```
Check available partitions:
Run on: Slurm controller node
```
sinfo
scontrol show partition
```

Verify a default partition exists in slurm.conf:

File: /etc/slurm/slurm.conf

PartitionName=normal Nodes=compute-[01-10] Default=YES MaxTime=INFINITE State=UP

Update sbatch job script with the right partition:
```
#SBATCH --partition=normal
```

`slurmdbd` Connection Issues¶

Symptom

slurmctld logs show errors connecting to the Slurm database daemon:

error: slurmdbd: Sending PersistInit msg: CONNECTION REFUSED
error: slurmdbd: DBD_ID_REGISTER failed

Cause

The slurmdbd service is not running.
The MariaDB database backend is down.
Network or firewall issues between the controller and the database node.
Incorrect database credentials in slurmdbd.conf.

Resolution

Check slurmdbd status:
Run on: Slurm controller node
```
systemctl status slurmdbd
```
Check the database backend:
Run on: Slurm controller node
```
systemctl status mariadb
```
Verify slurmdbd.conf settings:
Run on: Slurm controller node
```
grep -i storage /etc/slurm/slurmdbd.conf
```

Test database connectivity:

Run on: Slurm controller node

mysql -u slurm -p -h localhost slurm_acct_db -e "SELECT 1;"

Check the slurmdbd log:
Run on: Slurm controller node
```
tail -100 /var/log/slurm/slurmdbd.log
```
If credentials changed, update slurmdbd.conf and restart:
Run on: Slurm controller node
```
systemctl restart slurmdbd
systemctl restart slurmctld
```

NVIDIA GPU, CUDA, and DCGM Issues¶

`nvidia-smi` Not Found or Driver Not Communicating¶

Symptom

nvidia-smi: command not found or nvidia-smi exits with a non-zero return code.

Cause

NVIDIA driver installation failed during provisioning.
GPU hardware is absent on this node.

Resolution

Verify GPU hardware is present on the node:
Run on: GPU compute node
```
lspci | grep -i nvidia
```
If confirmed present, re-install the driver:
Run on: GPU compute node
```
dnf install -y cuda-drivers
```
Review the driver installation log for error details:
Run on: GPU compute node
```
cat /var/log/nvidia_install.log
```

CUDA Toolkit Not Available on Node¶

Symptom

nvcc: command not found or /usr/local/cuda is empty.

Cause

Toolkit installation did not complete on the designated installer node due to a repository or NFS error.
NFS mount for the CUDA toolkit was not established at provisioning time.

Resolution

Verify the NFS mount at /usr/local/cuda is present:
Run on: GPU compute node
```
mount | grep cuda
```
If absent, re-mount manually. If the toolkit is not installed on the NFS share, review the installation log on the installer node:
Run on: installer node (login/compiler or compute)
```
cat /var/log/cuda_toolkit_install.log
```

CUDA Toolkit NFS Mount Failed¶

Symptom

/usr/local/cuda is empty or not mounted after provisioning.

Cause

NFS server was unreachable at provisioning time.
The NFS export is not configured with no_root_squash.

Resolution

Verify NFS server reachability from the node.
Verify the NFS export includes no_root_squash.

Re-mount manually:

Run on: GPU compute node

mount -t nfs <NFS_SERVER>:<path>/hpc_tools/cuda /usr/local/cuda

Verify the fstab entry is present for persistence.

`nvidia-dcgm` Service Inactive or Failed¶

Symptom

systemctl status nvidia-dcgm shows inactive or failed state.

Cause

DCGM package installation failed due to an unavailable repository or a CUDA version mismatch.
The NVIDIA driver was not functional at the time DCGM attempted to start.

Resolution

Verify driver is functional:
Run on: GPU compute node
```
nvidia-smi
```
Identify the installed CUDA version:
Run on: GPU compute node
```
nvidia-smi | grep "CUDA Version"
```
Re-install the matching DCGM package and restart the service.
Review the DCGM setup log for errors:
Run on: GPU compute node
```
cat /var/log/dcgm_setup.log
```

DCGM Not Installed¶

Symptom

nvidia-dcgm service is not present on the Slurm node, and /var/log/dcgm_setup.log is missing.

Cause

dcgm.metrics_enabled is set to false under telemetry_sources in telemetry_config.yml, so Omnia intentionally skips DCGM installation during Slurm node cloud-init.

Resolution

Set dcgm.metrics_enabled: true under telemetry_sources in input/telemetry_config.yml.
Re-run provisioning for affected Slurm nodes.

Validate:

Run on: GPU compute node

systemctl status nvidia-dcgm
dcgmi discovery -l

DCGM Package Version Mismatch¶

Symptom

DCGM package installation fails with No match for argument or No packages found.

Cause

The CUDA major version on the node does not have a matching datacenter-gpu-manager-4-cuda<N> package available in the configured local repository.

Resolution

Verify the CUDA version:
Run on: GPU compute node
```
nvidia-smi | grep "CUDA Version"
```
Confirm the corresponding DCGM package is present in the local Pulp repository.
Update local_repo_config.yml to include the correct DCGM package version and re-run local_repo.yml.

`nvidia-peermem` Not Loading¶

Symptom

lsmod does not show nvidia_peermem. Workloads requiring GPUDirect RDMA fail to initialize.

Cause

Kernel headers were not available at provisioning time, causing the DKMS build to fail.
Base NVIDIA kernel modules were not loaded prior to nvidia-peermem load attempt.

Resolution

Verify kernel headers:
Run on: GPU compute node
```
ls /lib/modules/$(uname -r)/build
```

Install if missing:

Run on: GPU compute node

dnf install -y kernel-devel-$(uname -r)

Load the module:
Run on: GPU compute node
```
modprobe nvidia-peermem
```
Review the installation log:
Run on: GPU compute node
```
cat /var/log/nvidia_peermem_install.log
```

Note

If RDMA is not required for any workload on this node, this warning is non-blocking.

Munge Authentication Failure¶

Symptom

Slurm commands fail with authentication errors such as:

srun: error: Munge encode failed: Failed to access"/var/run/munge/munge.socket.2"
slurmd: error: Munge authentication failed

Cause

The munge service is not running on one or more nodes.
The Munge key is not identical across all nodes in the cluster.

Resolution

Verify Munge is running on all nodes:

Run on: omnia_core container

ansible slurm_cluster -m shell -a "systemctl status munge"

Verify the Munge key is identical across all nodes:
Run on: omnia_core container
```
ansible slurm_cluster -m shell -a "md5sum /etc/munge/munge.key"
```

All nodes should report the same MD5 hash.

If keys differ, redistribute the key from the controller node and restart Munge on all affected nodes:
Run on: affected node
```
systemctl restart munge
```

MariaDB Connection Error¶

Symptom

slurmdbd fails to start or slurmctld logs show database connection errors. The sacctmgr command returns errors.

Cause

The MariaDB service is not running on the Slurm controller node.
Database credentials in slurmdbd.conf do not match the MariaDB configuration.
The Slurm database was not initialized.

Resolution

Check MariaDB is running on the control node:
Run on: Slurm controller node
```
systemctl status mariadb
```
Test database connectivity:
Run on: Slurm controller node
```
mysql -u slurm -p -e "SHOW DATABASES;"
```

If MariaDB is stopped, start it and restart Slurm services:

Run on: Slurm controller node

systemctl start mariadb
systemctl restart slurmdbd
systemctl restart slurmctld

`slurmctld` Fails to Start After Config Rollback¶

Symptom

After rolling back a Slurm configuration backup, slurmctld fails to start.

Cause

The backup may reference nodes that no longer exist in the cluster.

Resolution

Run cleanup and redeploy with provision.yml:

Run on: omnia_core container

ansible-playbook /opt/omnia/utils/slurm_config_util.yml --tags slurm_cleanup
ansible-playbook provision.yml

New Nodes Show DOWN After Adding¶

Symptom

Newly added compute nodes appear as down in sinfo after running provision.yml and PXE booting.

Cause

The slurmd service failed to start on the new node.
Cloud-init did not complete successfully.

Resolution

Verify slurmd is running on the new node:

Run on: new compute node

systemctl status slurmd
journalctl -u slurmd --no-pager -n 20

Check cloud-init completed successfully:
Run on: new compute node
```
cat /var/log/cloud-init-output.log | tail -20
```

Resume the node from the controller:

Run on: Slurm controller node

scontrol update nodename=<node> state=resume reason="added"

Node Still Appears in sinfo After Removal¶

Symptom

After removing a node, it still appears in sinfo output.

Cause

The provision.yml playbook did not complete successfully.
The Slurm controller has not been reconfigured.

Resolution

Verify that provision.yml completed successfully and check the Slurm controller logs:
Run on: Slurm controller node
```
journalctl -u slurmctld --no-pager -n 20
```
If jobs were running on the removed node, they may show as FAILED or NODE_FAIL in accounting:
Run on: Slurm controller node
```
sacct --starttime=today --state=FAILED,NODE_FAIL
```

Resubmit affected jobs as needed.

CUDA Toolkit and DCGM Setup Failure: Manual Recovery¶

Symptom

Automated GPU setup fails during provisioning.

Cause

Repository unavailability, NFS connectivity issues, or node initialization errors.

Resolution

Perform all recovery steps as root on the affected node. Verify that the shared NFS path is reachable and repositories are accessible before proceeding.

Step 1: Verify Prerequisites

Run on: affected GPU node

showmount -e <NFS_SERVER_IP>
lspci | grep -i nvidia
dnf repolist | grep -i cuda
df -h /usr/local

Step 2: Recover NVIDIA Driver

If nvidia-smi is missing or returning errors:

Run on: affected GPU node

dnf install -y cuda-drivers
nvidia-smi

Step 3: Recover CUDA Toolkit

The recovery procedure differs depending on the node topology.

Scenario A -- Login or compiler node present in the cluster:

The login/compiler node is the designated installer. It installs the toolkit to the shared NFS location at /hpc_tools/cuda. Compute nodes mount this path at /usr/local/cuda.

On the login or compiler node:

Run on: login/compiler node

ls /hpc_tools/cuda/bin/nvcc 2>/dev/null && echo "Toolkit present" || echo "Toolkit NOT present"

If not present, trigger the installation manually:

Run on: login/compiler node

CUDA_INSTALL_MANUAL=true /usr/local/bin/install_cuda_toolkit.sh

On a compute node (after toolkit is confirmed on NFS):

Run on: GPU compute node

mount | grep cuda

If absent, re-mount manually:

Run on: GPU compute node

mount -t nfs <NFS_SERVER>:<hpc_tools_path>/hpc_tools/cuda /usr/local/cuda

Scenario B -- No login or compiler node in the cluster:

Compute nodes install the toolkit themselves to /hpc_tools/cuda.

Run on: any compute node

ls /hpc_tools/cuda/bin/nvcc 2>/dev/null && echo "Toolkit present" || echo "Toolkit NOT present"

If not present:

Run on: any compute node

CUDA_INSTALL_MANUAL=true /usr/local/bin/install_cuda_toolkit.sh

Note

Run this only after confirming no active toolkit installation is already in progress. Review /var/log/cuda_toolkit_install.log to check current installation status.

Step 4: Recover DCGM

If the nvidia-dcgm service is inactive or failed:

Run on: GPU compute node

nvidia-smi | grep "CUDA Version"
dnf install -y datacenter-gpu-manager-4-cuda<N>
systemctl enable nvidia-dcgm
systemctl start nvidia-dcgm

Validate:

Run on: GPU compute node

systemctl status nvidia-dcgm
dcgmi discovery -l

Step 5: Recover nvidia-peermem (RDMA environments only)

Run on: GPU compute node

ls /lib/modules/$(uname -r)/build
dnf install -y kernel-devel-$(uname -r)
modprobe nvidia-peermem

Validate:

Run on: GPU compute node

lsmod | grep -E 'nv_peer_mem|nvidia_peermem'

Log file reference:

/var/log/nvidia_install.log -- NVIDIA driver installation output
/var/log/cuda_toolkit_install.log -- CUDA toolkit installation output and timing
/var/log/dcgm_setup.log -- DCGM package install, service startup, GPU discovery
/var/log/nvidia_peermem_install.log -- nvidia-peermem DKMS build and load output

Benchmark Assets Missing on Slurm Nodes¶

Symptom

Benchmark tool directories are missing or incomplete under /hpc_tools.
Expected benchmark artifacts are not visible on login/compiler/compute nodes.

Cause

Shared NFS path (/hpc_tools) is not mounted or not accessible.
pull_benchmarks.sh or benchmark_tools.list is missing under /hpc_tools/scripts.
Pulp mirror endpoint is unreachable from the node.
Required benchmark content is not available in local repository/Pulp.
Tool directory already exists and contains files (script skips re-download by design).
Architecture mismatch (for example, msr-safe on aarch64, which is skipped by design).

Resolution

Verify NFS and scripts path:
Run on: affected node
```
ls -ld /hpc_tools
ls -l /hpc_tools/scripts
```
Expected files: /hpc_tools/scripts/pull_benchmarks.sh and /hpc_tools/scripts/benchmark_tools.list.
Run the runtime staging script and review output:
Run on: affected node
```
/hpc_tools/scripts/pull_benchmarks.sh
```

Review the runtime log:

Run on: affected node

tail -n 200 /var/log/pull_benchmarks.log

Validate staged benchmark directories:

Run on: affected node

ls -l /hpc_tools/osu-micro-benchmarks /hpc_tools/imb /hpc_tools/likwid /hpc_tools/papi /hpc_tools/geopm /hpc_tools/sionlib

If a tool was skipped as already present, remove that tool directory only if refresh is required, then re-run /hpc_tools/scripts/pull_benchmarks.sh.

Note

msr-safe is expected only on x86_64.

`sacct` Erroring Out or Returning Empty Results¶

Symptom

The sacct command returns no output or empty results when querying job accounting information.

Cause

slurmdbd service is not running.
MariaDB service is not running (slurmdbd depends on MariaDB).
slurmdbd cannot communicate with the database.
Port 6819 (slurmdbd port) is not listening.

Resolution

slurmdbd service is not running:

Restart the slurmdbd service and verify its operational status:
Run on: Slurm controller node
```
systemctl restart slurmdbd
systemctl status slurmdbd
```
If the service fails to start, review the system logs for error details
Run on: Slurm controller node
```
journalctl -u slurmdbd -n 50 --no-pager
```
MariaDB not running:

Restart MariaDB and allow it to fully initialize before restarting slurmdbd
Run on: Slurm controller node
```
systemctl restart mariadb
systemctl restart slurmdbd
```
Database credential mismatch:

Verify that the StorageUser and StoragePass credentials in /etc/slurm/slurmdbd.conf match the actual MariaDB user credentials:
Run on: Slurm controller node
```
grep -E 'StorageUser|StoragePass|StorageLoc' /etc/slurm/slurmdbd.conf
```
If the credentials are incorrect, update slurmdbd.conf with the correct values and restart the service:
Run on: Slurm controller node
```
systemctl restart slurmdbd
```
ClusterName mismatch:

Compare the cluster name configured in slurm.conf with what slurmdbd recognizes:
Run on: Slurm controller node
```
grep ClusterName /etc/slurm/slurm.conf
sacctmgr show clusters
```
If the cluster names do not match, re-register the cluster with the correct name:
Run on: Slurm controller node
```
sacctmgr add cluster <correct_cluster_name>
```
Port 6819 blocked by firewall:

Verify that port 6819 (slurmdbd port) is open in the firewall:
Run on: Slurm controller node
```
firewall-cmd --list-ports | grep 6819
```
If the port is not listed, add it to the firewall and reload the configuration:
Run on: Slurm controller node
```
firewall-cmd --add-port=6819/tcp --permanent
firewall-cmd --reload
systemctl restart slurmdbd
```

Validation

After applying the appropriate fix, confirm that accounting is functioning correctly:

Run on: Slurm controller node

# Verify the cluster is registered with slurmdbd
sacctmgr show clusters

# Query recent job accounting data
sacct -S now-1hours

# Confirm accounting storage type configuration
scontrol show config | grep AccountingStorage

Slurm RPM Build Failures¶

Symptom

rpmbuild fails when building Slurm RPMs from source.

Cause

Missing development dependencies on the build host.
Missing kernel headers on aarch64 build hosts.
CUDA toolkit not installed before building with GPU support.

Resolution

Install missing development packages:
Run on: build host
```
dnf install -y <missing-package>-devel
```
For aarch64 builds, install kernel headers:
Run on: aarch64 build host
```
dnf install -y kernel-devel kernel-headers
```
Verify CUDA is installed before running rpmbuild with GPU support:
Run on: build host
```
ls /usr/local/cuda/lib64/stubs/libnvidia-ml.so
```

Info

Setup Slurm -- Slurm cluster setup guide.
Slurm With GPU -- GPU configuration for Slurm.
Add Remove Nodes -- Adding or removing Slurm nodes.

Slurm Issues¶

slurmctld Not Starting¶

Nodes Entering DRAINED State¶

Nodes Stuck in DOWN State¶

Job Submission Failures¶

slurmdbd Connection Issues¶

NVIDIA GPU, CUDA, and DCGM Issues¶

nvidia-smi Not Found or Driver Not Communicating¶

CUDA Toolkit Not Available on Node¶

CUDA Toolkit NFS Mount Failed¶

nvidia-dcgm Service Inactive or Failed¶

DCGM Not Installed¶

DCGM Package Version Mismatch¶

nvidia-peermem Not Loading¶

Munge Authentication Failure¶

MariaDB Connection Error¶

slurmctld Fails to Start After Config Rollback¶

New Nodes Show DOWN After Adding¶

Node Still Appears in sinfo After Removal¶

CUDA Toolkit and DCGM Setup Failure: Manual Recovery¶

Benchmark Assets Missing on Slurm Nodes¶

sacct Erroring Out or Returning Empty Results¶

Slurm RPM Build Failures¶

`slurmctld` Not Starting¶

`slurmdbd` Connection Issues¶

`nvidia-smi` Not Found or Driver Not Communicating¶

`nvidia-dcgm` Service Inactive or Failed¶

`nvidia-peermem` Not Loading¶

`slurmctld` Fails to Start After Config Rollback¶

`sacct` Erroring Out or Returning Empty Results¶