Path A: Slurm Quick Start

Overview

Use this deployment path to build and provision a Slurm HPC cluster with Omnia.

The workflow prepares the Omnia Infrastructure Manager (OIM), synchronizes catalog content with Repository Manager, and builds images for the Slurm functional groups with Image Build Manager. After you provide a reviewed PXE mapping, either manually or through Discovery, Orchestrator provisions the nodes and validates the Slurm deployment. Complete the stages in the order shown because each stage supplies input to the next stage.

What you will build

Cluster role	PXE mapping functional group	Purpose
OIM (management)	Not applicable	Hosts the shared Omnia environment and runs the deployment workflows. The OIM does not join the Slurm cluster.
Slurm controller (head node)	slurm_control_node_x86_64	Runs slurmctld, slurmdbd, and the accounting database. The controller is x86_64 only.
Compute node	slurm_node_x86_64 or slurm_node_aarch64	Runs slurmd and executes jobs submitted to the cluster.
Login node	login_node_x86_64 or login_node_aarch64	Provides interactive SSH access for users to submit jobs.
Login/compiler node	login_compiler_node_x86_64 or login_compiler_node_aarch64	Provides login access and compiler toolchains for building applications.

The table shows the role-and-architecture names generated by Discovery. The PXE mapping can instead use the matching catalog-qualified names, such as slurm_control_node_rhel_10_0_x86_64, when they match the selected catalog. The catalog must define every role and architecture used in the mapping, and Image Build Manager must produce an image for each one. See Create a mapping file for the supported names and mapping rules.

Slurm deployment workflow

Required module sequence and output handoffs

Start on the OIM

Configure and set up the OIM

omnia.env → omnia.sh --setup-venv

Learn more: OIM setup >>

Synchronize catalog content

Make catalog repositories available for image builds

Learn more: Repository Manager >>

Build Slurm node images

Create images for the selected Slurm functional groups

Learn more: Build OS images >>

Provide the PXE mapping

OME Discovery or a manual CSV

Learn more: Create the PXE mapping >>

Configure and run Orchestrator

OpenCHAMI, Slurm provisioning, and optional PXE boot

Learn more: Provision nodes >>

Verify provisioning and Slurm services

Confirm node provisioning and check Slurm with sinfo

Learn more: Verify Slurm >>

Slurm cluster ready

Prerequisites

• Run the entire workflow as the root user on the OIM.
• Use an Omnia source checkout on the OIM.
• Use Python 3.11 or later. The setup script checks for Python 3.12, then Python 3.11, and then Python 3.
• For the initial setup, set SYSTEM_ADMIN_NIC_IPV4 in src/main/omnia.env to an IPv4 address assigned to an OIM interface. Review the project name, shared data path, hostname, domain, Omnia version, and catalog path in the same file.
• Select a catalog whose functional layers include Slurm. The catalog package sources must map to repositories configured for Repository Manager.
• Prepare the admin-network values required by Orchestrator and the shared storage referenced by the Slurm cluster configuration.
• For OME discovery, have the OME address and credentials available. For automated PXE boot, the mapping must contain the applicable BMC information and Orchestrator must be able to collect the BMC credentials.

Procedure

Before starting the workflow, change to the working directory. Run this command once and remain in this directory for all subsequent OIM steps:

Run on: OIM host

cd /src/main

1. Configure and set up the OIM

1. For the initial setup only, edit the environment configuration from the Omnia source tree:

   Run on: OIM host

   vi omnia.env
   

2. Create the shared virtual environment, install module dependencies, stage the module input templates, and copy the catalog samples:

   Run on: OIM host

   ./omnia.sh --setup-venv
   

   Initial setup installs src/main/omnia.env as /etc/omnia/omnia.env. After setup, /etc/omnia/omnia.env is the authoritative environment file and is preserved by subsequent setup runs. Edit the installed file for normal configuration changes. Use ./omnia.sh --setup-venv --force-env only when you intend to replace the installed file with src/main/omnia.env.

   This command runs each selected module's domain-init.sh. Do not run the individual initialization scripts again unless a module was skipped or setup used --deps-only.

3. Load the installed environment and activate the shared virtual environment in the current shell:

   Run on: OIM host

   source /etc/profile.d/omnia-env.sh
   source "$OMNIA_DATA_PATH/activate-omnia.sh"
   repo_manager_path="${OMNIA_DATA_PATH}/repo_manager"
   image_build_manager_path="${OMNIA_DATA_PATH}/image_build_manager"
   discovery_path="${OMNIA_DATA_PATH}/discovery"
   orchestrator_path="${OMNIA_DATA_PATH}/orchestrator"
   

   Run these commands in each new shell before using the paths based on environment variables in this guide.

For all environment and setup options, see Configure the environment and Set up the OIM.

2. Configure and run Repository Manager

1. Review these staged inputs:

   • $repo_manager_path/input/$OMNIA_PROJECT_NAME/repo_manager_config.yml
   • $repo_manager_path/input/$OMNIA_PROJECT_NAME/repo_manager_endpoint_config.yml
   • The catalog JSON identified by CATALOG_FILE_PATH

   Ensure the selected catalog includes the required Slurm functional layers and that each selected package source resolves through the configured RPM repository, container registry, or artifact URL.

   At minimum, the catalog must provide an x86_64 controller layer for the controller operating-system version and a compute layer for every operating-system version and architecture used by the mapped compute nodes:

   Required functional layer	Required Slurm component
   slurm_control_node_rhel_<major>_<minor>_x86_64	slurm_custom_group and slurm_control_node_group
   slurm_node_rhel_<major>_<minor>_<arch>	slurm_custom_group and slurm_node_group

   For the documented validated x86_64 RHEL 10.0 deployment, use slurm_control_node_rhel_10_0_x86_64 and slurm_node_rhel_10_0_x86_64. The slurm_custom_group component is mandatory in both layers. Select slurm_x86_64.json or slurm_x86_64_no_vast.json from src/main/samples/catalogs/10.0/ for this topology. For aarch64 or mixed-architecture nodes, select the corresponding shipped Slurm catalog instead. Do not create a catalog containing only the groups shown in this table; the shipped catalogs include the complete base OS, dependency, and package definitions required by the deployment.

   Verify the selected catalog before continuing:

   Run on: OIM host

   python3 - "$CATALOG_FILE_PATH" <<'PY'
   import json
   import sys
   
   try:
       with open(sys.argv[1], encoding="utf-8") as catalog_file:
           layers = json.load(catalog_file)["catalog"]["functionallayer"]
       for layer in layers:
           print(f"{layer['name']}\t{','.join(layer['components'])}")
   except (OSError, json.JSONDecodeError, KeyError, TypeError) as error:
       print(f"Catalog validation failed: {error}", file=sys.stderr)
       raise SystemExit(1)
   PY
   

2. Run the complete standard Repository Manager flow:

   Using omnia.sh (recommended)Using ansible-playbook

   Run on: OIM host

   cd <OMNIA_SOURCE_PATH>/src/main
   ./omnia.sh --run repo_manager
   

   Run on: OIM host

   source /opt/omnia/activate-omnia.sh
   cd <OMNIA_SOURCE_PATH>/src/repo_manager/playbooks
   ansible-playbook repo_manager.yml
   

   The flow validates the environment and inputs, collects or reuses credentials, deploys Pulp, synchronizes the selected content, and writes:

   $repo_manager_path/output/$OMNIA_PROJECT_NAME/repo_status.yml
   

   Do not continue until overall_status is success.

For the configuration and credential procedure, see Create Local Repositories.

3. Configure and run Image Build Manager

1. Review the staged image_build_config.yml under:

   $image_build_manager_path/input/$OMNIA_PROJECT_NAME/
   

   Its repo_manager_output_path must identify the successful repo_status.yml. With functional_groups_source: "catalog", the Image Build Manager resolves the image package sets from CATALOG_FILE_PATH. With functional_groups_source: "config", also configure package_groups.yml in the same project input directory.

2. Run the complete standard image-build flow:

   Using omnia.sh (recommended)Using ansible-playbook

   Run on: OIM host

   cd <OMNIA_SOURCE_PATH>/src/main
   ./omnia.sh --run image_build_manager
   

   Run on: OIM host

   source /opt/omnia/activate-omnia.sh
   cd <OMNIA_SOURCE_PATH>/src/image_build_manager/playbooks
   ansible-playbook image_build_manager.yml
   

   The flow validates the configuration, collects or reuses the applicable S3 and aarch64 credentials, prepares MinIO when selected, deploys the local registry, builds the selected functional-group images, and writes:

   $image_build_manager_path/output/$OMNIA_PROJECT_NAME/build_status.yml
   

   Confirm that overall_status is success and that every Slurm functional group used in the PXE mapping has a corresponding image.

For configuration, build modes, and direct playbook alternatives, see Build Images.

4. Provide the PXE mapping

Choose one method. Orchestrator consumes the reviewed file as $orchestrator_path/input/$OMNIA_PROJECT_NAME/pxe_mapping_file.csv.

Discover nodes through OMECreate the mapping manually

1. Configure discovery_config.yml and network_spec.yml under $discovery_path/input/$OMNIA_PROJECT_NAME/. Set ome_ip to a valid, non-loopback OME IPv4 address.

2. Run Discovery:

   Using omnia.sh (recommended)Using ansible-playbook

   Run on: OIM host

   cd <OMNIA_SOURCE_PATH>/src/main
   ./omnia.sh --run discovery
   

   Run on: OIM host

   source /opt/omnia/activate-omnia.sh
   cd <OMNIA_SOURCE_PATH>/src/discovery/playbooks
   ansible-playbook discovery.yml
   

3. Review the timestamped mapping and discovery report under $discovery_path/output/$OMNIA_PROJECT_NAME/. Then copy the latest mapping to the Orchestrator input directory. With the installed environment active, run:

   Run on: OIM host

   cp "$discovery_path/output/$OMNIA_PROJECT_NAME/bmc_pxe_mapping_file.csv" \
     "$orchestrator_path/input/$OMNIA_PROJECT_NAME/pxe_mapping_file.csv"
   

Discovery intentionally leaves this handoff to the operator so that node hostnames, functional groups, and group assignments can be reviewed before provisioning.

Edit the staged Orchestrator mapping directly:

Run on: OIM host

vi "$orchestrator_path/input/$OMNIA_PROJECT_NAME/pxe_mapping_file.csv"

Preserve the header defined by the source template. Provide a slurm_control_node_ functional group and the required Slurm compute, login, or login/compiler groups for the intended cluster. The Slurm controller list cannot be empty.

The following example uses the shipped RHEL 10.0 x86_64 Slurm catalog:

Example: pxe_mapping_file.csv

FUNCTIONAL_GROUP_NAME,GROUP_NAME,SERVICE_TAG,PARENT_SERVICE_TAG,HOSTNAME,ADMIN_MAC,ADMIN_IP,BMC_MAC,BMC_IP,IB_NIC_NAME,IB_IP
slurm_control_node_rhel_10_0_x86_64,grp0,SLM001,,nid001,02:00:00:00:01:01,172.16.107.51,02:00:00:00:02:01,172.17.107.51,,
slurm_node_rhel_10_0_x86_64,grp1,SLM002,,nid002,02:00:00:00:01:02,172.16.107.52,02:00:00:00:02:02,172.17.107.52,InfiniBand.Slot.7-1,192.168.0.101
slurm_node_rhel_10_0_x86_64,grp1,SLM003,,nid003,02:00:00:00:01:03,172.16.107.53,02:00:00:00:02:03,172.17.107.53,InfiniBand.Slot.7-2,192.168.0.102
login_node_rhel_10_0_x86_64,grp2,SLM004,,nid004,02:00:00:00:01:04,172.16.107.54,02:00:00:00:02:04,172.17.107.54,,

Important

Replace every sample service tag, MAC address, IP address, and hostname with values from the target servers. Keep the exact 11-column header; leave optional fields empty with consecutive commas. For a Slurm-only deployment, leave PARENT_SERVICE_TAG empty. Populate IB_NIC_NAME and IB_IP together, or leave both empty. Use unique lowercase hostnames without a domain suffix; when dns_enabled is true, use nid001 through nid999. Ensure the admin addresses belong to a configured admin subnet and every functional group exists in the selected catalog and successful image-build output. If another RHEL version or architecture is selected, use its exact catalog group names.

Using RHEL 10.0 and RHEL 10.2 nodes

When the selected catalog and Image Build Manager output contain both RHEL 10.0 and RHEL 10.2 Slurm images, the PXE mapping can assign the required image to each node by using its exact functional-group name.

For example:

Functional Group Name	Group Name
slurm_control_node_rhel_10_2_x86_64	controller
slurm_node_rhel_10_0_x86_64	compute
slurm_node_rhel_10_2_x86_64	compute

Before provisioning, verify:

• repo_status.yml reports success for every required RHEL version and architecture.
• build_status.yml contains a successful image for every functional group used in the mapping.
• All controller and compute images contain a compatible Slurm version.
• Nodes whose mapping changed are reprovisioned and PXE booted with the new image.

Platform-dependent HPC content is stored under /hpc_tools/platforms/<os>/<version>/<architecture> and selected from the operating system actually running on each node.

For the complete mapping schema and OME procedure, see Discover Nodes and Create a Mapping File.

5. Configure and run Orchestrator

1. Review the staged files under $orchestrator_path/input/$OMNIA_PROJECT_NAME/:

   Input	Slurm quick-start requirement
   orchestrator_config.yml	Controls pxe_mapping_file_path, repo_manager_output_path, image_build_manager_output_path, catalog_file_path, and enable_pxe_boot.
   set_pxe_boot_config.yml	Controls server restart behavior, force-restart behavior, the PXE override mode, the PXE target device, and node-registration verification timing.
   network_spec.yml	Configure the OIM interface, admin subnet, DHCP range, router, and any optional InfiniBand network.
   omnia_config.yml	Configure slurm_cluster, including its cluster name and storage references.
   storage_config.yml	Define the mounts named by slurm_cluster; the referenced storage must be reachable where configured.
   pxe_mapping_file.csv	Assign the intended nodes to Slurm functional groups and ensure corresponding images exist in build_status.yml.
   security_config.yml	Configure this file when the selected catalog enables OpenLDAP.

   Orchestrator derives Slurm support and the cluster OS metadata from the catalog.

   The storage references in $OMNIA_DATA_PATH/orchestrator/input/$OMNIA_PROJECT_NAME/omnia_config.yml must match mount names defined in $OMNIA_DATA_PATH/orchestrator/input/$OMNIA_PROJECT_NAME/storage_config.yml:

   • Each slurm_cluster[].nfs_storage_name value must match a mounts[].name value.
   • Each non-empty slurm_cluster[].vast_storage_name value must match a mounts[].name value.

   Warning

   The default configuration enables PXE booting, force-restarts mapped servers, and applies the PXE override continuously. If only one provisioning cycle is required, set boot_source_override_enabled: once in set_pxe_boot_config.yml.

2. Run the complete standard Orchestrator flow:

   Using omnia.sh (recommended)Using ansible-playbook

   Run on: OIM host

   cd <OMNIA_SOURCE_PATH>/src/main
   ./omnia.sh --run orchestrator
   

   Run on: OIM host

   source /opt/omnia/activate-omnia.sh
   cd <OMNIA_SOURCE_PATH>/src/orchestrator/playbooks
   ansible-playbook orchestrator.yml
   

   The untagged flow performs prechecks, collects or reuses credentials, prepares OpenCHAMI and catalog-selected services, provisions the Slurm functional groups, validates provisioning, and performs iDRAC PXE boot when enable_pxe_boot: true. Do not run a second PXE-boot command after this flow unless you intentionally need to repeat that operation.

For detailed Slurm and provisioning settings, see Configure Slurm, Configure Storage, and Provision Nodes.

Verification

1. Confirm that the three module contracts report success:

   Run on: OIM host

   grep '^overall_status:' "$repo_manager_path/output/$OMNIA_PROJECT_NAME/repo_status.yml"
   grep '^overall_status:' "$image_build_manager_path/output/$OMNIA_PROJECT_NAME/build_status.yml"
   grep '^overall_status:' "$orchestrator_path/output/$OMNIA_PROJECT_NAME/orchestrator_status.yml"
   

2. Review the provisioning summary and generated inventory:

   Run on: OIM host

   cat "$orchestrator_path/output/$OMNIA_PROJECT_NAME/provisioning_report.yml"
   cat "$orchestrator_path/output/$OMNIA_PROJECT_NAME/orchestrator_inventory.yml"
   

   Node-registration verification must be enabled in set_pxe_boot_config.yml by setting enable_node_registration: true. Confirm that orchestrator_status.yml reports verification_enabled: true and unverified_count: 0. The cluster is not ready for use while any node remains unverified.

3. On the Slurm controller, verify the services, controller connectivity, node state, and job execution:

   Run on: Slurm controller

   systemctl is-active slurmctld
   systemctl is-active slurmdbd
   scontrol ping
   sinfo
   srun -N 1 hostname
   

4. On each Slurm compute node, verify the node daemon:

   Run on: Slurm compute node

   systemctl is-active slurmd
   

Next steps

• Configure Slurm to supply or merge custom Slurm configuration files.
• Configure Slurm with GPUs when the selected catalog and compute nodes include NVIDIA GPU support.
• Set up NVIDIA HPC SDK when the catalog includes the required SDK content.
• Run HPC benchmarks after the required benchmark assets have been staged by Orchestrator.
• Use Add Nodes and Remove Slurm Nodes for supported node lifecycle changes.

Troubleshooting

• If a later module reports a missing upstream contract, verify that the preceding status file exists and has overall_status: success.
• If Slurm configuration is skipped, verify that the catalog contains a Slurm functional layer and that the mapping contains a functional group beginning with slurm_control_node_.
• If an image validation fails, compare every mapping functional group with functional_group_images in build_status.yml.
• If physical nodes do not PXE boot, confirm enable_pxe_boot: true, review failed_nodes.json, and inspect orchestrator_status.yml in the Orchestrator project output directory.
• Review the module logs under /var/log/omnia/<domain>/ and the project logs under <OMNIA_DATA_PATH>/<domain>/log/.
• See Orchestrator troubleshooting for component-specific investigations.