Operations & Maintenance

Day-2 operations for managing a running Omnia cluster. These guides cover common administrative tasks you will perform after the initial deployment is complete---scaling or re-provisioning the cluster, managing repositories and logs, and cleaning up domains or the OIM environment.

Tip

If you have not yet deployed your cluster, start with the Get Started Tutorials. The procedures in this section assume a working Omnia environment.

Environment and content lifecycle

• Maintain the Main environment to audit dependency declarations or remove the installed OIM environment.
• Clean up the OIM by removing deployed domains in reverse dependency order before removing the shared execution environment.
• Clean up built images selectively or completely from S3 and the local registry while preserving Image Build Manager services and configuration.

Repository Manager

• Configure VAST storage and build VAST RPMs before adding the hosted VAST repository to Repository Manager.
• Build Slurm RPMs for x86_64 or aarch64 nodes before adding the hosted repository to Repository Manager.
• Build LDMS producer RPMs for the target architecture, host the repository, and synchronize it through Repository Manager before building Slurm node images.
• Update repositories after catalog changes to synchronize revised catalog content and regenerate repo_status.yml.
• Resynchronize local RPM repositories to force selected or all catalog-required RPM remotes to check upstream.

Node lifecycle

• Add nodes through the direct Orchestrator workflow and use a custom inventory to PXE boot only the new physical nodes.
• Remove Slurm compute nodes omitted from the current desired mapping, with source-defined handling for active jobs.
• Add nodes through BuildStreaM by committing the revised Orchestrator mapping and running the deploy pipeline.
• Reprovision a cluster when existing nodes must be provisioned again.

BuildStreaM lifecycle

• Update the BuildStreaM catalog.
• Clean up image groups.
• Retry a failed pipeline.

Diagnostics and recovery

• Collect cluster logs with the Utils collect workflow.
• Back up OIM logs with the Utils backup_oim_logs workflow to local or NFS storage.
• Use the Slurm configuration utilities to back up, remove, or restore the active Slurm configuration.
• Use Log Management for general log inspection.