General Issues¶

Issues that affect the OIM, core containers, OpenCHAMI services, SSH connectivity, system recovery, and Ansible Vault operations.

Common Container Debugging Tools¶

Use the following commands to troubleshoot container issues across Omnia services.

View all Omnia containers

Run on: OIM host

podman ps -a

View container logs

Run on: OIM host

podman logs -n 200 <container>

Test outbound connectivity from a container

Run on: OIM host

podman exec -it <container> sh -lc 'curl -I https://example.com'

Omnia Core Container Fails to Deploy¶

Symptom

omnia.sh aborts early.
podman pull fails.
Container starts but cannot write to shared path.

Cause

Podman pull or authentication issues.
Time synchronization failure.
Invalid OIM hostname.
NFS or SELinux permission issues.

Resolution

Check container status:

Run on: OIM host

podman ps --format 'table {{.Names}}\t{{.Status}}'

Check container logs:
Run on: OIM host
```
podman logs -n 200 omnia_core
```

Check time synchronization:

Run on: OIM host

timedatectl status
chronyc tracking || chronyc sources -v

Validate OIM hostname (no dots, underscores, commas, uppercase, leading/trailing hyphens, or leading digits; FQDN must be 64 characters or fewer).

Validate NFS mount and SELinux labeling:

Run on: OIM host

podman run --rm -v /shared:/mnt:z registry.access.redhat.com/ubi10/ubi sh -lc 'touch /mnt/.rw'

Re-run omnia.sh.

Prepare OIM Failures¶

Symptom

Certificate or TLS failures during prepare_oim.yml.
Expected container not created.
Service is running but unreachable.

Cause

Invalid or expired TLS certificates.
Container image pull failures.
Network connectivity issues.
Incorrect configuration parameters.

Resolution

Verify container inventory:

Run on: OIM host

podman ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'

Review container logs for specific error messages.
Verify network connectivity and TLS certificate validity.
Re-run prepare_oim.yml after correcting the issue.

Ansible Vault Decryption Failures¶

Symptom

Playbook execution fails with:

Expected output

ERROR! Attempting to decrypt but no vault secrets found

Cause

The vault password file (.omnia_config_credentials_key) is missing, incorrect, or inaccessible to the playbook execution context.

Resolution

Verify the vault password file exists in the correct location.
Ensure the file has the correct permissions (readable by the user running the playbook).
Re-run the playbook with the correct vault password file:
Run on: omnia_core container
```
ansible-playbook playbooks/omnia.yml --vault-password-file /root/.vault_pass
```

If the vault password is lost, recreate the credentials file:

Run on: omnia_core container

cp input/credentials.yml input/credentials.yml.bak
ansible-vault create input/credentials.yml

For more information on managing encrypted parameters, see Encrypted Parameters Management.

OIM Cleanup NFS Directory Deletion Failure¶

Symptom

oim_cleanup.yml fails with: rmtree failed: [Errno 39] Directory not empty.
Specific error on directories like /share_omnia_k8s/<node_ip>/kubelet/pods.
Cleanup process completes partially but leaves NFS share directories intact.

Expected output

[ERROR]: Task failed: Module failed: rmtree failed: [Errno 39] Directory not empty: '/share_omnia_k8s/10.20.0.15/kubelet/pods'

Cause

Kubernetes processes (kubelet, crio) on compute nodes or OIM have open file handles to NFS share directories.
NFS shares are still mounted and in use on compute nodes.

Note

The OIM cleanup process cleans the contents of NFS shares for both Slurm and Kubernetes. Active processes or mounts may prevent successful cleanup.

Resolution

Manually delete the problematic directories on the OIM node:

Run on: OIM host

# /share_omnia_k8s is the mounted NFS share directory
cd /share_omnia_k8s/<node_ip>/kubelet/pods
rm -rf *
cd /share_omnia_k8s/
rm -rf <node_ip>

Re-run the OIM cleanup playbook:
Run on: omnia_core container
```
cd /omnia/utils
ansible-playbook oim_cleanup.yml
```

Tip

If manual deletion also fails with "Device or resource busy" errors, power off the compute nodes before attempting manual cleanup.

Symptom

SSH connections fail with one of the following errors:

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
Permission denied (publickey,gssapi-keyex,gssapi-with-mic)
ssh: connect to host <ip> port 22: Connection refused

Cause

Outdated SSH key after node re-provisioning.
cloud-init not rendered on the target node.

Resolution

Remove the stale key:
Run on: OIM host
```
ssh-keygen -R <hostname_or_ip>
```
Retry login or reprovision the node.

SSH to omnia_core Container Fails After Switching to Root With Sudo¶

Symptom

After successful execution of the omnia.sh script, a message is displayed indicating that you can log in to the omnia_core container using ssh omnia_core. However, this fails if you initially logged in to the OIM node as a non-root user and then switched to the root user using the sudo command.

Cause

SSH access to the omnia_core container depends on direct root login. When a user logs in as a non-root user and switches to root using the sudo command, the SSH session may not have the required permissions or environment configuration to access the container using ssh omnia_core.

Resolution

Edit the SSH configuration file and set PermitRootLogin yes:
File: /etc/ssh/sshd_config
```
PermitRootLogin yes
```
Restart the SSH service:
Run on: OIM host
```
systemctl restart sshd
```
Log out and re-login to the OIM node directly as the root user.
Restart the omnia_core container:
Run on: OIM host
```
podman restart omnia_core
```

OpenCHAMI Issues¶

Certificate Expiration¶

Symptom

OpenCHAMI certificates have expired, causing service communication failures. This can also cause the cloud-init server to fail when running provision.yml.

Cause

The OpenCHAMI certificate has expired, or one or more openchami.target services are not running.

Resolution

Check if OpenCHAMI target dependencies are satisfied:
Run on: OIM host
```
systemctl list-dependencies openchami.target
```

Update the certificate and restart the target:

Run on: OIM host

sudo openchami-certificate-update update <OIM_hostname>.<domain>
sudo systemctl restart openchami.target

If certificate expiry issues persist, restart the acme-deploy service:
Run on: OIM host
```
systemctl restart acme-deploy
```
If any other service under the OpenCHAMI target failed, restart it:
Run on: OIM host
```
systemctl restart <service_name>
```
Wait for the OpenCHAMI target and all its dependencies to become active:
Run on: OIM host
```
systemctl is-active openchami.target
```

Token Expired¶

Symptom

OpenCHAMI access token has expired.

Resolution

Run on: OIM host

export <OIM_HOSTNAME>_ACCESS_TOKEN=$(sudo bash -lc 'gen_access_token')

`provision.yml` Fails: prepare_oim Needs to Be Executed¶

Symptom

The provision.yml playbook fails with an error indicating that prepare_oim needs to be executed first.

Cause

The OpenCHAMI container is not up and running.

Resolution

Perform a cleanup using oim_cleanup.yml and re-run prepare_oim.yml to bring up the OpenCHAMI containers. After prepare_oim.yml completes successfully, re-deploy the cluster.

Cluster Not Recovering After Power Cycle¶

Symptom

After the Omnia Infrastructure Manager (OIM) and cluster nodes are powered on, one or more of the following conditions may occur:

One or more compute nodes remain NotReady, Down, Unknown, or otherwise unavailable
Kubernetes nodes do not rejoin the cluster
Kubernetes pods remain in Pending, ContainerCreating, CrashLoopBackOff, or Unknown state
Slurm nodes remain DOWN, DRAIN, NOT_RESPONDING, or are missing from sinfo
Omnia services on the OIM are inactive or failed
OpenCHAMI, Pulp, authentication, provisioning, or related Podman containers are not running
Compute nodes cannot reach the OIM over the management network
NFS-backed directories are unavailable or commands accessing them hang
Cluster applications fail because shared storage, DNS, time synchronization, or control-plane services are unavailable

Note

Do not immediately reprovision an affected node. First determine whether the failure is caused by an unavailable OIM service, network connectivity, shared storage, or a node-local service.

Cause

The compute nodes were powered on before the OIM became fully operational
One or more services associated with omnia.target failed to start
An Omnia Podman container stopped or entered an unhealthy state
The management interface, routing, DNS, firewall, or VLAN configuration did not recover correctly
An NFS server or NFS mount is unavailable
Time synchronization has not recovered
Kubernetes or Slurm node services failed to start
A node retained stale network, mount, or runtime state after the power interruption
The node operating system or provisioning state was damaged by an ungraceful shutdown

Resolution

Perform the following checks in order. Resolve OIM-wide dependencies before troubleshooting individual compute nodes.

1. Verify that the OIM is fully operational

Log in to the OIM and verify that the operating system has completed startup:

Run on: OIM host

uptime
systemctl is-system-running
systemctl --failed --no-pager

If systemctl is-system-running reports starting, wait for startup jobs to complete and run the command again. If it reports degraded, examine the failed units:

Run on: OIM host

systemctl --failed --no-pager

Do not power-cycle the compute nodes again until the required OIM services are operational.

2. Verify Omnia services on the OIM

Check the Omnia core service and the services associated with omnia.target:

Run on: OIM host

systemctl status omnia_core.service --no-pager
systemctl list-dependencies omnia.target

The Omnia deployment documentation identifies omnia_core.service, pulp.service, omnia_auth.service, and the OpenCHAMI services under openchami.target (smd, bss, cloud-init-server, hydra, acme-deploy) as dependencies that may be present under omnia.target. The exact set depends on the deployed configuration.

List failed Omnia and OpenCHAMI-related services:

Run on: OIM host

systemctl --failed --no-pager
systemctl status omnia.target --no-pager
systemctl status openchami.target --no-pager

For each failed service, inspect its journal. For example:

Run on: OIM host

journalctl -u omnia_core.service -b --no-pager
journalctl -u pulp.service -b --no-pager
journalctl -u openchami.target -b --no-pager

To display recent high-priority errors from the current boot:

Run on: OIM host

journalctl -b -p err..alert --no-pager

If a service failed because one of its dependencies was unavailable, correct the dependency first. Then restart only the affected service:

Run on: OIM host

systemctl restart <service_name>
systemctl status <service_name> --no-pager

Replace <service_name> with the failed unit displayed by systemctl --failed. Avoid repeatedly restarting omnia.target without first reviewing the failed service logs. Repeated restarts can obscure the original failure and unnecessarily interrupt healthy services.

3. Verify Omnia Podman containers

List all containers, including containers that exited during startup:

Run on: OIM host

podman ps -a

Check a specific container:

Run on: OIM host

podman ps -a --filter name=<container_name>
podman inspect <container_name> --format 'status={{.State.Status}} exit_code={{.State.ExitCode}} error={{.State.Error}}'

View its recent logs:

Run on: OIM host

podman logs --tail 200 <container_name>

If the container is managed by a systemd unit, restart the corresponding systemd service rather than starting the container manually:

Run on: OIM host

systemctl restart <service_name>
systemctl status <service_name> --no-pager

Use the container logs and the associated systemd journal to determine whether the failure is related to storage, port binding, certificates, database availability, or another service dependency.

4. Verify network recovery between the OIM and compute nodes

On the OIM, check the management interfaces, addresses, routes, and NetworkManager state:

Run on: OIM host

ip -brief address
ip route
nmcli device status
nmcli connection show --active

Test connectivity to each affected compute node:

Run on: OIM host

ping -c 4 <compute_node_ip>
ip neigh show <compute_node_ip>

If hostnames are used, verify name resolution separately:

Run on: OIM host

getent hosts <compute_node_hostname>
ping -c 4 <compute_node_hostname>

On the affected compute node, test the return path to the OIM:

Run on: compute node

ip -brief address
ip route
ping -c 4 <oim_ip>
getent hosts <oim_hostname>

Interpret the results as follows:

No management IP address: Check the interface and NetworkManager connection
No route to the OIM: Correct the route, VLAN, or gateway configuration
IP address works but hostname fails: Investigate DNS or /etc/hosts
OIM can reach the node but the node cannot reach the OIM: Check asymmetric routing, firewall rules, bonding, or VLAN configuration
Duplicate or stale neighbor entry: Check for an IP address conflict before clearing the entry

To inspect recent network-service errors: